DetNet Minutes for IETF 118 (Prague and Online)


DetNet Session 1: MONDAY, November 6, 2023 -- Session III

15:30-17:00 CET (14:30-16:00 UTC)

Sesson recording:

Slot Start Duration Information

01) 15:30 15 Chairs Title: Intro, WG Status, Draft Status

Note that RAW docs in flight will be renamed xxx-detnet-raw-xxx, as was suggested in the chairs' intro at IETF 117.

The draft needing contribution is

From chat: Abdussalam Baryun: the draft has 6 authors are they participating or maybe they need one to join? I thought it has about three but 6 maybe enough.

Carlos Bernardos: in addition to the documents noted in the slides, we have had off-line discussions and intend to submit a new version of the RAW Industrial Requirements document this week.

Lou Berger: It comes in as a DetNet working group item since already adopted by RAW WG item. Just resubmit with the new name.

02) 15:45 20 min Title: Reliable and Available Wireless Architecture

Presenter: Pascal Thubert

Lou Berger: if you adopt the changes Janos proposed, to move some of the functions down to the forwarding layer, it removes the layer violations.

Pascal Thubert: It's already done that way. Janos' picture makes things very clear and I really love that picture and agree with it. However the text has been updated, so shouldn't have the layer violation. Let me know if there is still something left to do.

Janos Farkas: Good text in the document. Agree on many points, e.g., requests and indications going through an API are handled properly. PAREO as such, the combination of ARQ and PREOF is a layer violation. Putting the two together - because they reside at completely different layers - that was the point I was trying to make. Moving the PLR to the forwarding layer makes it easier.

Pascal Thubert: I think I see what you are saying. PAREO achieves a certain set of functions through cooperation between layers. Can view PAREO like packet forwarding, it is something that is being achieved from a set of functions by their cooperation across multiple layers to achieve something global.

Janos Farkas: Then some part of the text maybe needs refinement. Because the definition of PAREO says it is a superset of HARQ and PREOF; creating such a superset itself seems a violation. Could not come up with a good name yet. The overall operation, with requests and indications going up and down to an API, that is proper. Let's define a new stand-alone name.

Pascal Thubert: Let's discuss further off-line.

Kiran Makhijani: Clarification question on terminology. Why change the DetNet Controller Plane to Operational Plane?

Pascal Thubert: It comes from the OAM world. The controller plane includes operation and management. When you look closer at the controller plane, you find what we do in RAW lives in the operational subset. We are merely in the operational piece of the controller plane.

From chat: Janos Farkas:
"It includes the Data Plane and Operational Plane (e.g., OAM) aspects."

Kiran Makhijani: Still not entirely clear. Let's continue on the mailing list (ML).

Pascal Thubert: The changes are not cast in stone. If they are deemed not a good idea, we can roll back.

Lou Berger: Thank you Pascal for the many changes to the document. They have been substantive. Really appreciate that. A little more work to do. Would like to close out the remaining topics on the list in the next month or so. Would like to wrap up before the next meeting.

03) 16:05 10 min Title: RAW multidomain extensions
Presenter: Carlos Jesus Bernardos Cano

Lou Berger: goes back to the prior presentation, where we agreed on the comment/question how does this tie in to DetNet multidomain? Take one of your pictures and instead of having RAW talking to RAW, have RAW talking to wired. Want a picture where you have e2e service that goes over wireless and wired. Filling that in would be very helpful. The point made earlier, there are still things to work out from any DetNet multidomain. In prior work done with PLRs (Points of Local Repair), there was a notion of protection domains (very much analogous to protection graphs or recovery graphs) and an explicit decision not to do recovery across administrative domains, so now worth revisiting. Is it worthwhile to mix protection domains across administrative boundaries? A question for the WG. What does an inter-domain DetNet look like.

Carlos Jesus Bernardos Cano: In another project, dealing with e2e DetNets path across different technologies (wire, wireless, 3GPP, TSN, WiFi) so your point is very relevant. Good to feed some of the work there into the WG.

Lou Berger: as chair, I can say that this is within scope of the WG; as contributor, I can say I would be very interested in that work.

04) 16:15 10 min Title: MIPv6 RAW mobility
Presenter: Carlos Jesus Bernardos Cano

Carlos Jesus Bernardos Cano: Is this potentially relevant to the WG? If so, should it be done here or DMM WG or in cooperation?

Toerless Eckert: If the whole IPv6 mobility is working perfectly fine without DetNet reliability, throughput and latency requirements, then fine to do in DetNet/RAW, but not sure if that is the case. If about basic mobility with mobile IPv6, if that would be something new for RAW networks in the firstplace, then worry that DetNet WG has all the expertise.

Carlos Jesus Bernardos Cano: have been involved in mobility in both WGs, so I have a good overview of both sides of the picture. Probably DetNet WG would be missing some knowledge on the mobility side. But DMM WG might be missing some knowledge on DetNet.

Lou Berger: Is this single or multidomain?

Carlos Jesus Bernardos Cano: Solutions are single domain. One of the options however is specifically positioned for multidomain in the future.

Lou Berger: worry that the solution approach may differ. In a single domain using a PLR approach is viable and multidomain may not be viable.

Janos Farkas: What is wireless specific? Or would it be like wireless or changing between wireless and wireline?

Carlos Jesus Bernardos Cano: Can move between wireline and wireless domains, that would be something else to look at. At least you will have to move from wireless or to wireless. There are things specific to wireless requirements, how to ensure and to compute.

05) 16:25 10 min Title: Extensions to enable wireless reliability and availability in multi-access edge deployments
Presenter: Carlos Jesus Bernardos Cano

Lou Berger: What is specific between the application and the controller that is specific to RAW? Why isn't this just DetNet?

Carlos Jesus Bernardos Cano: Agree, this could be DetNet.

Lou Berger: right now, the only application-facing API we have is a YANG model. You were saying we could use this from the application-facing side. So, if you wanted to explore this use case and look at other alternative ways for an application to talk to a DetNet and that could include a RAW network, that would be something we'd like to hear about. We had a controller plane framework that seems to have stalled. If you have a contribution in this area, it would be worth hearing about. My only suggestion is use the terminology "a DetNet network, e.g., a RAW network". Also, appreciate that you plan for a terminology update.

06) 16:35 10 min Title: Using Deterministic Networks for Industrial Operations and Control
Presenter: Kiran Makhijani

Lou Berger: at last meeting, we asked for interest and didn't get a lot of people who had read it. Let's do a quick poll inside the room to see if it is more than last time. Last time a poll resulted in about a third of the room, which was a small number.

Poll: Who has read this draft?

From chat: Lou Berger: if you've read the ocn draft, please say so here.
From chat: Toerless Eckert, Jinoo Joung, Peng Liu have all read the draft.

Result: Two people in the room (plus 3 on line).

Lou Berger: This is really small. I think this is interesting work, and you are also pushing bounds quite a bit in certain areas. In addition to app flow and DetNet flow you now have a new flow term in the document called a flowlet. If you want to go that way, spend a good amount of time understanding the relationship there before jumping into new formats. You have said your draft is informational but it is defining packet formats. I'm sure there is something interesting here, yet still need to capture the interest of the group.

Toerless Eckhert: One of the interesting feedbacks would be if there are any particular subsets of what was presented that people are interested in? For me, I worry that overall in DetNet we don't have any application-facing APIs. We've got a controller plane and most legacy deployments will expect that everything is done on behalf of application devices through the controller because that's classically how it was done in TSN. Very worried that in DetNet environments we would rather have typical APIs going into applications. So that's the subset of what's been presented in which I was particularly interested.

Lou Berger: That's great because this was a similar conclusion with the prior presentation; there might be something there on the API side. Perhaps collaborate with Carlos Jesus Bernardos Cano on a document contribution there and see if that will generate more interest. Take little steps instead of one massive one.

07) 16:45 15 min Title: Update from enhanced data plane meetings

Presenter: David Black (remote)

Overall questions (from David): Now what do we do? How would/should open meetings help achieve the WG goal to select mechanisms to standardize?

Lou Berger: You are suggesting there might be multiple types of queueing algorithms, so it might be reasonable for the WG to pursue multiple solutions aimed at different classes of either requirements or problems. Help the WG understand the different types of problems you're solving and the different requirements you're addressing, so we can understand if we're comparing apples and oranges, as we go between different queueing solutions.

Toerless Eckert: the attendees of these meetings were largely authors of requirement documents and or some of the proposals. Interested to hear more technical discussion about how can we inherit existing TSN mechanisms, unchanged. While don't think TSN solutions support the large scale, do think they could be applicable to DetNet for other scale networks. Worried about if we can assume TSN ATS can simply be deployed as a DetNet solution without introducing added complexity - that you always have to have a DetNet layer and then also TSN layer 2 solution, as opposed to a TSN ATS queueing layer that directly works on DetNet tuple flows as opposed to the Ethernet flow classifiers in TSN.

Lou Berger: Sounds like you've just volunteered to write a new informational draft on how DetNet can make use of existing TSN queueing mechanisms without running all of TSN.

Toerless Eckert: Need to collaborate with those who know TSN well, and get rid of the duplicate layer of complexity, inherting the subset of TSN that's applicable, which is the ATS queueing but with DetNet classification, hop-by-hop on every DetNet hop.

Lou Berger: What you said is completely valid. Nothing precludes using an existing queueing mechanism whatever wherever you get it with DetNet. We don't have applicability documents or documents that articulate how you do it. Maybe it's a BCP or an Informational doc. We don't have those. Personally agree, it would be informative to those who have not been working in this area for years.

Toerless Eckert: would be great if those working on the TSN side could please volunteer to jointly undertake this task. To return to David's leading question, how do we get to further agreement? Need more time to brainstorm.

Lou Berger: People who are building systems understand how to solve it for their system. However we may be running into the situation where folks have no interest to publish information that helps their competitor build a competing product. It doesn't take a new standard to do this, is just takes some information on how to combine the parts.

Toerless Eckert: On the standards side, it may be as little as exposing all the necessary configuration elements for the controller plane. Having that as a YANG model, a big pain point for others less familiar with YANG models. Want to get DetNet deployable at least so the controller can provision a solution across more than one vendor.

From chat: Janos Farkas: 802.1DC provides a subtraction of TSN QoS tools. So the TSN side is happening
802.1DC includes YANG.

From chat: Lou Berger: I don't understand why a change in classification is needed in TSN to reuse a TSN queuing mechanism.

David Black: Write an outline of a draft that indicates what you think needs to be specified, look at 802.1DC and figure out how much of what you think the problem is the IEEE is in the process of solving.

Janos Farkas: from the TSN side, such a document is happening. It's really mature. It's at IEEE SA ballot, meaning open to review by the entire world, literally. If you want to take a look and comment, you are welcome, please contact Janos.

From chat: Toerless Eckert: Lou, I don't think the TSN documents have the same strict separation queuing/classification as we have. I may be wrong. But if it's not well separated then it may be unclear for implementers what can and what cannot be inherited.

Toerless Eckert: The primary thing that comes to mind is the five or six tuple classification of IP packets as opposed to VLAN and all the layer 2 stuff.

Janos Farkas: That is there in 802.1DC, including IP header fields. I actually have a couple people following your request to provide more information to the DetNet WG. I see value in a corresponding DetNet draft which may be easier to find overall in the Internet and to give pointers and so on.

Lou Berger: Keeping in mind the IETF model for service delivery, we have separated out classification from traffic treament. If they are providing queueing, they don't need to know about how we're doing the classification. That becomes a Detnet prob. Classification, where we identify based on the 5 or 6 tuple, then use one of their queueing mechanisms, doesn't require closer support from them for classification.

Toerless Eckert: What's the minimum set of docs that an implementor of DetNet has to read to get something working. If they need to read all the TSN docs including stuff they don't care about, then that's not a good solution.

Lou Berger: It will be interesting to try to do what David suggested. Put together a document that assesses how much is there already in the IEEE documents. If there are enough changes needed in TSN to support DetNet we can put together a liaison to suggest those changes.
Certainly have had the reverse, where IEEE comes to IETF to request changes in our technology. We cannot change their technology but we can ask for problems that we have identified to be addressed.

Toerless Eckert: If they will have classification on IP addresses or on the 5 tuples that we want, they may already have more than what's needed by pure layer 3 equipment. We may not need all the VLAN stuff. Maybe the queueing can be taken over. We want to evaluate which TSN mechanisms can be used in DetNet.

Lou Berger: Those interested in working on such a document please identify yourselves on the ML!

From chat: Janos Farkas: P802.1DC it is at initial IEEE SA ballot, i.e., the entire document is open to comments. P802.1Qdv is on queueing, at initial stage = Task Group ballot. Contribution driven too

Janos Farkas: Note the ongoing project in Qdv, in early stages, like task group balloting. It's contribution driven. One can contribute individually, if you see the need.

David Black: at this stage, unclear what are the next steps for the open meetings. More discussion at Wednesday meeting, time permitting, and or take to the ML.

17:00 END

DetNet Session 2: WEDNESDAY, November 8, 2023 -- Session I

09:30 - 11:30 CET (09:30 - 11:30 UTC)
Congress Hall 1

Sesson recording:

Slot Start Duration Information

08) 09:30 5 min Title: Intro, WG Status, Draft Status

Presenter: Chairs

Jan 15th a possible week for an interim session for completion of the RAW architecture I-D, should it be needed.

09) 09:35 10 min Title: Requirements for Scaling Deterministic Networks

Presenter: Peng Liu

No quesions/comments.

10) 09:45 10 min Title: Deadline Based Deterministic Forwarding

Presenter: Shaofu Peng

Shaofu Peng: Status of the doc is that the content is basically mature and provides details for implemention, so would like to request WG adoption.

Janos Farkas: As was discussed at the end of the Monday DetNet session, the goal of the open meetings is to come to convergence on which queueing solutions to be selected by the WG, e.g., based on what criteria, so we do not just start adopting.

Lou Berger: Very important to understand which of those solutions are competing or complementary or hitting solving different problem spaces.

Toerless Eckert: Would like to have one example topology with the maximum set of flows, and then from each proposal, have a description on how the calculated latency guaranties from the mechanisms have to be done. Otherwise everybody is presenting it somewhat differently, everybody is assuming a different amount of background that you have. Perhaps we could discuss in the side meetings a kind of standard reference example and then the description for each of the mechanisms. Because I had trouble following the first one here on the slides.

From chat: Lou Berger: for those interested in the queueing topic, please see wiki page for information on past/future open meetings:

From chat: Antoine Fressancourt: From my understanding, before meeting we need to make sure what are the WG expectations about the evaluation of the queueing mechanisms, e.g. do we need to work on a similar example as Toerless suggested ?

From chat: Antoine Fressancourt: If we position our solutions relatively to others, do we use the set of requirements from the large scale detnet requirement draft? in which version?

From chat: David Black: Forthcoming -05 contains useful update to latency/jitter scaling requirements.

From chat: Antoine Fressancourt: Exactly.

From chat: Jinoo Joung: Evaluation criteria should include the explicit mathematical expression of E2E latency and jitter bounds.. without these, we depend only on the author's self-evaluation.

From chat: Antoine Fressancourt: Then we need a harmonized notation.

From chat: Antoine Fressancourt: Bearing in mind that the mechanisms discussed act on a variable set of system parameters, so maybe even with a harmonized notation we will realize that the equation we get for end to end delay are hard to compare

From chat: Jinoo Joung: Unifying notations would be not a problem. There are various approaches calculating the bounds, but the results for different queuing schemes can be easily compared. That's what happens in the research community.

From chat: Yizhou Li: The paper and/or implementation reports could be strong evidences to support author's self-evaluation. So it would be good to show those if any. It could help to converge.

From chat: Rodney Van Meter: One of our students has been doing fantastic work on very low latency "glass-to-glass" video transmission via UDP/IP generated using FPGAs. One of the use cases is flying racing drones (which can go 200km/h, so control lag matters!), another is real-time collaborative music performance over the net. They have done glass-to-IP-packet latencies of far less than a single frame time at the camera end; when doing uncompressed 8K video, latency is a single scan line. Bandwidths over WLAN obviously aren't high enough for that, so they use intraframe compression for flying the drones.
He doesn't have much in English yet, but one paper in Japanese (subscription required, unfortunately):

From chat: Rodeny Van Meter: Here's a paper of his on the racing drone stuff. Paper is open access, but in Japanese:

11) 09:55 10 min Title: Timeslot Queueing and Forwarding Mechanism

Presenter: Shaofu Peng

Toerless Eckert: if we would go forward with this, this is a single draft describing TCP/IP. Two layers that should be separated from each other. That's basically what my proposal is about. This orchestration thingy is something that we should put on the edge and consider a different layer from the hop-by-ohp forwarding.

12) 10:05 10 min Title: Latency Guarantee with Stateless Fair Queuing (C-SCORE)

Presenter: Jinoo Joung

13) 10:15 10 min Title: Flow interleaving for scaling detnet data planes with minimal end-to-end latency and large number of flows

Presenter: Toerless Eckert

No questions/comments.

14) 10:25 10 min Title: Enhanced Use cases for Scaling Deterministic Networks

Presenter: Junfeng Zhao

Abdussalam Baryun: Regarding the data, the bandwidth and the delays mentioned, are they real experience numbers (you mentioned 20 milliseconds and less than 15-20 milliseconds) that have been tested or are they statistical? Were these figures tested already, or statistical?

Toerless Eckert: I think I have seen similar sources from other references, please add reference materials. These are a very good set of applications and use cases not seen in the original draft.

Junfeng Zhao: Originally from 3GPP and also research reports.

Lou Berger: I view this as an addition or expansion on the Use Cases RFC, which has a section that summarizes common themes, I think chapter 11 or 12. Have you seen any impact to that section based on these new cases? That would be very helpful to note, especially if there is impact.

Junfeng Zhao: Can add that in next version.

Lou Berger: That would be really helpful because your summary slide helped us understand how the different use cases need to be supported by our mechanisms.

Abdussalam Baryun: Please provide pointers to the references you used.

15) 10:35 15 min (with 16) Title: Differentiated DetNet QoS for Deterministic Services

Presenter: Quan Xiong

Kiran Makhijani: Related to my work, do you have any requirement re interface from end systems to DetNet? Or is it something you would like to consider? Is it something worth considering. It is a question for the group, how applications or end systems will interface with DetNet?

Quan Xiong: We didn't cover that yet. However, the use case you proposed has been covered. And you proposed in the mailing list the applications and their topologies and SLA requirements. You also mentioned that flows within a single application will have different bounded latency requirements. These two aspects have been covered by DetNet queue requirments, but not the interface.

Lou Berger: The way I read this document, you are trying to come up with specific classes for use in DetNet and that is quite different from a class-based service, which is very different from what's in the DetNet architecture, where we have per-flow service, which can then be aggregated. It looks like your moving away from that, is that your intent?

Quan Xiong: Yes

Lou Berger: You really have to talk about the impact to the architecture as that is a fundamental change. I'm not sure it is DetNet.

Lou Berger: I'd like to better understand the motivation. You could have a solution that has virtual output queues and virtual queueing is something that's been in industry for many years and gotten us to 64K queues per interface on some implementations. So underneath, being able to go from individual flows into some hierarchy of queueing, that all makes sense in an implementation, but what I'm reading here is a fundamental change in our service architecture.

Quan Xiong: Two main motivations: want to resolve scalability issues (because aggregate flows have scalability issues) and want to provide fine-grained traffic scheduling and resource management.

Lou Berger: How are you doing fine-grain when you are doing class-based? It would be very useful to understand where you see things are breaking down with aggregation and where the service model needs to change. Will look for further explanation of the fine-grain concept in the next version.

Tianji Jiang: In 3GPP community already has a project done, to make 5GS a logical DetNet node. It's like a black box, but within, there are so many flows requiring different queueing aspects, like per hop behaviors, per hop delay, bandwidth, jitter. And there are liaisons between 3GPP and this group. That is requiring per hop behavior plus the fine-grain things you mentioned.

Lou Berger: Sounds to me that you just said that you're using the virtual node concept that we've had for a long time also in the IETF.

Tianji Jiang: Node is per behavior, but the granularity actually is per flow, because in 5G system you have PDU session.

Lou Berger: Please send more to the list so everyone can understand that.

16) Title: Traffic Engineering Extensions for Enhanced DetNet

Presenter: Quan Xiong

Toerless Eckert: It will be hard to validate or invalidate the claims you propose, given that DetNet already has a variety of mechanisms it already can do. All the past work is around bounded latency, though you don't have to have banded latency to call something DetNet. You may have PREOF with or without bandwidth guarantees. We have various subsets. Maybe you are giving new names to subsets of DetNet?

Lou Berger: We could look at this as just a form of aggregation. Instead of doing full 6 tuple classification, we do 1 tuple, on DSCP. This is an applicability of DetNet, as opposed to a change of DetNet. Disagreeing with my earlier comments! Thank you for coming up with 3GPP because that changed my perspecitve.

Toerless Eckert: There was a side meeting about traffic engineering. You can do all the complex things together, which we can also do in DetNet, but we don't have to do. There is value for DetNet marketing purposes, to have better understanding about the fact that different types of applications and use cases only need to use different subsets of DetNet capabilities and maybe even simple examples of that. The underlying mechanisms would be DetNet plus what we want to do for large-scale DetNets.

Abdussalam Baryun: In this presentation and the previous one, there is a difference between service class or traffic flow. When using TE, we are usually focused on the flow and not the class.

Lou Berger: Basically, he is looking for information on the description of class usage vs per flow usage. If we look at that as a form of aggregation and describe that in that context, that would be helpful.

17) 10:50 10 min Title: Jitter Reduction Mechanism for DetNet

Presenter: Rubing Liu

Lou Berger: Are you assuming the latency is accumulated in the control plane or in the data packets themselves?

Rubing Liu: The control plane to calculate the delay.

Lou Berger: Do you see that we need to change anything then to make this solution works, other than the control plane?

Rubing Liu: See the Reference Architecture backup slide, which shows the control plane to calculate the delay path.

Lou Berger: In this document you'd like to see a control plane solution to calculate latency across the network or to propagate latency across the network?

Rubin Liu: Yes.

Tianji Jiang: Seeking advice about your explanation since you are going to accumulate the delay, to use the edge node to do all the work. Back to your the previous discussion, for a logical DetNet node from the 5GS side. It's going to join the IP domain as a DetNet node, so it's looking for per hop behavior. If that per hop node has already exceeded its SLA, will you discard work toward the end of the edge, or do you prefer to do the work on the DetNet node itself?
If the system cannot satisfy the SLA, where are you going to drop the packets?

Janos Farkas: Let's continue on the ML to better understand the proposed solution and the question raised.

11:00 END