RTGWG Interim Chairs: Chris Bowers Jeff Tantsura Date: Thursday 2021-06-03 Time: 14:00 UTC https://datatracker.ietf.org/meeting/interim-2021-rtgwg-01/session/rtgwg There were some interesting discussions on the chat, and they were copied to the end of the minutes. 1. Agenda Bashing - 10 mins AD: Martin Vigoureux Chairs: Jeff Tantsura, Chris Bowers 2. Problem Statement - 30 mins Gyan Mishra Joel Halpern: this question is likely to come up across a number of slides and I would appreciate you if you can give me a useful answer now. First, you refer to finer grained QoS we have a lot of QoS tools including some which are finer grained, like the various use of SR with resource reservation, like frankly the use of MPLS with RSVP TE. you're obviously looking for something finer than that. We had tried something finer than that it was called RSVP and Int-serv, and it was really well thought out and carefully planned and it could meet all the service requirements, and was proved to be completely undeployable because the scaling simply meant that the network core had to scale as the number of users in terms of a lot of ts state, which was a disaster. How does this avoid that? How does this whole space avoid that conceptual problem without simply duplicating what we already have and existing tools. Gyan: This is part of building solution and really working with the WG to actually help define and architect the solution. Shuping will go through the solution. You have a valid point, the existing solutions have worked well for 20-30 years. We have lots of tools, the big change now with 5G is network slicing, how many slices can we have? it's a huge paradigm shift. we don't have all the answers, but we hope APN can help to solve and fill that gap. No doubt MPLS RSVP TE has worked well, also segment routing. Just now with slicing, we don't know how many services each slice can carry, and how many slices. APN is to fill the gap and provide granularity, extend the capability to provide the SLA. THe point is duly noted, and we should learn from the problems and actually feed that back into the solution. We learn from the mistakes and build something that can work. Joel: I didn't hear an answer to the question, but I'll let you continue because there's no point in belaboring it. Linda: In 5g, for example edge computing, the network becomes much smaller, we're talking about some kind of closed loop networks, the servers are very close to the cell tower. So we're talking about very small vertical networks which is not going to the public Internet. So the number of nodes involved is much smaller, and it's all under control. Joel: none of the problem statement says that it's restricted to that problem. That might be an interesting problem to solve. As a restricted case but that's not what any of the problem statements I've seen say in terms of restriction of the goal. Linda: The problem statement has some limitation on the scope of the network. Just ask a question to Joe, do you think it's better to limit the size of the network into some kind of vertical of special purpose network, where the number of nodes involved is under control. Joel: I'm not claiming to tell the folks who are pushing for APN what problem they want to solve, what I heard from Gyan that he wanted to solve a much broader problem, which is fine, I just want to understand why how what the implications are. If they want to, if everybody involved wants to solve a smaller problem that's more likely to be effective, but that's not my call. Gyan: The scope for APN at this point is closed domain. It could be multiple AS, but a closed domain, within an operator. Within a mobile core, or data centers. Jeff T: please add domain specification for APN. Joel: yes, that will help. Gyan: we’ll add it. There have been discussions in the context of APN. Shuping: I’ll also cover it in the solution discussion. Kireeti: it will also be useful to get some sense of the scale that you're looking at, because there are millions of applications, and millions of simultaneous application depending on the network, even in a restricted domain you'll have hundreds, if not thousands. So what the scale is, I think would be an interesting and important factor. Going back to Joel's question. And also the relation between APN, and network slicing. so within a slice, you need APN, what would the scale be within a slice? or they're completely separate concepts. Gyan: so your question is the mapping of application ID to slicing, is it 1:1 or 1:n mapping? Kireeti: exactly. Gyan: that's something APN framework will document. Ramakrishnan: From APN point of view, are we going to differentiate only the application specific packets at this kind of control point of differentiation, which be be running on the overlay. Gyan: it will be on the overly where the application flow that's being instantiated. The marking is done on the outer header for the application. The SD WAN controller will see the marking and provide the appropriate application based routing across the fabric. Shuping: this is for operator owned SD-WAN, it’s for both overlay and underlay. Linda: CPE1 can be located in a shopping mall or some remote places, the controller can mark the APN ID so when go into the operators domain they can perform more stringent policies for the traffic from specific location. That's where I see that application ID can be very useful. Ramakrishnan: There will be lots of applications, how we are going to make sure a specific application to a specific mapping? are we going to mark a group of applications to one service? Linda: not all applications will be marked. only specific application for premium service. Gyan: The marking is more intelligent only for certain services. Not every flow needs it, only interesting traffic gets mapped. The controller will do the mapping according to policy. Michael Richardson: Can you explain how App X is distinguished from Y by CPE1? Gyan: This is similar to MPLS forwarding class. Michael Richardson: So it's distinguished by the destination of the traffic. Saumya: is it going to be agnostic to the fabric? Gyan: it’s the goal to make it fabric or data plane agnostic. It will be covered in the solution. Shuping: the APN attribute is carried in the tunnel, depends on the tunnel encapsulation. I’ll cover it in the solution. Jeff T: Please continue, and we may continue the discussion in solution part. Michael Richardson: My experience with BNG is that traffic across the metro network is all PPPoE, so I don't think you get to do anything to look deeper into the packet as to destinations. The home router, I don't think it speaks APN. so I don't how this works, either fairly invasive deep packet inspection or participation of the home router gateway. Zhenbin: in BNG, the PPPoE session will get the user information. There is a QinQ encapsulation with VLAN info, also service vlan to identify the service, such as IPTv. The encapsulated packet will use different tunnel, to map customer vlan or service vlan to APN ID and parameter, and use this info to steer traffic. Chris Bower: with respect to the previous scenario. We’ll circle back to the given scenarios. the proposed solution may shed more lights. Feel free to come back if you're not happy with the explanation. we're going to balance between drilling in deep right here and then drilling in deeper later on. Greg Mirsky: usually the network between RG and BNG is referred to as an access network, and BNG is on the edge of the domain. Well, at least that's how the Broadband Forum presents it. Another, when you envision the phone, Connecting to RG. What type of a service you think that phone provides? this UE provides to subscriber? is it 5G? Gyan: My guess would be voice over IP, not 5G. Greg: that's where it becomes interesting. If it's no different from, for example PC, then we can simplify the use case and take the phone out. But if it's 5G, now we're dealing with wireline and wireless convergence. Gyan: that's a good point. The phone makes it more complex. Greg: why RG is connecting to BNG over metro network? Gyan: it could be anything over your service provider to BNG. Greg: I would imagine to be connected to Metro network, the device will have to support much more applications. And that probably would not be economical, and that's why usually RG aggregated by BNG devices that are at between access network and metro network. so basically the BNG creates their edge of Metro network by aggregating subscribers over the access network. So that's the RG devices much simpler and more economical. ChongFeng: agree with Greg, BNG should be at the edge of the metro network, that means that metro network should be the upstream of BNG. The figure should be changed. Gyan: Yes, I agree. The picture needs to be redrawn. Linda: For the tunnels in this slide, is the mobile transport where the APN marking is? or GTP-U tunnel? Zhenbin: outside the GTP-U tunnel, there will be another tunnel to traverse the mobile transport network. Linda: now we have two layers, is it possible they can collapse into one? can it be done at IETF? Question to AD. Joel: are you asking IETF to modify GTP? we don't own GTP. Alvaro: Joel is right. Let’s not talk about that here in this meeting. We've been beaten the scenarios more then we need to at this point. This is not BOF, we’re not going to discuss what IETF can do and can't do. It will be discussed at the BOF if approved. Jeff T: let’s continue. Peng Liu: just to share something about EC. As EC is being deployed, and there are several successful commercialization case. And also, at the edge computing has a key function in Edge network device, so we can match the work together. The slide shows the function can be deployed at the edge network device, so traffic classification and some application requirements can be done at the edge, so new applications such as cloud gaming and remote control, which has been introduced in the edge. Chongfeng Xie: comments about this slide. Traffic should go directly to MEC, not through UPF. Linda: earlier you said that the APN services is not for every applications. So this app at edge, they get some kind of indication on what type of service or applications, they actually marking? or there should be a controller? Gyan: there might be a controller. Shuping will talk more in the solution. 3. Solution Discussion - 45-70 mins Shuping Peng / Robin Li Linda: regarding the APN ID and user ID, are they eight bits or longer? Shuping: we didn’t specify the length, it’s a question to be discussed. we'd like to get people's opinion. Haoyu: In terms of extensibility, TLV architecture provides better extensibility, also friendly to hardware. For parsing, TLV is proper. Shuping: Thank you. Chongfeng: Current design include the APN ID and parameters, and I agree with it. For the architecture, on the architecture slide, SRv6 should be part of IPv6, not separate. TianRan: I agree with the way you showed here to produce one common header structure and then apply to different transport. This looks reasonable, there are existing examples, like iOAM. Giuseppe Fioccola: for alternate marking we also use a similar approach with extension header. I want to point out that RFC 8799 introduced the concept of controlled domain, refining IPv6 for reasons such as support of options and securities. It's better to make it in controlled domain. I think this direction is right. Linda: if we’re talking about overlay tunnels, like VxLan or GRE, they have fixed number of bits for keys or real ID. does that mean the application ID condensed into a fixed number, like 32? Shuping: Yes, that's something we need to consider. How shall we consider the encapsulations of the various data planes, and it should be somehow adaptive. This is a topic we need to investigate. Haoyu: in MPLS WG, there is discussion about how to provide generic mechanism to support in Network Services, and other meta data. So, I can see one proposal we are talking about adding the extension headers to MPLS label after the late MPLS label stack. so that's a perfect place to have this APN attributes. So I think that definitely helps this. Shuping: it will be good if you could provide such information in the list. Haoyu: yes, I will. Jingrong Xie: it might worth considering that BEER tunnel also carry APN attribute. Shuping: yes, that could be included, a topic to investigate. where do you suggest we put the APN attribute in the multicast scenario? Jingrong: BEER tunnel. Jeff T: that question should be addressed in more detail in BoF. Gyan: when used with a controller, will the five tuple header information be kept by the controller? then the actual APN ID are basically like an entropy label or hash that goes into the APN ID attribute field? Shuping: for the matching relationship between the existing information, like the five tuple in the packet header and the APN attribute will be stored in the controller. for the APN attribute,what will be included, like the APN ID and maybe the parameters will also be in the controller. it's like the network device will get the rule to construct the APN attribute. Gyan: the APN ID will be encoded into the packet, but other info like delay, jitter etc. will be stored by the controller, I guess. Shuping: yes, that's part of the design, and is one of the topic we'll look into. Linda: will you consider BGP-Flowspec? use the flowspec mapping to the APN ID. Shuping: possible. Kausik Majumdar: if APN ID is based on the five tuple, how do you determine this APN attribute is associated with low latency versus high bandwidth? Shuping: that's enforced by the controller. Kausik Majumdar: you data only carries APN ID, you're trying to apply certain policy, how do you map? Shuping: that's based on the APN ID and parameters. When the packet reaches the head end, it will find the SR policy that satisfy its requirements, and the head end will do the traffic steering. Kausik Majumdar: To do that, when the APN attribute coming from the APN edge to the head end, you need to know this APN attribute is for the low latency or kind of a high bandwidth, then only you can apply certain policy. If the APN ID doesn't provide that characteristics, how can you apply? Gyan: The controller may have such information. but if the actual information is in the data packet, would the network node actually know how to interpret that information? or only the controller doing the interpretation? Kausik Majumdar: exactly. that's what I'm trying to get in. Zhenbin: there are two possible ways. First, such information can be encapsulated in the APN packet, when arrive at the head end, traffic will be steered according to the APN ID. THe policy can be applied from controller to head end. Second way, APN ID has group ID, which means low latency or high bandwidth, so the controller can also apply the policy matching this ID to satisfy the requirements. Gyan: that makes sense. The solution has to be controller based. The controller has the brain, the node itself doesn't understand the ID itself. does that sound right? Yingzhen: Just want to mention that there are solutions in IGP that can be used to propagate non routing informations using a separate instance, transport instance, without interference with the regular routing calculation. I'll send some detail information to the list. Haisheng Yu: how is the APN ID defined? there might be two ways, one by data time and version, another is to define by requirement. Robin: for different operators, there might be different rules. some might use team, some may use low latency, since it's local it's operator's choice. it doesn't need to be standardized. Chris Bower: it will be helpful for you clarify the forwarding state of the protocol. There are discussion of control plane, but not much forwarding plane. The discussion of control plane should be on top of data plane, should it just be SR TE? If you're leveraging pre existing mechanisms okay which pregnancy mechanisms are you gonna leverage, etc. From chat history: from Stewart Bryant to Everyone: 7:18 AM Joel: every idea needs to live int its own time, Previously it was not worth the effort. We live in new times with AI based network tuning and a requirement for more bespoke network service offering from Stefano Previdi to Everyone: 7:20 AM isn't rsvp-te stateful and APN stateless ? from Zhenbin Li to Everyone: 7:20 AM RSVP-TE has challenges in the scalability. So it is conservative to deploy the TE solution. Now with SR, there is SR with better scalability and more TE path will deployed. from Shuping Peng to Everyone: 7:21 AM There is the mismatch between the highly demanding service requirements (e.g. 5G) and the evolving network capabilities (e.g. SR policies, network slices), and APN fits in that gap. from Michael Richardson to Everyone: 7:42 AM Hi chairs, if we can't get the problem statement understood, then the technical parts don't matter. from Shuping Peng to Everyone: 7:44 AM hi Michael, please put your question here in the Chat box, so we could save some time. Thank you! from Michael Richardson to Everyone: 7:48 AM Chris, that's great but the speaker did tell us to ask questions as we go. from saumya to Everyone: 7:51 AM In a SD-WAN (campus) deployment...there can be alternative paths across multiple operator domains....MPLS, LTE, Vxlan....will this mandate multiple operators domains mapping to single APN domain.... ? from Zhenbin Li to Everyone: 7:51 AM Hi Matin, the tunnel is not between the RG and the BNG. it is in the metro network. The PEs is not drawn in the figure. from Zhenbin Li to Everyone: 7:52 AM The tunnel is used beween the PEs of the metro network can be LDP/RSVP-TE/SR tunnel.s from Zhenbin Li to Everyone: 7:55 AM @Greg in some operator's network, the BNG is deployed on the higher place. from Zhenbin Li to Everyone: 7:56 AM I agree that noe the operators try to change to deploy the BNG to the lower place. from Michael Richardson to Everyone: 7:56 AM If the BNG is moved, then this scenario is the same as the previous one. This diagram is actually useful and represents 90% of DSL deployment in Canada. from Linda Dunbar to Everyone: 7:57 AM is the Tunnel on page 10 inside the GTP-u tunnel? from Zhenbin Li to Everyone: 7:57 AM tunnel is outside the GTP-u from Routing Area Working Group to Everyone: 8:05 AM @Robin - if you feel a better clarification could be provided - please send it to rtgwg list from Zhenbin Li to Everyone: 8:06 AM Sure from Greg Mirsky Guest to Everyone: 8:06 AM @Zhenbin, it is very interesting scenario. Could you kindly share more information (with BBF hat on)? from Zhenbin Li to Everyone: 8:10 AM @Greg I am very glad to share the information for more discussion. Thanks. rtgwg-01-rtgwg from Routing Area Working Group to Everyone: 8:09 AM Yes - recording will be shared from Zhenbin Li to Everyone: 8:10 AM @Greg I am very glad to share the information for more discussion. Thanks. from Gyan Mishra to Everyone: 8:15 AM Thanks everyone for your comments on the APN problem statement discussion! from Linda Dunbar to Everyone: 8:19 AM can we have both fixed APN ID and TLV based? from Linda Dunbar to Everyone: 8:20 AM Fixed APN ID structure would be useful for traditional tunnels, such as VxLAN or GRE from Haoyu to Everyone: 8:22 AM But it's not scalable and hard to change from saumya to Everyone: 8:22 AM SD-WAN Gateways have their own proprietary security solutions....how does this solution design aligns with that ? from Zhenbin Li to Everyone: 8:25 AM Hi Saumya, the solution is to map the 5-tuple information to the APN ID and according to the APN ID steer traffic. I think the 5-tuple informaiton is available even if the the security solution exists. from Dhruv Dhody to Everyone: 8:29 AM Regarding the use of PCEP in Slide 9, i think the PCE central control architecture RFC 8283 would be a good fit with the APN attributes as central controller instructions! from saumya to Everyone: 8:29 AM It's possible that security solution treats 5-tuple differently for traffic qualification than what APN ID implies from Routing Area Working Group to Everyone: 8:30 AM BGP-FS could potentially be used from Zhenbin Li to Everyone: 8:33 AM @Saumay the traffic is steered in the WAN network (underlay network). It is not in the SD-WAN network (overlay network). For the WAN network, it should apply the security solution in general. from Kentaro Ebisawa to Everyone: 8:37 AM Assuming controller needs to configure policy on all nodes in slide 9, what would be the benefit compaired to SRv6/SR-MPLS + FlexAlgo which only requires controller to set policy on edge nodes? from Daniel Bernier to Everyone: 8:37 AM +1 from Greg Mirsky Guest to Everyone: 8:39 AM Apologize if I've missed it, what device is the edge of APN - RG or BNG? from Zhenbin Li to Everyone: 8:41 AM PE of the Metro network. It is not drawn in the piciture. We will refine the figure. from Zhenbin Li to Everyone: 8:41 AM RG is just to tag the QinQ information. from Greg Mirsky Guest to Everyone: 8:44 AM @Robin, thank you for the clarification. from Routing Area Working Group to Everyone: 8:45 AM +1 Chris (as contributor) from Zhenbin Li to Everyone: 8:48 AM Now steer traffic to SRv6/SR-MPLS + FlexAglo is depending on the VPN or the prefix. If finer granuliarty is needed, 5-tuple is always used. But there is challenge which mentioned in the problem statement. from Shuping Peng to Everyone: 8:50 AM @Kentaro, that is because APN attribute is not only used for one single network service from Kentaro Ebisawa to Everyone: 8:51 AM @shuping I see. Thanks for the clarification.