Date: Thursday, November 11, 2021 - Session I
Time: 7:00-9:00 EST - 12:00-14:00 UTC – 120 mins
Chairs: J/E/M
Jianfei (Jeffrey) He jefhe@foxmail.com
Eve Schooler eve.m.schooler@intel.com
Marie-Jose Montpetit marie@mjmontpetit.com
Join via Meetecho: https://meetings.conf.meetecho.com/ietf112/?group=coinrg&short=&item=1
Live minutes: https://codimd.ietf.org/notes-ietf-112-coinrg
Chat archived at: https://jabber.ietf.org/jabber/logs/coinrg/2021-11-11.html
Materials: https://datatracker.ietf.org/meeting/112/session/coinrg
Video: https://www.youtube.com/watch?v=H4vPRkqyXYo
Note takers: Jianfei He, Eve Schooler
Paper: https://ccronline.sigcomm.org/2021/revitalizing-the-public-internet-by-making-it-extensible/
Cloud/content providers building private IP-based networks with many PoPs.
These PoPs apply in-network services: Flow termination, Caching, Load balancing and have significant impact on latency/reliability,
Public internet can’t do it since it is not part of the architecture. We need to build this in-network processing into a coherent architecture.
Four Questions and anwsers:
(1) Why is the Internet so hard to change? L3’s Dual Role (Allows all L2 networks to interconnect, Must support all application requirements), Prevents Change
(2) How can we overcome this barrier? Extensible Internet (EI). Introduce a new network layer above L3: Service layer (SL) or L3.5 . SL offers new in-network services to hosts, is merely the architecturally coherent version of hyperscaler private networks. L3 is still the interface to L2 networks, but L3.5 is the interface to L4.
(3) What would this new Internet look like? SNs are servers deployed at network “edge”. Every host associated with a service node. Typical communication pattern: Source → SN → SN → Destination. Key: All Services In Software. Standards: Open-source code, not specs. 3 necessary software components on SNs: Standardized service modules; Standardized execution environment (WORA); Open-source runtime/orchestration available. Some Potential in-network Services.
Basic services: Flow termination, caching, load balancing etc. Support for other delivery models: Multicast, pub/sub, redirection, QoS etc. Support for security and privacy: DDoS, attestation, oDNS-like, Tor-like etc.
Incorporate other frameworks: Istio, OPA, telemetry,…
Support for new architectures: ICN designs, other radical redesigns,…
(4) Given the failures, why is now different? Nothing changes for hosts/domains if they choose not to use EI. Compute resources at the edge are already there. There is an alternative for public internet now: the private cloud networks. The clouds could easily reduce the public Internet to primarily the last mile. EI Based On Simple Conjecture.
Current Status: Prototype of SN done.
Future Plans: Deploy on FABRIC and other testbeds, Engage research community (via FABRIC), Continue discussions with industry.
Q(Adrian Farrel): will the transport protocol be end-to-end (e2e) or be terminated and restarted at service nodes?
A: there will be an e2e reliability layer. But it is an open question if the IP pipe is end2end. It is also not mandated by the architecture.
Q(Adrian Farrel): we should take encryption into account, whether the service nodes are able to access the encrpted packet. Layer 4.5 instead of Layer 3.5?
A: There will be end2end encrption, but we need to have an option for endpoints to say: here is the parts we are willing to let the intermediate nodes see. The flexibility is left to the applications themselves.
Q(Dirk Trossen): Is it a barrier to require a standardized public services?
A: When you support the execution enviroment, it is simple to implement these services. But if you don’t make it uniform, then the application developped can assume these public internet services are available. That is a critical part of the proposal.
Q(Dirk Trossen): Somebody can load the app to my execution enviroment and run it.
A: it will be the rule to require the service nodes to update to new published services.
Q(Dave Oran): Is service composition model considered?
A: We specifically do not allow users to arbitarily compose services, because we think it is impossible to have a genenral rule how to compose.
Q(Dave Oran): how about internal links? Any thoughts on service chaining or VNFs?
A: In current implementation, we merge several pieces of code when an integration of service components are required. We are not doing NFV chaining.
Q(??): what’s the relation between EI and sfc/sd-wan?
A: for service chaining, it’s up to you to decide to go through various boxes. Service chaining doesn’t give you any global serivce definition.
Q(??): how do the EI nodes communicate with each others?
A: will have some discovery process for service nodes. For control plane, there will be wide variety.
From the chat window:
[12:14:48] <Dirk Kutscher> To some extent, QUIC and MASQUE are looking a whole lot like L3.5 already.
[12:16:40] <Adrian Farrel> But isn’t QUIC sort of Layer 4.2?
[12:17:48] <Adrian Farrel> And that leads me to a question for Scott about whether transport protocols are being run end-to-end or between SNs. In that case, isn’t he talking about something above L4 (possibly L5)?
[12:18:08] <Dirk Kutscher> Yeah, the mapping is not perfect, but MASQUE is about different services types within QUIC connections, with support for intermediaries.
[12:18:09] <David Oran> q: what’s the model for composing multiple services? is this like the existing service chaining and/or VNFs?
[12:18:12] <Spencer Dawkins> When we chartered QUIC, the IAB had been talking about a “UDP-based layer 3.5” UNDER QUIC. That was SPUD/PLUS.
[12:18:47] <Spencer Dawkins> We didn’t charter that work, so the layer 3.5 got wrapped into QUIC.
[12:19:18] <Lou Berger> netflix openconnect seems like a customized version of 3.5 more than quic
[12:20:28] <Dirk Trossen> Q: If SN is a L3.5 service overlay, why demanding that a single SN must run ‘agreed’ public services? Isn’t that a barrier to entry by party X to roll out edge-based SNs with a new service?
[12:20:55] <Dirk Trossen> meaning, may we just embolden the ‘big player’ market as of now by demanding ‘big SNs’?
[12:21:18] <Colin Perkins> And how does/should this programmable service node model be pushed down into lower layers?
[12:22:58] <Cedric Westphal> what are the incentives to deploy this? Hyperscalers do it as part of their business model. But how running these SNs is monetized? Who bears the cost?
[12:23:21] <Joerg Ott> Right, how to best benefit from L2/L3 capabilities e.g. for efficiency rather than just running all as p2p overlay? Fan-out for pub/sub, for example
[12:23:45] <David Oliver> open source with reproducible builds on the Sys provides a trust model I would think
[12:24:30] <Cheng Li> good question
[12:25:06] <Dirk Kutscher> What if shrinking the Internet is the economically reasonable way to go for hyperscalers seeking to establish/defend monopolies? What are the incentives for a disruptive change like this?
[12:25:13] <Dirk Trossen> @cedric connects to my question since if the barrier is to do all public services, who can afford it?
[12:25:59] <Joerg Ott> Or, in more general terms, when we find that E-SN-SN-E isn’t quite enough, would we move towards multiple SNs in the end?
[12:26:06] <Dirk Trossen> @DirkK getting my novel service out quickly may be an incentive but I’m struggling with the “SN/public services” model
[12:28:10] <David Oliver> SNs that do not run “the common services” encourages more balkanization, doesn’t it?
[12:28:50] <David Oliver> SNs with common services, running open source software with reproducible builds provides an access point trust model
[12:28:55] <Stewart Bryant> By definition you cannot make it uniform. because as soon as it is deployed someone will need to add a new feature
[12:30:08] <Stewart Bryant> What happens when you hit a hardware limit?
[12:30:49] <Luigi Iannone> running open source software does not prevent customization which prevents trust based solely on which software is used
[12:31:09] <Dirk Trossen> @David but it goes back to DirkK’s question on the incentive for deploying SNs. Wouldn’t run my own ‘balkanized SN’ be much better for an existing monopolist?
[12:32:07] <Dirk Trossen> Similar to the previous occasion of Scott’s talk, I think the struggle here is about the economic incentive model that would make this work to counter the centralization we observe today
[12:32:36] <David Oliver> @dirk “the problem of centralization” - often discussed here at IETF - is certainly a problem since HyperScaler A can just say “you guys wanna be on MY version of the Internet”
[12:34:04] <Dirk Trossen> @david My point exactly. So how can you force a hyperscaler to effectively open up their “SNs” today to the agreed public services? It sounds like a governance/regulation issue more than a technical one.
[12:34:12] <David Oliver> @dirk possible answer: I think we are going to struggle in the future with the balkanization of the Internet by NATION STATES. “governments vs companies” might provide that economic incentive. That is the companies will want to live in the most places possible, so if this L3.5 promises that, it’s better
[12:36:33] <Spencer Dawkins> This seems like a REALLY important discussion to have on a mailing list. This is fundamental enough to justify more public conversation.
[12:36:43] <Dirk Trossen> Hmm, listening to Geoff Huston’s talk yesterday, I was not under the impression that nation states are the problem really. Saw “CDN” on his slides
[12:37:28] <Dirk Trossen> @Spencer +1 on the need for a forum for this kind of work
[12:39:07] <Spencer Dawkins> @Dirk - it’s certainly relevant to COINRG, but not limited to COINRG. Is it reasonable to start that discussion in COINRG, and see where it goes?
[12:39:33] <Marie-Jose Montpetit> @Scott good idea :)
[12:40:13] <Marie-Jose Montpetit> @Spencer ok to start the discussion
[12:40:24] <Luigi Iannone> After the first presentation Scott made there was some exchange on https://www.jiscmail.ac.uk/cgi-bin/webadmin?A0=SARAH
[12:41:10] <Luigi Iannone> But that is an external ML
[12:41:45] <Scott Shenker> Chat is not a great place to fully describe the incentives, but here is one case. Currently the clouds spend $$$ for the large user-facing networks, and the edge nodes deploy largely generic services. Why wouldn’t they want a common infrastructure that deployed these generic services at the edge, and focus on the more proprietary backend services? In addition, once this becomes the overall architecture, hosts can select services, which enables a much wider spectrum of in-network services.
[12:42:46] <Scott Shenker> We’ve talked to people at all the major clouds, and they think such a story makes sense.
[12:43:32] <Colin Perkins> This RG has a mailing list, that seems like a reasonable venue to continue discussion
[12:43:37] <Dirk Kutscher> Many hyperscalers have their overlays from the provider edge to their regional data centers. I am not sure, they urgently need something else.
[12:45:24] <Scott Shenker> They don’t need it “urgently” but (i) they’d save money and (ii) this opens the door to a much wider array of in-network services that would improve security, privacy, and application performance.
[12:45:28] <Dirk Kutscher> … although it would clearly be desirable for any newcomers that want to deploy their service in similar ways…
[12:45:39] <Spencer Dawkins> @Scott - is there an obvious community that we could invite to participate in a discussion on the COINRG mailing list? It’s probably best that we don’t Balkanize discussion of ways to respond to Balkanization, I think.
[12:47:00] <Scott Shenker> I don’t know what the right communities are, but that’s a good thought.
[12:47:43] <Dirk Trossen> @DirkK if as a newcomer, I will need to support the public services, isn’t that a burden?
[12:49:54] <Dirk Kutscher> Depends on the specific design, but you could imagine that your service needs a special in-network-computing system on-top of the network of execution environments, and that EI would enable you to roll out your “slice”.
[12:51:40] <Dirk Trossen> Right, but this is more about a general execution environment, which I agree with. But the need to provide a defined set of public services as part of your roll-out sounds like an investment to be done as part of your market entry.
[12:55:39] <Cedric Westphal> @DirkT: you need to roll out a box, and you also need some critical mass of SN so that src-SN-SN-dest path can deliver the service.
[12:55:54] <David Oran> @Dirk - i think there’s a high level conceptual disconnect here - I don’t think the goal is to enable individual parties to innovate by providing clever services that can differentiate them, but rather to carefully extend the Internet service model from the current best effort single packet delivery to a reasonably small set of richer services that can be provided everywhere and depended on by all internet users.
[12:59:11] <Dirk Trossen> @David got it, had looked at EI in the hope that the current economic and technical centralization could be overcome with an SN model. But it seems to boil down to commoditize GAFA in-network services without much breathing space for others to get in edge-ways. From that perspective, I can understand the support from cloud players that Scott mentioned.
[13:01:33] <David Oran> Well, just like today, one would hope/expect that there aren’t any significant barriers to innovating on top of the globally supported services.
[13:03:19] <David Oran> …and just like today with peering and access economics, there are “interesting” cost shifting, arbitrage and phenomena that may or may not change the power relationships among users, ISPs. hyperscalers, and enterprises…
[13:04:52] <David Oran> …it’s likely that the biggest obstacles to overcome is the cost sharing model for power and rack space in enough POPs globally
Paper: https://www.usenix.org/conference/nsdi21/presentation/lao
In the typical distributed training approach, Parameter server(PS) architecture has many-to-one communication bottleneck. There is research to move the aggretation to the programmable switch. This work extends that by supporting multi-rack and dynamic resource partition to support multi tenants. The aggregation transmission protocol is developed to support reliability. Rethink reliability: Recovery from packet loss, Ensure exact once aggregation, Memory leak: aggregators are reserved forever, but not used. Rethink congestion control: N flows merged into one flow communication; Drop congestion signal, i.e., ECN; Improve the floating point computation; Convert gradients to 32-bit integer at workers by a scaling factor; Aggregation overflow at switch
Paper: https://doi.org/10.1145/3460417.3482975
Slides: https://conferences2.sigcomm.org/acm-icn/2021/assets/3-2-kutscher-387d18eb69a8104287bf75f800cc6a6042a8a00b89b0996bdd4abf7ab60542bd.pdf
In COIN, we have been discussing two strands. One coming from dataplane programmability, explores how this can be useful distributed systems, evolving some protocols. The other comes from distribtued computing, how we can learn from distributed computing, maybe change our view on networking and the relationsip between computing and networking. Maybe there will be some confluence at the end.This paper is more about the latter strand (distributed computing view).
Dataflow: Structured Distributed Data Processing. Split to different paths for parallel computing.
Mainstream Implementations: Apache BEAM; Unified programming model for data processing pipelines.
Dataflow applications: Apache Flink, Samza, Spark, Google Cloud Dataflow.
It is difficult for Flink to provide “buffer debloating” and “elastic jobs” because it is connection-based approach to connect the task managers. This is not a trivial problem since overlays do not match the inherent logic of processing immutable data objects.
Proposal: IceFlow (Information-Centric Dataflow).
Concepts: Just Names; Computation results as Named Data Objects; Asynchronous data production; Flow control (some coupling between consumers and producers); Garbage collection.
Take-aways for COIN: IceFlow is an example for new protocol work; Breaking up overlays; Dataflow for Flink, other interaction classes next, e.g., Kafka.
A proposal for an EU wide project.
IOT today a fragmented verticalized world. Many systems don’t talk to one another.
Why MODA?
Application development still faces important pain-points: Proprietary, fragmentary and verticalized approaches requiring overlays, multiple gateways and different cloud applications and providers; IoT applications need to respect universal data privacy and enforce digita sovereignty.
The computer board as a new Internet paradigm. Hence it needs an operating system (a Meta-OS). Move away from telephone network models and client server heritage. The IoT-to-edge-to-cloud is one realization of that vision
Data valorization is key: Maximizing data valorization by acting on it inside the network.
What is MODA?
MODA is the operating system for the new distributed Internet
It provides the infrastructure that allow applications to be easily developed such as: Discovery services; Communications and publish-subscribe; Semantic integration and data management; Implementation of commonly-used functionalities; APIs, tools and libraries for running code across multiple heterogeneous nodes.
Main functionalities: Orchestrating, reusability; “Extreme modularity”; Managing the network processing units, Supporting data and Intelligence services
Link to COINRG Research Topics: Discovery (storage, function, computation), Distributed abstractions and protocols, Decentralized security & trust, Federated learning, Use cases.
Q(Daniell King): is there any link to the project?
A: will ask the project management.
From the chat window:
[13:24:02] <Daniel King> Is there a link to the MODA website, or paper? There are no links, papers or project websites in the slides, and various search engines have many projects with similar names.
[13:29:03] <Daniel Bernier> would be interesting to see the diffs between the MODA approach versus other initiatives like Caladan or LegoOS, etc
[13:35:04] <Adrian Farrel> Is MODA a research project (perhaps currently applying for funding), an open source project (being established prior to full set up), a commercial consortium, or what?
[13:35:26] <Hannes Tschofenig> What happened actually to LegoOS?
[13:37:13] <Marie-Jose Montpetit> @Daniel: happy to talk
[13:37:44] <Marie-Jose Montpetit> @Adrian: applying for funding - but may morph
[13:39:07] <Adrian Farrel> @hannes. LegoOS git repo seemed to go quiet some time ago
[13:40:06] <Marie-Jose Montpetit> @Adrian: we plan to write some architecture paper somewhere once we get our heads above the water.
Draft: https://datatracker.ietf.org/doc/html/draft-irtf-coinrg-use-cases-01
This draft is addressing item #2 in the charter.
Changes: Regrouping the use cases, sharpening and tightening the taxonomy, preparing the analysis.
Next Steps:
Finish aligning the use cases according to tightened taxonomy, continue aligning the draft with COIN RG terminology; draft-kutscher-coinrg-dir-02, Q:
Start with the analysis: condense opportunities, research questions, requirements, identify aspects relevant across all use cases
Questions: will there be an update to this? where to collect the COIN RG terminology otherwise?
From the chat window:
[13:33:47] <Dirk Kutscher> Sorry for letting it expire – there are many ideas for updating the draft – just had too many things going on.
[13:39:13] <Ike Kunze> @DirkK: No worries. We just wondered where it might be best to collect the terminology.
Draft: https://www.ietf.org/archive/id/draft-kunze-coinrg-transport-issues-05.txt
“This draft focuses on the potential opportunities and research questions for the design of transport protocols that may assume the availability of in-network computing capabilities.”
Intention of this Draft:
Provide insights into (transport) technology areas, research questions and challenges, ongoing efforts and concepts under study, outline possible future work (in COIN and elsewhere), Gap analysis.
Future plans:
Clearer linkage to various use cases in revised/future use case draft. More existing work (such as MTP work presented at HotNet2021 as new transport work). Possibly turn research questions into requirements language at later stage.
Gap analysis -> really need help here.
Adopt as RG draft towards one key output towards scope #4 in COIN charter?
Q (Peng Liu): There is a link to Dyncast. Could this be a potential way to solve flow affinity and service equivelence?
A: yes, that link is still there although the dyncast draft is expired. The mention of dyncast aspects need to put more properly in the draft. Yes, it need to be clarified.
Q(Spencer): Is there a github repo so that we can contribute to it especially the gap analysis?
A: Yes. https://github.com/ike-kunze/transport-issues
From the chat window:
[13:38:22] <Marie-Jose Montpetit> Related to the current presentation - a paper in Hotnets today:
TCP is Harmful to In-Network Computing: Designing a Message Transport Protocol (MTP)
Brent E. Stephens (University of Utah); Darius Grassi and Hamidreza Almasi (University of Illinois at Chicago (UIC)); Tao Ji (UT Austin); Balajee Vamanan (University of Illinois at Chicago (UIC)); Aditya Akella (UT Austin)
[13:43:48] <Spencer Dawkins> Is there a github repo for the transport draft?
[13:45:04] <Ike Kunze> @Spencer: for a previous version, yes. I think we have not pushed the current version, but can do that.
Draft: https://datatracker.ietf.org/doc/draft-fink-coin-sec-priv/03/
Update: Related Work w.r.t.
(1) Encryption and Integrity Checks
• Chen et al. (2020) [1]: AES encryption with scrambled lookup tables on P4-based hardware switches
• Yoo et al. (2021) [2]: Cryptographically secure keyed hash functions on P4-based hardware switches
(2) Authentication
• Almaini et al. (2021) [3]: Authentication in the data plane of P4-based hardware switches
(3) Anonymization:
• Moghaddam et al. (2019) [4]: Use P4-based hardware switches to rewrite source addresses and hide path information, e.g., using randomization
• Wang et al. (2020) [5]: Encrypt IPv4 addresses to obfuscate traffic on P4-based hardware switches
(4) IDS
• Lewis et al. (2019) [6]: Outsource IDS functionality / preprocessing to P4-based software switches to reduce load on subsequent IDS
(5) Network Monitoring:
• Sonchack et al. (2018) [7]: Flow monitoring with P4- based hardware switches
Conclusion:
Increasing interest of the research community. Recent publications show relevance and feasibility. High-ranked venues (USENIX Security, SIGCOMM SPIN Workshop, EuroSys, …). First proofs of concept using programmable hardware switches. Hot research topic, many ideas left to investigate
Draft indicates broad and valuable potential of COIN.
From the chat window:
[13:24:18] <Jari Arkko> A comment on draft-fink-coin-sec-priv. The draft is interesting and of course addresses some relevant questions, such as how COIN and edge systems can assist for instance IOT devices in various security tasks. Thanks for writing it! Comment: for me at least it would also be useful to discuss how applications can trust the edge and COIN systems, and what mechanisms might be needed to support/enforce that trust. It could of course be that such discussion is in some other document… Anyway, the situation seems like a two-way street in many ways: what do we need in the devices and the network and compute elements to achieve a particular goal?
[13:49:04] <Dirk Kutscher> BTW, the PPM work that is behind the “priv” BOF (https://datatracker.ietf.org/meeting/112/materials/agenda-112-priv-02) is a secure MPC approach to privacy-preserving in-network computing (for data collection)
Draft: https://datatracker.ietf.org/doc/draft-li-coinrg-compute-resource-scheduling/
Due to limited time, this presentation was skipped.
Upcoming events
Next meetings
Total: 120 minutes