Minutes of the PTTH BoF at IETF 123.

Many thanks to our note takers:

• Tommy Jensen
• Emily Heron
• Cullen Jennings

Welcome to the PoTaToH BOF. (PTTH)

This is a non WG forming BOF.

Discussions from 2009 to now in various forums until AD suggested BoF.

In the traditional client-server communication model, the client
initiates a handshake to establish communication (often complicated by
NATs, firewalls, etc.). then HTTP can be sued bidirectionally. This
topic is reversing this: the server initiates communication with the
client.

Presented by Kazuho from Fastly

Terminology: to dedup "client" and "server" uses, let's use "worker" to
describe the node processing the HTTP request.

Use cases

• hidden servers
• • status quo: backend servers rely on block-by-default firewalls
    using fixed IP addresses
• • with PTTH, we no longer need to configure firewalls and can use
    hostnames instead of hard coded IP addresses

• load balancing / dynamic scaling

• • status quo: adding a server means tedious CDN config changing
• • with PTTH, just clone isntances of workers with no changes
    needed to CDN config

• prioritization

• • status quo: we don't know how much load the backend can handle
    (because each instance is behind a load balancer), increasing
    latency as we push the limit
• • with PTTH, we can now see concurrent requests and know when we
    are limiting each worker's limits

• limiting concurrency of heavy requests

• • status quo: queueing requests to be consumed by processing
    servers
• • with PTTH, reverse proxy acts as queue

• standardization

• • status quo: custom CDN gateways installed into customer servers
• • with PTTH, each worker can handle this without outside gateways

HTTP Workshop 2024, multiple server vendors expressed interest.

How are PTTH workers supposed to auth themselves? workers to each other
is an internal matter, no longer dependent on PKI

WG discussion

Erum Welling: sense of the increased load coming from the worker to the
CDN? Where are you seeing the overhead?

• We receive connection from worker then immediately send requests
  from CDN to worker (no overhead)

Martin Seemann: isn't it bad to have bespoke auth between workers?
Should use TLS certs or similar standard

• Sure, it can be whatever you want it to be

Emile Stephan: what are these servers? they are the same as the workers?

• yes

Harald Slvestrand: when the request hits the CDN, it then immediately
goes to the worker, but later you said there was a queue before workers.
Which is it?

• Depends on how customer configures CDN. We should not standardize
  this (leave it to CDN).

Cullen Jennings has to step out - Emily Heron taking over as back up
note taker

• noted, thanks

Presented by Marius Kleidl

Control of proxying process through reverse HTTP

Currently, we assume reverse proxy that forward to origin with load
balancer. In this setup, the origin does not have control over reverse
proxy config.

Alternatively, when the origin is now a PTTH worker connecting to proxy,
it can push per-connection config. Proxy still forwards requests but
with worker input.

Potential configs:

• control rate of incoming requests ("at most 1,000 concurrent
  requests, or RPS" etc.)
• request selectors ("only forward requests matching this
  path/hsotname/header values" etc.)
• • example given: separate authentication service

• connection draining (graceful shutdown, "stop sending requests and
  drain current ones")

All of this is currently possible, but has to be manually configured in
a non-standardized way. This doesn't completely eliminate proxy config,
but reduces a lot.

WG discussion

Erum Welling: now you're pushing config privileges to workers (change to
security model). We need to really think about how this can be abused.
For example, insider threat would mean the worker cannot be trusted.

• Chairs: to be fair, speakers were asked to brainstorm use cases, not
  necessarily threat models.

Ted Hardie: two possibilities: (1) PTTH setting up connection with
config protocols, (2) using HTTP mecahnisms to control the proxy (not a
PTTH side channel).

• you don't have to have a separate control plane or protocol

Presented by Hrant Novikov from Meta (10 min)

Reverse HTTP: use cases and missing functionality

At Meta, we maintain some one-off deployments that are separate from
corp environment with L3 connectivity between them, firewall holes only
for what the one-off deployments need. My team operate L7 tunnels to use
HTTP as a substrate for arbitrary TCP/UDP sues cases such as SSH. This
involves a client on the device and internet-facing service. This is
cumbersome doing for every deployment.

Better alternative: have a reflector open to the Internet, but the
one-off deployment now does the connecting to the reflector (reverse
HTTP). Reflector subsumes the role of DNS, BGP, etc. with a direct link.

Workarounds:

• This can be accomplished with H2C in normal HTTP, but with multiple
  layers of flow control and generally doesn't work with datagram
  semantics.
• HTTP/3 over HTTP/3 would generally work, but has heavy impact on
  MTUs versus a single use of QUIC.

Idealized protocol would

• minimize RTTs
• allow datagram semantics
• minimize MTU impact
• allow TCP fallback
• semantics to signal peers that substrate is going away
• possible integration with pre-existing load balancers

WG discussion

Marius Kleidl: do you have real-world deployment of the L7 example you
gave?

• Yes

Presented by Yaroslav Rosomakho from Zscaler (10 min)

Reverse HTTP for CONNECT

In traditional VPNs, client contains to concentrator, gets IP address,
and establishes bidirectional communication. Lots of options (IKEv2,
CONNECT-IP, WG, etc.)

In ZTNA architecture (Zero Trust Network Access), client and worker both
connect to proxy to carry L4 traffic.

• for client -> proxy, we have CONNECT, CONNECT-UDP, etc. with great
  interop
• for worker -> proxy, it is not so good (requires IP addresses,
  routing, MTU issues, TCP-over-TCP perf issues)
• • Note: capsules work well for CONNECT-UDP listen, but not as good
    for TCP (don't want to multiplex TCP connections in the same
    stream)

• for proxy -> client, we have RDP, SSH, etc. for remote assist that
  require config today that server initiating would solve

Desired properties

• server=initiatied for HTTP/3 and /2
• flow encapsulation like CONNECT-*
• regular and reverse flows

WG discussion

Kazuho: do you mean the same HTTP connection for regular and reverse
flows? or one for each direction?

• The same connection to reduce connection state overhead

Presented by Lucas Pardue fronm Cloudflare (10 min)

PTTH

Cloudflare has both CDN / reverse proxy and the ZTNA (ZT SASE) use cases
in production today.

Cloudflare product: Cloudflare Tunnel (connects servers of HTTP and
other protocols safely to Cloudflare). Cloudflared service running
locally on customer server that connects upstream to Cloudflare, with
customer service connecting locally. We label this "worker" and suddenly
it looks a lot like PTTH.

• TODO: link from slide 3

WG discussion

(no questions)

Open Mic

Ted Hardie: thank you, all valuable use cases. There are two very
different scopes: (1) we want transport initiator and make it the HTTP
server (the CDNI use cases look like a previous draft with added
signaling), and (2) the ZT scope and HTTP substrate, which begs the
question "what is the waste of the Internet"? Some of your use cases
have nothing to do with HTTP and we have to recognize people will use
this as their own substrate for other things over HTTP. When writing a
charter in the future, you need to define this scope carefully (huge gap
between work needed for these scopes)

• Chairs: where are you on this spectrum of scope possibilities?
• Ted: do the simple things first and more later. CDN use case is
  doable in relatively low time.
• Speakers: it is very easy to get flow control, auth horribly
  wrong, so doign that work here in a standar way would make the
  Internet a better place.

Martin Thompson: +1 to Ted, easiest to focus on the CDN use case.
Unfortunately, that would give everything else for free (looking at
Lucas' presentation) whether intended or not because people will use the
established HTTP themselves for everything, whether we solve the problem
in a standard way or not. The charter needs to ack that and be careful
given we will probably have to do a larger scope.

• Alessandro: I'm not sure I understand why we have to take on the
  extra scope, leave these up to other use cases, groups

Mike Blanche: +1 to Ted. I am a user of Cloudflare Tunnel for non-HTTP
traffic, so I selfishly want that, but starting simple makes sense.

Theo Dimitrakos: We have AI agents acting on behalf of users today. Best
way to control those requests is by having corp or user policy enforced
by reverse proxy. This could be in scope also.

Alessandro Ghedini: Imagine I want to expose a homemade website in a
hosted VPN to another across providers... different than corp cases
presented and would still benefit.

Chairs: it's possible to do work in existing WGs or new WG. This is up
to ADs, though we want to hear feedback on this decision also.

Chairs: we have only heard positive energy so far for doing this work
somewhere at the IETF, so please speak up if you are opposed.

Philipp Tiesel: we (SAP) have a product that works by on-prem system
connecting to the cloud, getting WebSocket traffic from cloud
integration to on-prem application. PTTH would help with auditing,
modern practices, and therefore a good thing to work on. I think we are
solving the right problem, but suggest we do not do this in httpbis WG
(it's already a zoo).

• Chairs: please share the requirements from your use case on the list

Benjamin Schwartz: I've attempted drafts for this space a couple of
times. I heard Ted talk about "origin slicing" (worker wants to claim
some subset of methods, paths, etc. of an origin) which I think is
useful, but we shouldn't try to sovle this yet. Instead, an opaque
deployment defined box to handle configuration (we provide a channel,
then what you do within it is non-standard).

• Martin: we don't get to decide whether there's slicing or not given
  proxies already have discretion on what they get to forward. We need
  to tackle that also.

Ben Schwartz: we should do this by splitting httpbis into semantics and
transport topics. Alterantively, a new focused WG.

• Mark: having semantics and transport aspects of HTTP in the same WG
  is a feature, not a bug

Mirja Kühlewind: I support solving this in the IETF.

Martin Thompson: given the confusion around the scope, it would be a
good idea to go through WG-forming BoF process even if just to strongly
define scope limits, though I'm leaning toward new WG also.

Mark Nottingham: this all sounds familiar... we hesitated to adopt this
in httpbis for a reason: triggered follow-on work, poorly defined scope.
I'm not saying don't do it, but this not being tightly scoped makes me
nervous.

• Chairs: we do intend to

Abhijit Singh: (left queue???)

Mirja Kühlewind: are there things only HTTP can provide, or are we just
using HTTP because it is what is familiar?

• Speakers: a huge benefit of HTTP is it "just works" (allowed by
  default through firewalls, etc.). Another benefit is separating
  semantics of whatever-over-HTTP from Layer 4 semantics for flexible
  usage (see MASQUE). HTTP already does everything we need in the
  forward direction, so why invent the wheel? IT is also very
  extensible.
• Martin Thompson: MASQUE isn't relevant in this context. What is
  relevant is having clients be able to make requests, and HTTP
  already solved cross-version communication and other things. That's
  the real answer to me.
• Mirja Kühlewind: what if you don't already have an HTTP stack? It
  seems odd to constrain this to the assumption that the client wants
  to send HTTP requests.

Martin Thompson: I predict the very first thing that will happen after
PTTH starts is browsers will be asked to add a worker implementation.

• Speakers: Laughter. There are many problems with this.

Mirja Kühlewind: you suggested it might be worse to standardize in
speecific use cases to let vendors invent, but I didn't see requirements
for interop. Is it always single vendor?

• Speakers: In our use case, interop is a choice by using existing
  protocols. Also, on the worker side, vendors are doing fundamentally
  similar things. It would be beneficial to standardize this.

Marco Munizaga: HTTP requests can be made from both sides as presented.
Do all your cases need this? Mine does.

• Speakers: this might everything more complex (you already have
  control mechanisms in HTTP), I don't think it's necessary. Sometimes
  other protocols don't work the way HTTP does (FTP, SSH, etc.). Not
  sure how you would handle auth in the other direction.

Erum Welling: where are security considerations addressed? new to the
IETF

• Chairs: welcome! At this stage, we are trying to determine the scope
  of the work to see where it fits, then whoever works on it is
  absolutely required to think through and document the security
  considerations. So later, just not necessarily now.

Poll: Do you understand the use cases described today? Yes 48, No 0, No
opinion 1

Poll: Do you think the IETF should work on this? (These use cases and
something that could address this problem space) Yes 40, No 1, No
Opinion 6

Poll: Do you think a tightly scoped charter in this space is possible?
(is it worth it for the proponents to spend time writing one?) Martin:
These are two different questions. ReAsk: Do you think an acceptable to
you charter in this space is possible? Yes 45, No 2, No opinion 3

Mark Nottingham: A tightly scoped charter would be one that just does
the CDN use case and that could be done in the HTTP Working Group which
wouldn't need a charter. The bigger thing is what we keep on trying to
wrap our heads around and that's where I have doubts that we can
actually make progress in a reasonable amount of time. That's why I
voted no.

Ted Hardie:If this is simple reverse, we're doomed. Need to start with a
use case thats easy to understand and has immediate benefit. Cant get
tightly scoped - best hope is well scoped. +1 Martin

Cullen Jennings: Meangingless to go as simple or small as possible. At
min slicing up the name space needs to be in.

Ben Schwartz: Slicing should be out, excluded from scope. Need byte
string container to move blob from worker to proxy.

• (Chairs cut off this discussion as getting ahead of ourselves
  talking about specific bits on the wire)

Erum Welling: Shouldnt be throwing solutions out without proper
discussion. Tied to http group, how can we do this seperately?

• Chairs: this is normal for IETF. For example, the masque WG was
  formed separate from httpbis even though they are tied. WGs are
  often required to liaise.

Poll: Would you be willing to review or author drafts in this space? (If
we do this, are people willing to actually do the work?) Yes 21, No 8,
No Opinion 7

[as AD] Mike Bishop: happy to see there were multiple different use
cases. Next steps should be a new mailing list and plan to BoF next time

• Chairs: ok, we will announce the new list on the httpbis list once
  it exists