HTTP Message Signatures for automated traffic Architecture
draft-meunier-web-bot-auth-architecture-00

Web Bot Auth                                                  T. Meunier
Internet-Draft                                                Cloudflare
Intended status: Informational                             15 April 2025
Expires: 17 October 2025

       HTTP Message Signatures for automated traffic Architecture
               draft-meunier-web-bot-auth-architecture-00

Abstract

   This document describes an architecture for identifying automated
   traffic using [HTTP-MESSAGE-SIGNATURES].  The goal is to allow
   automated HTTP clients to cryptographically sign outbound requests,
   allowing HTTP servers to verify their identity with confidence.

About This Document

   This note is to be removed before publishing as an RFC.

   The latest revision of this draft can be found at
   https://thibmeu.github.io/http-message-signatures-directory/draft-
   meunier-web-bot-auth-architecture.html.  Status information for this
   document may be found at https://datatracker.ietf.org/doc/draft-
   meunier-web-bot-auth-architecture/.

   Discussion of this document takes place on the WG Web Bot Auth
   mailing list (mailto:web-bot-auth@ietf.org), which is archived at
   https://mailarchive.ietf.org/arch/browse/web-bot-auth/.  Subscribe at
   https://www.ietf.org/mailman/listinfo/web-bot-auth/.

   Source for this draft and an issue tracker can be found at
   https://github.com/thibmeu/http-message-signatures-directory.

Status of This Memo

   This Internet-Draft is submitted in full conformance with the
   provisions of BCP 78 and BCP 79.

   Internet-Drafts are working documents of the Internet Engineering
   Task Force (IETF).  Note that other groups may also distribute
   working documents as Internet-Drafts.  The list of current Internet-
   Drafts is at https://datatracker.ietf.org/drafts/current/.

   Internet-Drafts are draft documents valid for a maximum of six months
   and may be updated, replaced, or obsoleted by other documents at any
   time.  It is inappropriate to use Internet-Drafts as reference
   material or to cite them other than as "work in progress."

Meunier                  Expires 17 October 2025                [Page 1]
Internet-Draft      HTTP Message Signatures for Bots          April 2025

   This Internet-Draft will expire on 17 October 2025.

Copyright Notice

   Copyright (c) 2025 IETF Trust and the persons identified as the
   document authors.  All rights reserved.

   This document is subject to BCP 78 and the IETF Trust's Legal
   Provisions Relating to IETF Documents (https://trustee.ietf.org/
   license-info) in effect on the date of publication of this document.
   Please review these documents carefully, as they describe your rights
   and restrictions with respect to this document.  Code Components
   extracted from this document must include Revised BSD License text as
   described in Section 4.e of the Trust Legal Provisions and are
   provided without warranty as described in the Revised BSD License.

Table of Contents

   1.  Introduction  . . . . . . . . . . . . . . . . . . . . . . . .   3
   2.  Motivation  . . . . . . . . . . . . . . . . . . . . . . . . .   3
     2.1.  HTTP layer choice . . . . . . . . . . . . . . . . . . . .   4
   3.  Conventions and Definitions . . . . . . . . . . . . . . . . .   4
   4.  Architecture  . . . . . . . . . . . . . . . . . . . . . . . .   5
     4.1.  Deployment Models . . . . . . . . . . . . . . . . . . . .   5
     4.2.  Generating HTTP Message Signature . . . . . . . . . . . .   5
       4.2.1.  Anti-replay . . . . . . . . . . . . . . . . . . . . .   6
       4.2.2.  Sending a request . . . . . . . . . . . . . . . . . .   6
     4.3.  Requesting a Message signature  . . . . . . . . . . . . .   7
     4.4.  Validating Message signature  . . . . . . . . . . . . . .   7
     4.5.  Key Distribution and Discovery  . . . . . . . . . . . . .   8
       4.5.1.  Out-of-band communication between client and
               origin  . . . . . . . . . . . . . . . . . . . . . . .   8
       4.5.2.  Public list . . . . . . . . . . . . . . . . . . . . .   8
       4.5.3.  Signature-Agent header  . . . . . . . . . . . . . . .   8
   5.  Security Considerations . . . . . . . . . . . . . . . . . . .   8
     5.1.  Performance Impact  . . . . . . . . . . . . . . . . . . .   9
     5.2.  Key Compromise Response . . . . . . . . . . . . . . . . .   9
     5.3.  Shared Secrets Considered Harmful . . . . . . . . . . . .   9
     5.4.  Key Reuse Considered Harmful  . . . . . . . . . . . . . .   9
   6.  Privacy Considerations  . . . . . . . . . . . . . . . . . . .   9
     6.1.  Public Identity . . . . . . . . . . . . . . . . . . . . .   9
     6.2.  No Human Correlation  . . . . . . . . . . . . . . . . . .  10
     6.3.  Minimizing Tracking Risks . . . . . . . . . . . . . . . .  10
   7.  IANA Considerations . . . . . . . . . . . . . . . . . . . . .  10
   8.  References  . . . . . . . . . . . . . . . . . . . . . . . . .  10
     8.1.  Normative References  . . . . . . . . . . . . . . . . . .  10
     8.2.  Informative References  . . . . . . . . . . . . . . . . .  11
   Appendix A.  Implementations  . . . . . . . . . . . . . . . . . .  11

Meunier                  Expires 17 October 2025                [Page 2]
Internet-Draft      HTTP Message Signatures for Bots          April 2025

   Acknowledgments . . . . . . . . . . . . . . . . . . . . . . . . .  12
   Author's Address  . . . . . . . . . . . . . . . . . . . . . . . .  12

1.  Introduction

   Agents are increasingly used in business and user workflows,
   including AI assistants, search indexing, content aggregation, and
   automated testing.  These agents need to reliably identify themselves
   to origins for several reasons:

   1.  Regulatory compliance requiring transparency of automated systems

   2.  Origin resource management and access control

   3.  Protection against impersonation and reputation management

   4.  Service level differentiation between human and automated traffic

   Current identification methods such as IP allowlisting, User-Agent
   strings, or shared API keys have significant limitations in security,
   scalability, and manageability.  This document defines an
   architecture enabling agents to cryptographically identify themselves
   using [HTTP-MESSAGE-SIGNATURES].  It proposes that every request from
   bots to be signed by a private key owned by its provider.  This way,
   every origin can validate the service identity.

2.  Motivation

   There is an increase in agent traffic on the Internet.  Many agents
   choose to identify their traffic today via IP Address lists and/or
   unique User-Agents.  This is often done to demonstrate trust and
   safety claims, support allowlisting/denylisting the traffic in a
   granular manor, and enable sites to monitor and rate limit per agent
   operator.  However, these mechanisms have drawbacks:

   1.  User-Agent, when used alone, can be spoofed meaning anyone may
       attempt to act as that agent.  It is also overloaded - an agent
       may be using Chromium and wish to present itself as such to
       ensure rendering works, yet it still wants to differentiate its
       traffic to the site.

Meunier                  Expires 17 October 2025                [Page 3]
Internet-Draft      HTTP Message Signatures for Bots          April 2025

   2.  IP blocks alone can present a confusing story.  IPs on cloud
       plaforms have layers of ownership - the platform owns the IP and
       registers it in their published IP blocks, only to be re-
       published by the agent with little to bind the publication to the
       actual service provider that may be renting infra.  Purchasing
       dedicated IP blocks is expensive, time consuming, and requires
       significant specialist knowledge to set up.  These IP blocks may
       have prior reputation history that needs to be carefully
       inspected and managed before purchase and use.

   3.  An agent may go to every website on the Internet and share a
       secret with them like a Bearer from [OAUTH-BEARER].  This is
       impractical to scale for any agent beyond select partnerships,
       and insecure, as key rotation is challenging and becomes less
       secure as the consumers scale.

   Using well-established cryptography, we can instead define a simple
   and secure mechanism that empowers small and large agents to share
   their identity.

2.1.  HTTP layer choice

   This architectures operates solely at the HTTP layer.  It allows
   signatures to be generated and verified without modifying the
   transport layer or TLS stack.  It enables flexible deployment across
   proxies, gateways, and origin servers, and aligns with existing
   tooling and infrastructure that already inspect and manipulate HTTP
   headers.

   Because the signature is embedded in the request itself, it travels
   with the message through intermediaries, preserving end-to-end
   verifiability even when requests are forwarded or transformed within
   the HTTP layer.

3.  Conventions and Definitions

   The key words "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT",
   "SHOULD", "SHOULD NOT", "RECOMMENDED", "NOT RECOMMENDED", "MAY", and
   "OPTIONAL" in this document are to be interpreted as described in
   BCP 14 [RFC2119] [RFC8174] when, and only when, they appear in all
   capitals, as shown here.

   The following terms are used throughout this document:

   *User*  An entity initiating requests through an agent.  May be a
      human operator or another system.

   *Agent*  An orchestrated user agent (e.g. Chromium, CURL).  It

Meunier                  Expires 17 October 2025                [Page 4]
Internet-Draft      HTTP Message Signatures for Bots          April 2025

      implements the HTTP protocol and constructs valid HTTP requests
      with [HTTP-MESSAGE-SIGNATURES] signatures.

   *Origin*  An HTTP server receiving signed requests that implements
      the HTTP protocol and verifies [HTTP-MESSAGE-SIGNATURES]
      signatures.  It acts as a verifier of the signature as defined by
      [HTTP-MESSAGE-SIGNATURES].

4.  Architecture

+--------+               +---------+                           +----------+
|        |               |         |         Exchange          |          |
|        |               |         |<===== Cryptographic =====>|          |
|        |               |         |         material          |          |
|  User  +--- Request -->|  Agent  |                           |  Origin  |
|        |               |         +--- Request + Signature -->|          |
|        |               |         |<-------- Response --------+          |
|        |<-- Response --+         |                           |          |
|        |               |         |                           |          |
+--------+               +---------+                           +----------+

   A User initiates an action requiring the Agent to perform an HTTP
   request.  The Agent constructs the request, generates a signature
   using its signing key, and includes it in the request as defined in
   Section 3.1 of [HTTP-MESSAGE-SIGNATURES] along with Signature-Agent
   header for discovery for its verification key.  Upon receiving the
   request, the Origin ensures it has the verification key for the
   Agent, validates the signature, and processes the request if the
   signature is valid.

4.1.  Deployment Models

   Signature verification can be performed either directly by origins or
   delegated to a fronting proxy.  Direct verification by origins
   provides simplicity and control.  Proxy verification offloads
   processing and enables shared caching across multiple origins.  The
   choice depends on traffic volume and operational requirements.

4.2.  Generating HTTP Message Signature

   [HTTP-MESSAGE-SIGNATURES] defines components to be signed.

   Agents MUST include the following component:

   @authority  as defined in Section 2.2.1 of [HTTP-MESSAGE-SIGNATURES]

   Agents MUST include the following @signature-params as defined in
   Section 2.3 of [HTTP-MESSAGE-SIGNATURES]

Meunier                  Expires 17 October 2025                [Page 5]
Internet-Draft      HTTP Message Signatures for Bots          April 2025

   created  as defined in Section 2.3 of [HTTP-MESSAGE-SIGNATURES]

   expires  as defined in Section 2.3 of [HTTP-MESSAGE-SIGNATURES]

   keyid  MUST be a base64url JWK SHA-256 Thumbprint as defined in
      Section 3.2 of [JWK-THUMBPRINT] for RSA and EC, and in
      Appendix A.3 of [JWK-OKP] foe ed25519.

   tag  MUST be web-bot-auth

   The signing key is available to the agent at request time.
   Algorithms should be registered with IANA as part of HTTP Message
   Signatures Algorithm registry.

   The creation of the signature is defined in Section 3.1 of
   [HTTP-MESSAGE-SIGNATURES].

   It is RECOMMEND the expiry to be no more than 24 hours.

4.2.1.  Anti-replay

   Origins MAY want to prevent signatures from being spoofed or used
   multiple times by bad actors and thus require a nonce to be added to
   the @signature-params.

   Agents SHOULD extend @signature-parameters defined in Section 4.2 as
   follow

   nonce  Base10 encoded random uint32

   This nonce MUST be unique for the validity window of the signature,
   as defined by created and expires attributes.  Because the nonce is
   controlled by the client, the origin needs to maintain a list of all
   nonces that it has seen that are still in the validity window of the
   signature.

4.2.2.  Sending a request

   An Agent SHOULD send a request with the signature generated above.
   Updating the architecture diagram, the flow look as follow.

Meunier                  Expires 17 October 2025                [Page 6]
Internet-Draft      HTTP Message Signatures for Bots          April 2025

+---------+                                                                     +----------+
|         |                              Exchange                               |          |
|         |<=========================  Cryptographic  =========================>|          |
|         |                              material                               |          |
|  Agent  |                                                                     |  Origin  |
|         |     .------------------------------------------------------.        |          |
|         +-----| GET /path/to/resource                                |------->|          |
|         |     | Signature: abc==                                     |        |          |
+---------+     | Signature-Input: sig=(@authority);tag="web-bot-auth" |        +----------+
                | Signature-Agent: signer.example.com                  |
                '------------------------------------------------------'

   The Agent SHOULD send requests with two headers

   1.  Signature defined in Section 4.2

   2.  Signature-Input defined in Section 4.2

   Mentioned in a section below, the Agent MAY send request with an
   additional header 3.  Signature-Agent defined in Section 4.5.3

4.3.  Requesting a Message signature

   Section 5 of [HTTP-MESSAGE-SIGNATURES] defines the Accept-Signature
   field which can be used to request a Message Signature from a client
   by an origin.  Origin MAY choose to request signatures from clients
   that did not initially provide them.  If requesting, Origins MUST use
   the same parameters as those defined by the Section 4.2.

4.4.  Validating Message signature

   Upon receiving an HTTP request, the origin has to verify the
   signature.  The algorithm is provided in Section 3.2 of
   [HTTP-MESSAGE-SIGNATURES].  Similar to regular User-Agent check, this
   happens at the HTTP layer, once headers are received.

   Additional requirements are placed on this validation:

   *  During step 4, the Origin MAY discard signatures for which the tag
      is not set to web-bot-auth.

   *  During step 5, the Origin MAY discard signatures for which they do
      not know the keyid.

   *  During step 5, if the keyid is unknown to the origin, they MAY
      fetch the provider directory as indicated by Signature-Agent
      header defined in Section 4 of [DIRECTORY].

Meunier                  Expires 17 October 2025                [Page 7]
Internet-Draft      HTTP Message Signatures for Bots          April 2025

4.5.  Key Distribution and Discovery

   This section describes the discovery mechanism for the agent
   directory.

   The reference for discovery is a FQDN.  It SHOULD provide a directory
   hosted on the well known registered in Section 4 of [DIRECTORY].

   We add one field to the directory defined in the other draft:
   "purpose": Ideally matches some IANA registry for preferences

   Example

   {
     "keys": {
       "kty": "OKP",
       "crv": "Ed25519",
       "kid": "NFcWBst6DXG-N35nHdzMrioWntdzNZghQSkjHNMMSjw",
       "x": "JrQLj5P_89iXES9-vFgrIy29clF9CC_oPPsw3c5D0bs",
       "use": "sig",
       "nbf": 1712793600,
       "exp": 1715385600
     },
     "signature_agent": "https://directory.test",
     "purpose": "rag"
   }

4.5.1.  Out-of-band communication between client and origin

   A service submitting their key to an origin, or the origin manually
   adding a service to their trusted list

4.5.2.  Public list

   Could be a GitHub repository like the public suffix list.  The issue
   is the gating of such repositories, and therefore its governance.

4.5.3.  Signature-Agent header

   This is defined in the sibling draft.  This allows for backward
   compatibility with existing header agent filtering, and an upgrade to
   cryptographically secured protocol.

5.  Security Considerations

Meunier                  Expires 17 October 2025                [Page 8]
Internet-Draft      HTTP Message Signatures for Bots          April 2025

5.1.  Performance Impact

   Origins must account for the overhead of signature verification in
   their operations.  A local cache of public keys reduces network
   requests and verification latency.  The choice of signing algorithm
   impacts CPU requirements.  Origins should monitor verification
   latency and set appropriate timeouts to maintain service levels under
   load.

5.2.  Key Compromise Response

   An agent signing key might get compromised.

   If that happens, the agent SHOULD cease using the compromised key as
   soon as possible, notify affected origins if possible, and generate a
   new key pair.

   To minimise the impact of a key compromise, the origin should support
   rapid key rotation and monitor for suspicious signature patterns.

5.3.  Shared Secrets Considered Harmful

   Implementations MUST NOT use shared HMAC defined in Section 3.3.3 of
   [HTTP-MESSAGE-SIGNATURES].  Shared secrets break non-repudiation and
   make auditing difficult.  Each automated client SHOULD use a unique
   asymmetric keypair to ensure attribution, support key rotation, and
   enable effective revocation if needed.

5.4.  Key Reuse Considered Harmful

   Implementations MUST NOT reuse a signing key for different purposes.
   For example, if an agent implementor has two agents they want to
   differentiate, these should use distinct signing keys and signing key
   directories.

6.  Privacy Considerations

6.1.  Public Identity

   This architecture assumes that automated clients identify themselves
   explicitly using digital signatures.  The identity associated with a
   signing key is expected to be publicly discoverable for verification
   purposes.  This reduces anonymity and allows receivers to associate
   requests with specific agents.  If an agent wishes not to identify
   itself, this is not the right choice of protocol for it.

Meunier                  Expires 17 October 2025                [Page 9]
Internet-Draft      HTTP Message Signatures for Bots          April 2025

6.2.  No Human Correlation

   The key used for signing MUST NOT be tied to a specific human
   individual.  Keys SHOULD represent a role, company, or automation
   identity (e.g., "news-aggregator- bot", "example-crawler-v1").  This
   avoids accidental exposure of personally identifiable information and
   prevents the misuse of keys for user tracking or profiling.

6.3.  Minimizing Tracking Risks

   To limit tracking risks, implementations SHOULD avoid long-lived,
   globally unique key identifiers unless strictly necessary.  Key
   rotation SHOULD be supported, and clients SHOULD take care to avoid
   signing information that could be used to correlate activity across
   contexts, especially where sensitive user data is involved.

7.  IANA Considerations

   This document has no IANA actions.

8.  References

8.1.  Normative References

   [DIRECTORY]
              Meunier, T., "HTTP Message Signatures Directory", Work in
              Progress, Internet-Draft, draft-meunier-http-message-
              signatures-directory-00, 15 April 2025,
              <https://datatracker.ietf.org/doc/html/draft-meunier-http-
              message-signatures-directory-00>.

   [HTTP]     Fielding, R., Ed., Nottingham, M., Ed., and J. Reschke,
              Ed., "HTTP Semantics", STD 97, RFC 9110,
              DOI 10.17487/RFC9110, June 2022,
              <https://www.rfc-editor.org/rfc/rfc9110>.

   [HTTP-CACHE]
              Fielding, R., Ed., Nottingham, M., Ed., and J. Reschke,
              Ed., "HTTP Caching", STD 98, RFC 9111,
              DOI 10.17487/RFC9111, June 2022,
              <https://www.rfc-editor.org/rfc/rfc9111>.

   [HTTP-MESSAGE-SIGNATURES]
              Backman, A., Ed., Richer, J., Ed., and M. Sporny, "HTTP
              Message Signatures", RFC 9421, DOI 10.17487/RFC9421,
              February 2024, <https://www.rfc-editor.org/rfc/rfc9421>.

Meunier                  Expires 17 October 2025               [Page 10]
Internet-Draft      HTTP Message Signatures for Bots          April 2025

   [JWK-OKP]  Liusvaara, I., "CFRG Elliptic Curve Diffie-Hellman (ECDH)
              and Signatures in JSON Object Signing and Encryption
              (JOSE)", RFC 8037, DOI 10.17487/RFC8037, January 2017,
              <https://www.rfc-editor.org/rfc/rfc8037>.

   [JWK-THUMBPRINT]
              Jones, M. and N. Sakimura, "JSON Web Key (JWK)
              Thumbprint", RFC 7638, DOI 10.17487/RFC7638, September
              2015, <https://www.rfc-editor.org/rfc/rfc7638>.

   [RFC2119]  Bradner, S., "Key words for use in RFCs to Indicate
              Requirement Levels", BCP 14, RFC 2119,
              DOI 10.17487/RFC2119, March 1997,
              <https://www.rfc-editor.org/rfc/rfc2119>.

   [RFC8174]  Leiba, B., "Ambiguity of Uppercase vs Lowercase in RFC
              2119 Key Words", BCP 14, RFC 8174, DOI 10.17487/RFC8174,
              May 2017, <https://www.rfc-editor.org/rfc/rfc8174>.

8.2.  Informative References

   [OAUTH-BEARER]
              Jones, M. and D. Hardt, "The OAuth 2.0 Authorization
              Framework: Bearer Token Usage", RFC 6750,
              DOI 10.17487/RFC6750, October 2012,
              <https://www.rfc-editor.org/rfc/rfc6750>.

Appendix A.  Implementations

   This draft has a couple of implementations
   (https://github.com/cloudflareresearch/web-bot-auth)

   Clients:

   *  Chrome MV3

   *  Cloudflare Workers

   Servers:

   *  Caddy plugin

   *  Cloudflare Workers

   A demontstration server has been deployed to https://http-message-
   signatures-example.research.cloudflare.com/ (https://http-message-
   signatures-example.research.cloudflare.com/).

Meunier                  Expires 17 October 2025               [Page 11]
Internet-Draft      HTTP Message Signatures for Bots          April 2025

   It uses ed25519 example signing and verifying keys defined in
   Appendix B.1.4 of [HTTP-MESSAGE-SIGNATURES].

Acknowledgments

   TODO acknowledge.

Author's Address

   Thibault Meunier
   Cloudflare
   Email: ot-ietf@thibault.uk

Meunier                  Expires 17 October 2025               [Page 12]