Protocol Extension for Advanced Automation Control
draft-liao-aipref-autoctl-ext-00
This document is an Internet-Draft (I-D).
Anyone may submit an I-D to the IETF.
This I-D is not endorsed by the IETF and has no formal standing in the
IETF standards process.
| Document | Type | Active Internet-Draft (individual) | |
|---|---|---|---|
| Author | Liao Peiyuan | ||
| Last updated | 2025-04-09 | ||
| RFC stream | (None) | ||
| Intended RFC status | (None) | ||
| Formats | |||
| Stream | Stream state | (No stream defined) | |
| Consensus boilerplate | Unknown | ||
| RFC Editor Note | (None) | ||
| IESG | IESG state | I-D Exists | |
| Telechat date | (None) | ||
| Responsible AD | (None) | ||
| Send notices to | (None) |
draft-liao-aipref-autoctl-ext-00
AI Preferences L. Peiyuan
Internet-Draft Condé Nast
Intended status: Experimental 8 April 2025
Expires: 10 October 2025
Protocol Extension for Advanced Automation Control
draft-liao-aipref-autoctl-ext-00
Abstract
This document specifies extensions to the automation-preferences.txt
protocol, providing advanced controls for server-side automation
permissions. It builds upon the core specification by adding
sophisticated features such as rate limiting, automation technology
restrictions, API permissions, session requirements, and HTML asset
annotations. These extensions enable content providers to exercise
more granular control over automated interactions while maintaining
backward compatibility with implementations of the core protocol.
About This Document
This note is to be removed before publishing as an RFC.
The latest revision of this draft can be found at
https://datatracker.ietf.org/doc/draft-liao-aipref-autoctl-ext/.
Status information for this document may be found at
https://datatracker.ietf.org/doc/draft-liao-aipref-autoctl-ext/.
Discussion of this document takes place on the AI Preferences Working
Group mailing list (mailto:ai-control@ietf.org), which is archived at
https://mailarchive.ietf.org/arch/browse/ai-control/. Subscribe at
https://www.ietf.org/mailman/listinfo/ai-control/.
Status of This Memo
This Internet-Draft is submitted in full conformance with the
provisions of BCP 78 and BCP 79.
Internet-Drafts are working documents of the Internet Engineering
Task Force (IETF). Note that other groups may also distribute
working documents as Internet-Drafts. The list of current Internet-
Drafts is at https://datatracker.ietf.org/drafts/current/.
Internet-Drafts are draft documents valid for a maximum of six months
and may be updated, replaced, or obsoleted by other documents at any
time. It is inappropriate to use Internet-Drafts as reference
material or to cite them other than as "work in progress."
Peiyuan Expires 10 October 2025 [Page 1]
Internet-Draft automation-preferences-ext April 2025
This Internet-Draft will expire on 10 October 2025.
Copyright Notice
Copyright (c) 2025 IETF Trust and the persons identified as the
document authors. All rights reserved.
This document is subject to BCP 78 and the IETF Trust's Legal
Provisions Relating to IETF Documents (https://trustee.ietf.org/
license-info) in effect on the date of publication of this document.
Please review these documents carefully, as they describe your rights
and restrictions with respect to this document. Code Components
extracted from this document must include Revised BSD License text as
described in Section 4.e of the Trust Legal Provisions and are
provided without warranty as described in the Revised BSD License.
Table of Contents
1. Introduction . . . . . . . . . . . . . . . . . . . . . . . . 2
1.1. Relationship to Core Specification . . . . . . . . . . . 3
2. Conventions and Terminology . . . . . . . . . . . . . . . . . 3
3. Extended Protocol Specification . . . . . . . . . . . . . . . 4
3.1. Rate Limiting . . . . . . . . . . . . . . . . . . . . . . 4
3.2. Automation Technology Restrictions . . . . . . . . . . . 4
3.3. API and XHR Permissions . . . . . . . . . . . . . . . . . 5
3.4. Session Requirements . . . . . . . . . . . . . . . . . . 6
3.5. HTML Asset Annotation . . . . . . . . . . . . . . . . . . 7
4. Backward Compatibility . . . . . . . . . . . . . . . . . . . 8
5. Implementation and Enforcement . . . . . . . . . . . . . . . 9
6. Security Considerations . . . . . . . . . . . . . . . . . . . 10
7. IANA Considerations . . . . . . . . . . . . . . . . . . . . . 10
8. Future Work . . . . . . . . . . . . . . . . . . . . . . . . . 10
9. References . . . . . . . . . . . . . . . . . . . . . . . . . 11
9.1. Normative References . . . . . . . . . . . . . . . . . . 11
9.2. Informative References . . . . . . . . . . . . . . . . . 11
Sample Extended automation-preferences.txt File . . . . . . . . . 11
Author's Address . . . . . . . . . . . . . . . . . . . . . . . . 13
1. Introduction
This document extends the automation-preferences.txt protocol defined
in "Protocol for Basic Automation Control" [CORE-SPEC] by introducing
advanced directives and capabilities for more sophisticated control
over automated interactions. These extensions address complex
automation scenarios while maintaining backward compatibility with
implementations of the core specification.
Peiyuan Expires 10 October 2025 [Page 2]
Internet-Draft automation-preferences-ext April 2025
The extensions defined in this document enable content providers to
exercise more granular control over automated access, including rate
limiting, specific technology restrictions, API usage policies,
session validation requirements, and asset-level annotation methods.
These capabilities are designed to complement the basic controls
provided by the core specification, offering a progressive path to
more comprehensive automation management.
1.1. Relationship to Core Specification
This document builds upon the core specification without modifying
its requirements. All directives and mechanisms defined in the core
specification remain valid and are not redefined here. This document
assumes familiarity with the core specification and uses its
terminology and concepts throughout.
The extensions defined in this document are OPTIONAL for both servers
and clients. Implementations that support only the core
specification are considered compliant with the automation-
preferences.txt protocol, though they will not benefit from the
advanced controls defined here.
When both core and extended directives are present in an automation-
preferences.txt file, parsers that do not support the extensions
defined in this document MUST ignore the unrecognized directives, as
specified in the core specification's extension mechanism.
2. Conventions and Terminology
The key words "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT",
"SHOULD", "SHOULD NOT", "RECOMMENDED", "NOT RECOMMENDED", "MAY", and
"OPTIONAL" in this document are to be interpreted as described in
BCP 14 [RFC2119] [RFC8174] when, and only when, they appear in all
capitals, as shown here.
This document uses the terminology defined in the automation-
preferences.txt protocol [CORE-SPEC]. The following additional terms
are introduced in this document:
* *Rate limiting*: Constraints on the frequency or concurrency of
automated requests to prevent excessive server load.
* *Automation technology*: Specific tools or frameworks used for
automation, such as headless browsers or browser automation
protocols.
* *XHR/Fetch*: XMLHttpRequest or Fetch API calls performed
programmatically.
Peiyuan Expires 10 October 2025 [Page 3]
Internet-Draft automation-preferences-ext April 2025
* *Session validation*: Mechanisms to verify that automated requests
are part of a legitimate user session.
* *Asset annotation*: Metadata embedded within HTML documents to
specify automation policies for individual content elements.
3. Extended Protocol Specification
This section defines additional directives that extend the
automation-preferences.txt protocol. These directives may be used
alongside the core directives in any group within the automation-
preferences.txt file.
3.1. Rate Limiting
Rate limiting directives specify constraints on the frequency and
concurrency of automated requests to prevent excessive server load.
The following directives are defined:
* RequestLimit: Specifies the maximum number of requests allowed
within a time period, expressed as a count followed by a time unit
(e.g., "60/minute"). Supported time units are "second", "minute",
"hour", and "day".
* ConcurrentLimit: Specifies the maximum number of concurrent
connections allowed from a single client.
Example:
RequestLimit: 60/minute
ConcurrentLimit: 5
Figure 1
Rate limiting directives apply to all requests within the scope of
the group, regardless of HTTP method. If no rate limiting directives
are specified, clients SHOULD NOT assume any specific rate limits,
but SHOULD implement reasonable self-throttling to avoid overloading
the server.
3.2. Automation Technology Restrictions
Automation technology directives specify whether specific automation
tools or frameworks are permitted. The following directives are
defined:
* AllowCDP: Boolean value indicating whether the use of Chrome
DevTools Protocol (CDP) is permitted.
Peiyuan Expires 10 October 2025 [Page 4]
Internet-Draft automation-preferences-ext April 2025
* AllowHeadless: Boolean value indicating whether the use of
headless browsers is permitted.
* AllowSelenium: Boolean value indicating whether the use of
Selenium WebDriver is permitted.
* AllowPuppeteer: Boolean value indicating whether the use of
Puppeteer is permitted.
* AllowPlaywright: Boolean value indicating whether the use of
Playwright is permitted.
Example:
AllowCDP: false
AllowHeadless: false
AllowSelenium: false
AllowPuppeteer: false
AllowPlaywright: false
Figure 2
If an automation technology directive is not specified, clients
SHOULD NOT assume that the use of that technology is permitted.
Implementations SHOULD respect these directives when applicable, even
if the specific detection methods may vary.
3.3. API and XHR Permissions
API and XHR permission directives specify rules for API usage and
automated use of XMLHttpRequest, Fetch, or AJAX. The following
directives are defined:
* APIAutomation: Indicates how API endpoints may be accessed by
automated clients. Valid values are:
- _none_: No API automation is permitted.
- _with-key-only_: API automation is permitted only with proper
authentication.
- _open_: API automation is generally permitted.
* AllowXHR: Indicates how XMLHttpRequest or Fetch API may be used by
automated clients. Valid values are:
- _none_: No XHR/Fetch automation is permitted.
Peiyuan Expires 10 October 2025 [Page 5]
Internet-Draft automation-preferences-ext April 2025
- _read-only_: Only GET requests are permitted via XHR/Fetch.
- _open_: XHR/Fetch automation is generally permitted.
* DisallowFetchFrom: Comma-separated list of URL patterns from which
automated XHR/Fetch requests are prohibited. Wildcards MAY be
used.
Example:
APIAutomation: with-key-only
AllowXHR: read-only
DisallowFetchFrom: /account/*, /checkout/*, /admin/*
Figure 3
If API and XHR permission directives are not specified, clients
SHOULD assume the most restrictive value (i.e., "none" for
APIAutomation and AllowXHR).
3.4. Session Requirements
Session requirement directives specify whether automated requests
must be part of a legitimate user session. The following directives
are defined:
* RequireHumanInitiatedSession: Boolean value indicating whether
automated requests must be part of a session that was initiated by
a human user.
* SessionValidation: Specifies the method used to validate sessions.
Valid values are:
- _cookie-based_: Sessions are validated using HTTP cookies.
- _token-based_: Sessions are validated using authentication
tokens.
- _oauth_: Sessions are validated using OAuth.
- _none_: No session validation is required.
* SessionTTL: Specifies the maximum time-to-live for a session,
expressed as a duration (e.g., "30m", "2h", "1d").
* RequireUserAgent: Boolean value indicating whether automated
requests must include a valid User-Agent header.
Peiyuan Expires 10 October 2025 [Page 6]
Internet-Draft automation-preferences-ext April 2025
Example:
RequireHumanInitiatedSession: true
SessionValidation: cookie-based
SessionTTL: 1h
RequireUserAgent: true
Figure 4
If session requirement directives are not specified, clients SHOULD
NOT assume any specific session requirements, but SHOULD include a
valid User-Agent header in all requests.
3.5. HTML Asset Annotation
In addition to a site-level automation-preferences.txt file,
automation preferences MAY be embedded directly within HTML documents
to annotate individual assets. This mechanism enables content
creators to specify fine-grained automation policies for particular
content items.
Authors SHOULD use structured data markup using JSON-LD embedded in a
<script> element. The JSON object SHOULD use a defined type (e.g.,
"AutomationPolicyAnnotation") and include relevant fields that mirror
those used in automation-preferences.txt.
Note that unlike site-wide directives, asset-level annotations SHOULD
NOT include HTTP method restrictions, request limits, or concurrency
limits, as these concepts apply to endpoints and services rather than
to individual content assets.
Example:
<script type="application/ld+json">
{
"@context": "https://schema.org",
"@type": "AutomationPolicyAnnotation",
"automationPolicy": "limited",
"allowCDP": false,
"allowHeadless": false,
"automationPurpose": {
"require": true,
"allowed": [[PLACEHOLDER_PURPOSE1], [PLACEHOLDER_PURPOSE2]],
"disallowed": [[PLACEHOLDER_PURPOSE3]]
},
"contactEmail": "automation-policy@example.com"
}
</script>
Peiyuan Expires 10 October 2025 [Page 7]
Internet-Draft automation-preferences-ext April 2025
Figure 5
When both a automation-preferences.txt file and HTML asset
annotations are present, the more specific rule (typically the HTML
annotation) SHALL be applied to the corresponding content asset.
Clients supporting HTML asset annotations SHOULD parse and respect
these annotations when present.
The annotation schema MAY include any directives defined in the core
or extension specifications. Fields in the annotation SHOULD use
camelCase naming to align with JSON-LD conventions, while maintaining
semantic equivalence to the corresponding directives in the
automation-preferences.txt file.
4. Backward Compatibility
The extensions defined in this document maintain backward
compatibility with implementations of the core specification. This
compatibility is achieved through the following mechanisms:
* All directives defined in this document are OPTIONAL.
Implementations that support only the core specification can
safely ignore these directives, as specified in the core
specification's extension mechanism.
* The extensions do not modify or override the behavior of any
directives defined in the core specification.
* Extended directives enhance but do not replace core functionality.
Implementations supporting these extensions SHOULD degrade gracefully
when interacting with servers or clients that support only the core
specification:
* Servers supporting extensions SHOULD still process all core
directives correctly, even if extended directives are also
present.
* Clients supporting extensions SHOULD still honor all core
directives, even if they do not recognize extended directives in a
file.
* When HTML asset annotations are not supported by a client, the
client SHOULD fall back to the site-level automation-
preferences.txt file for guidance.
Peiyuan Expires 10 October 2025 [Page 8]
Internet-Draft automation-preferences-ext April 2025
This approach ensures that the introduction of extensions does not
break existing implementations while providing a path for enhanced
functionality.
5. Implementation and Enforcement
Servers implementing the extensions defined in this document SHOULD:
* Employ detection mechanisms (e.g., CDP fingerprinting, headless
browser detection) to identify automated clients using specific
technologies.
* Implement rate limiting according to the specified directives.
* Validate sessions as required by the session requirement
directives.
* Process HTML asset annotations when interpreting automation
policies for specific content.
* Respond with appropriate HTTP status codes for non-compliant
requests, such as:
- 429 Too Many Requests for rate limit violations.
- 403 Forbidden for unauthorized automation technology use.
- 401 Unauthorized for missing or invalid authentication.
Clients supporting these extensions SHOULD:
* Honor rate limiting directives by self-throttling requests.
* Respect automation technology restrictions by avoiding prohibited
tools.
* Adhere to API and XHR permissions as specified.
* Establish and maintain valid sessions when required.
* Parse and respect HTML asset annotations when present.
Both servers and clients MAY implement additional detection and
enforcement mechanisms beyond those explicitly described in this
document, as long as they maintain compatibility with the specified
directives.
Peiyuan Expires 10 October 2025 [Page 9]
Internet-Draft automation-preferences-ext April 2025
6. Security Considerations
In addition to the security considerations mentioned in the core
specification, the extensions defined in this document introduce the
following considerations:
* *Rate Limiting*: Implementations of rate limiting SHOULD use
secure methods to track request counts and prevent circumvention
through IP spoofing or other means.
* *Technology Detection*: Methods used to detect specific automation
technologies MAY be circumvented by sophisticated clients.
Servers SHOULD employ multiple detection approaches and adapt to
evolving evasion techniques.
* *Session Validation*: Session validation mechanisms SHOULD be
resistant to replay attacks and session hijacking attempts.
* *HTML Asset Annotations*: Parsing of JSON-LD annotations MUST be
performed securely to prevent injection attacks or denial-of-
service through malformed input.
The extensions provide more granular control over automated access,
which can enhance security, but they also introduce complexity that
may lead to misconfiguration. Implementers SHOULD carefully test and
validate their configurations to ensure they provide the intended
protections.
7. IANA Considerations
This document has no IANA actions.
8. Future Work
Future enhancements to the automation-preferences.txt protocol MAY
include:
* Soliciting further feedback from browser vendors, content owners,
AI model and automation tool developers.
* Developing reference implementations and comprehensive detection
libraries.
* Formalizing the protocol in collaboration with the IETF and W3C.
* Expanding interoperability with related protocols for consistent
content preference signaling across the web.
Peiyuan Expires 10 October 2025 [Page 10]
Internet-Draft automation-preferences-ext April 2025
* Standardizing the HTML asset annotation schema through formal
registration with schema.org or similar organizations.
9. References
9.1. Normative References
[CORE-SPEC]
Liao, P., "automation-preferences.txt Protocol for Basic
Automation Control", Work in Progress, Internet-Draft,
draft-liao-aipref-autoctl-core-00, April 2025,
<https://datatracker.ietf.org/doc/html/draft-liao-aipref-
autoctl-core-00>.
[RFC2119] Bradner, S., "Key words for use in RFCs to Indicate
Requirement Levels", BCP 14, RFC 2119,
DOI 10.17487/RFC2119, March 1997,
<https://www.rfc-editor.org/rfc/rfc2119>.
[RFC8174] Leiba, B., "Ambiguity of Uppercase vs Lowercase in RFC
2119 Key Words", BCP 14, RFC 8174, DOI 10.17487/RFC8174,
May 2017, <https://www.rfc-editor.org/rfc/rfc8174>.
9.2. Informative References
[RFC9309] Koster, M., Illyes, G., Zeller, H., and L. Sassman,
"Robots Exclusion Protocol", RFC 9309,
DOI 10.17487/RFC9309, September 2022,
<https://www.rfc-editor.org/rfc/rfc9309>.
Sample Extended automation-preferences.txt File
The following is an example of a automation-preferences.txt file that
includes both core and extended directives:
Peiyuan Expires 10 October 2025 [Page 11]
Internet-Draft automation-preferences-ext April 2025
# Automation preferences for example.com
# Version: 2.0
# Last updated: 2025-04-08
# Group 1: Applies to the entire site
Host: example.com
Scope: /
AutomationPolicy: limited
AllowedMethods: GET, HEAD
DisallowedMethods: POST, PUT, DELETE, PATCH
RequireAutomationPurpose: true
AllowedPurposes: [PLACEHOLDER_PURPOSE1], [PLACEHOLDER_PURPOSE2]
DisallowedPurposes: [PLACEHOLDER_PURPOSE3]
ContactEmail: automation-policy@example.com
# Extended directives
RequestLimit: 60/minute
ConcurrentLimit: 5
AllowCDP: false
AllowHeadless: false
AllowSelenium: false
AllowPuppeteer: false
AllowPlaywright: false
APIAutomation: with-key-only
RequireUserAgent: true
AllowXHR: read-only
DisallowFetchFrom: /account/*, /checkout/*, /admin/*
RequireHumanInitiatedSession: true
SessionValidation: cookie-based
SessionTTL: 1h
# Group 2: Specific preferences for the /admin/ path
Host: example.com
Scope: /admin/
AutomationPolicy: strict
AllowedMethods: GET
DisallowedMethods: POST, PUT, DELETE, PATCH
AllowedPurposes: [PLACEHOLDER_PURPOSE1]
DisallowedPurposes: [PLACEHOLDER_PURPOSE2], [PLACEHOLDER_PURPOSE3]
# Extended directives for admin path
RequestLimit: 10/minute
ConcurrentLimit: 2
RequireHumanInitiatedSession: true
SessionValidation: token-based
SessionTTL: 30m
Figure 6
Peiyuan Expires 10 October 2025 [Page 12]
Internet-Draft automation-preferences-ext April 2025
Author's Address
Liao Peiyuan
Condé Nast
United States of America
Email: peiyuan_liao@condenast.com
Peiyuan Expires 10 October 2025 [Page 13]