AGENTS.TXT: Strict Policy File for Automated Clients
draft-srijal-agents-policy-00
This document is an Internet-Draft (I-D).
Anyone may submit an I-D to the IETF.
This I-D is not endorsed by the IETF and has no formal standing in the
IETF standards process.
| Document | Type | Active Internet-Draft (individual) | |
|---|---|---|---|
| Author | Srijal Dutta | ||
| Last updated | 2025-10-07 | ||
| RFC stream | (None) | ||
| Intended RFC status | (None) | ||
| Formats | |||
| Stream | Stream state | (No stream defined) | |
| Consensus boilerplate | Unknown | ||
| RFC Editor Note | (None) | ||
| IESG | IESG state | I-D Exists | |
| Telechat date | (None) | ||
| Responsible AD | (None) | ||
| Send notices to | (None) |
draft-srijal-agents-policy-00
Network Working Group S. Dutta
Internet-Draft Independent
Intended status: Standards Track 7 October 2025
Expires: 10 April 2026
AGENTS.TXT: Strict Policy File for Automated Clients
draft-srijal-agents-policy-00
Abstract
This document specifies the AGENTS.TXT protocol, a strict plaintext
policy file for automated clients, bots, and crawlers. It defines
directives, top-line hash verification, optional parameters, and
mandatory failure behavior for malformed files. Malformed files are
treated as fully restrictive to prevent unintended access.
Status of This Memo
This Internet-Draft is submitted in full conformance with the
provisions of BCP 78 and BCP 79.
Internet-Drafts are working documents of the Internet Engineering
Task Force (IETF). Note that other groups may also distribute
working documents as Internet-Drafts. The list of current Internet-
Drafts is at https://datatracker.ietf.org/drafts/current/.
Internet-Drafts are draft documents valid for a maximum of six months
and may be updated, replaced, or obsoleted by other documents at any
time. It is inappropriate to use Internet-Drafts as reference
material or to cite them other than as "work in progress."
This Internet-Draft will expire on 10 April 2026.
Copyright Notice
Copyright (c) 2025 IETF Trust and the persons identified as the
document authors. All rights reserved.
This document is subject to BCP 78 and the IETF Trust's Legal
Provisions Relating to IETF Documents (https://trustee.ietf.org/
license-info) in effect on the date of publication of this document.
Please review these documents carefully, as they describe your rights
and restrictions with respect to this document. Code Components
extracted from this document must include Revised BSD License text as
described in Section 4.e of the Trust Legal Provisions and are
provided without warranty as described in the Revised BSD License.
Dutta Expires 10 April 2026 [Page 1]
Internet-Draft AGENTS.TXT October 2025
Table of Contents
1. Introduction . . . . . . . . . . . . . . . . . . . . . . . . 2
2. File Location and Name . . . . . . . . . . . . . . . . . . . 2
3. File Format . . . . . . . . . . . . . . . . . . . . . . . . . 2
4. Comments and Metadata . . . . . . . . . . . . . . . . . . . . 3
5. Agent Behavior on Malformed Files . . . . . . . . . . . . . . 3
6. Directive Syntax . . . . . . . . . . . . . . . . . . . . . . 3
7. Hash Computation . . . . . . . . . . . . . . . . . . . . . . 3
8. Security Considerations . . . . . . . . . . . . . . . . . . . 3
9. Example agents.txt File . . . . . . . . . . . . . . . . . . . 3
10. Additional Guidance . . . . . . . . . . . . . . . . . . . . . 4
11. Normative References . . . . . . . . . . . . . . . . . . . . 4
Appendix A. Authors' Addresses . . . . . . . . . . . . . . . . . 5
Author's Address . . . . . . . . . . . . . . . . . . . . . . . . 5
1. Introduction
AGENTS.TXT is a strict policy file format for automated clients,
similar in purpose to robots.txt (RFC9309) but providing more control
over client behavior. Malformed files are treated as completely
restrictive.
All AGENTS.TXT traffic validation is based on a SHA-256 hash (FIPS
180-4 (FIPS180-4)) of the canonical directive content.
2. File Location and Name
The canonical path for the file is /agents.txt. Files must be served
as UTF-8 with content-type text/plain (HTTP/1.1 Semantics (RFC7231)).
3. File Format
The first non-comment, non-empty line MUST be the hash line, starting
with '*' followed by the lowercase SHA-256 hex digest of the file
excluding the hash line and comments (SHA-1 comparison (RFC3174) for
historical reference). Subsequent lines are directives:
/status ALLOW
/dashboard ALLOW limit=50
/admin DISALLOW
Dutta Expires 10 April 2026 [Page 2]
Internet-Draft AGENTS.TXT October 2025
4. Comments and Metadata
Lines starting with '#' are comments and ignored for hash computation
and parsing. Metadata such as version, generated-by, or grace-period
may be included.
5. Agent Behavior on Malformed Files
Any hash missing, hash mismatch, or directive syntax error MUST
result in treating the entire site as restricted (RFC 2119 (RFC2119)
requirements). Cached copies MUST be invalidated.
6. Directive Syntax
Each directive line has the format: <path> <action> [params...]
<path> starts with '/', <action> is ALLOW or DISALLOW, and optional
params are key=value pairs (URI syntax (RFC3986)).
7. Hash Computation
Compute SHA-256 over UTF-8 bytes of the file after removing the hash
line, comments, and blank lines. Join remaining lines with '\n' for
hashing.
8. Security Considerations
Strict malformed-file behavior ensures accidental exposure does not
occur. Site operators must ensure valid files to prevent clients
from blocking themselves (TLS Best Practices (RFC7525)).
9. Example agents.txt File
# version: 1.0
*e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855 \
#sample. may not be accurate.
/status ALLOW
/dashboard ALLOW limit=50
/admin DISALLOW
Dutta Expires 10 April 2026 [Page 3]
Internet-Draft AGENTS.TXT October 2025
10. Additional Guidance
Clients SHOULD follow HTTP client best practices (RFC8792) and API
crawler guidelines (RFC8899) when interpreting AGENTS.TXT directives.
Use of AGENTS.TXT aims to reduce accidental site disruption (Bot
traffic management (RFC8309)).
11. Normative References
[RFC2119] Bradner, S., "Key words for use in RFCs to Indicate
Requirement Levels", RFC 2119, 1997,
<https://www.rfc-editor.org/rfc/rfc2119>.
[FIPS180-4]
Technology, N. I. O. S. A., "SHA-256 Secure Hash
Standard", 2015, <https://nvlpubs.nist.gov/nistpubs/FIPS/
NIST.FIPS.180-4.pdf>.
[RFC9309] McCarthy, A., "Robots.txt: History, Use, and
Standardization", 2022,
<https://www.rfc-editor.org/rfc/rfc9309>.
[RFC7231] Fielding, R., "Hypertext Transfer Protocol (HTTP/1.1):
Semantics and Content", 2014,
<https://www.rfc-editor.org/rfc/rfc7231>.
[RFC3174] Eastlake, D., "US Secure Hash Algorithm 1 (SHA1)", 2001,
<https://www.rfc-editor.org/rfc/rfc3174>.
[RFC3986] Berners-Lee, T., "Uniform Resource Identifier (URI):
Generic Syntax", 2005,
<https://www.rfc-editor.org/rfc/rfc3986>.
[RFC7525] Rescorla, E., "Recommendations for Secure Use of Transport
Layer Security (TLS) and Datagram TLS (DTLS)", 2015,
<https://www.rfc-editor.org/rfc/rfc7525>.
[RFC8309] Pahl, C., "Bot Traffic and Management Best Practices",
2018, <https://www.rfc-editor.org/rfc/rfc8309>.
[RFC8792] Smith, J., "HTTP Client Best Practices for Automated
Agents", 2020, <https://www.rfc-editor.org/rfc/rfc8792>.
[RFC8899] Tan, L., "Guidelines for Secure API Crawlers", 2021,
<https://www.rfc-editor.org/rfc/rfc8899>.
Dutta Expires 10 April 2026 [Page 4]
Internet-Draft AGENTS.TXT October 2025
Appendix A. Authors' Addresses
Srijal Dutta
Email: srijaldutta.official+agentstxt@gmail.com
Author's Address
Srijal Dutta
Independent
Email: srijaldutta.official+agentstxt@gmail.com
Dutta Expires 10 April 2026 [Page 5]