Vocabulary for Expressing Content Preferences for AI Training

4.1. Permission

Basic indicators of whether content can be used for AI training.¶

• allow_training: Boolean¶

• restricted_training: public, non-commercial, internal, licensed¶

4.2. Purpose

Defines acceptable uses in training.¶

• purpose: String:¶

  • generation: Creating models that are capable of generating content¶

  • embedding: Converting content to vector representations¶

    • classification: Categorising or labelling content¶

    • summary: Creating condensed versions of content¶

    • paraphrase: Creating derivative versions of content¶

    • quotation: Repetition of a passage or fragment of original content¶

    • translation: Converting content between languages¶

4.3. Temporal Restrictions

Specifies the date range for training use.¶

• effective_date: ISO 8601 Date string¶

• expiration_date: ISO 8601 Date string¶

4.4. Content-Specific Granularity

Defines the scope of applicability. Refers to the level at which preferences apply within the content.¶

• scope: global, content-specific, conditional¶

4.5. Content Type

Specifies content types the preference applies to.¶

• mime_type: text, image, video, audio, application, [RFC2046].¶

4.6. Derivative Content

Allows or restricts derivatives like summaries.¶

• allow_derivatives: Boolean¶

• derivative_type: summary, paraphrase, translation¶

4.7. Data Retention

Defines content retention period post-training.¶

• retention_period: ISO 8601 Duration string (e.g., P3Y6M4DT12H30M5S)¶

4.8. Preference Persistence

Indicates if preferences should persist in derived datasets, or be optional. A derived dataset is the result of processing, transforming, or extracting information from the original source, such as aggregated statistics and summaries, or subsets of data.¶

• metadata_persistence: Boolean¶

4.9. Precedence

Conflicts should be resolved by assigning precedence values (e.g., high, medium, low) to rules, with a defined hierarchy that allows content producers to override publishers, domain operators, and others as necessary.¶

• precedence: Sets priority when preferences conflict with other layered preferences.¶

4.10. Geographic Restrictions

Specifies regions where preferences apply, ISO 3166-1.¶

• geo_limitations: Specifies geographic regions where training permissions apply.¶

5.1. HTTP Headers

Publishers can use HTTP headers to communicate AI-PREF preferences directly in response to client requests. This approach allows fine-grained control and easy integration into existing server configurations.¶

Example header:¶

AI-PREF: allow_training=true; purpose=generation,classification; retention_period=P3Y6M4DT12H30M5S

This header specifies that the content can be used for text generation and classification, with a retention period of 3 years, 6 months, 4 days, 12 hours, 30 minutes, and 5 seconds. The syntax and options should be carefully chosen to ensure compatibility with common web servers and clients.¶

5.2. Robots Exclusion Protocol (REP)

For publishers who already use REP (as defined in RFC9309), extending REP rules to include AI-PREF preferences could be beneficial.¶

Example rule:¶

User-agent: *
Allow-Training: non-commercial
Purpose: embedding, summarisation

This REP rule specifies that all user agents are allowed to use the content for non-commercial AI training, limited to embedding and summarisation purposes. Further extensions to REP could specify additional constraints, such as geographic limitations or temporal restrictions.¶

5.3. <meta> Tags for Sub-Document Level Control

To specify AI-PREF preferences at the level of individual HTML documents or specific parts of a document, <meta> tags and HTML attributes can be used.¶

Example <meta> tag:¶

<meta name="AI-PREF" content="allow_training=false; retention_period=0">

Example HTML attribute:¶

<div data-aipref="allow_training=false; retention_period=0">

The methods above specify that AI training is not allowed for the content of this document, with no retention period permitted. <meta> tags can be used to provide specific content preferences for a specific piece of content, and thus provide a flexible way to manage AI training signals at a more granular level.¶

5.4. “Well-Known” Locations

According to [RFC8615], “well-known” locations can serve metadata or configuration information that is easily discoverable by automated clients. AI-PREF preferences can be published at a “well-known” URL. There is already the Text and Data Mining Reservation Protocol (TDMRep) which has the same or overlapping intent.¶

Example:¶

https://example.com/.well-known/aipref

At this URL, a JSON or other structured format can specify AI-PREF preferences for the entire domain or specific content types.¶

Example JSON 1:¶

{
  "allow_training": false,
  "purpose": ["generation"],
  "retention_period": "0"
}

Example JSON 2:¶

{
  "version": "1.0",
  "resources": [
    {
      "path": "/videos/tutorial.mp4",
      "type": "video/mp4",
      "components": [
        {
          "name": "Introduction",
          "time-range": "00:00:00-00:01:00",
          "preferences": {
            "classification": "allowed",
            "embedding": "allowed"
          }
        },
        {
          "name": "Main Content",
          "time-range": "00:01:01-00:05:00",
          "preferences": {
            "generation": "prohibited",
            "summarization": "allowed"
          }
        }
      ]
    }
  ]
}

This approach simplifies discovery for automated clients and provides a centralised way to communicate content preferences across a domain.¶

TDMRep Example:¶

A rightsholder could expose a “well-known” TDMRep file at:¶

https://example.com/.well-known/tdmrep¶

Example TDMRep JSON Content:¶

{
  "version": "1.0",
  "license": "https://example.com/license",
  "contact": {
    "email": "tdm-support@example.com",
    "url": "https://example.com/contact"
  },
  "resources": [
    {
      "path": "/articles/",
      "type": "text/html",
      "restriction": "no-crawling"
    },
    {
      "path": "/api/data/",
      "type": "application/json",
      "restriction": "license-required"
    }
  ]
}

5.5. Embedded Metadata

Preferences for multimodal data can be embedded directly into file metadata (such as EXIF or XMP) as self-contained control signals. Compatibility and tamper resistance (e.g. signing) should be considered.¶

Example EXIF:¶

AI-Pref-Allow-Training: false
AI-Pref-Purpose: embedding
AI-Pref-Retention-Period: 0

Example PDF Metadata Using XMP:¶

<x:xmpmeta xmlns:x="adobe:ns:meta/">
   <rdf:RDF xmlns:rdf="http://www.w3.org/1999/02/22-rdf-syntax-ns#">
      <rdf:Description rdf:about=""
         xmlns:dc="http://purl.org/dc/elements/1.1/">
         <dc:Rights>Text mining allowed; Data sharing restricted</dc:Rights>
      </rdf:Description>
   </rdf:RDF>
</x:xmpmeta>

Preferences can be applied at the file level, or even to specific components (e.g., chapters in a PDF or frames in a video).¶

Example WEBVTT:¶

WEBVTT

00:00:00.000 --> 00:01:00.000
Usage Preferences: allow_training=true; purpose=generation,classification

00:01:01.000 --> 00:05:00.000
Usage Preferences: allow_training=false;

5.6. Content Credentials (ISO 22144)

TBD¶

5.7. ISCC (ISO 24138)

TBD¶

9.1. Normative References

[RFC2046]

Freed, N. and N. Borenstein, "Multipurpose Internet Mail Extensions (MIME) Part Two: Media Types", RFC 2046, DOI 10.17487/RFC2046, November 1996, <https://www.rfc-editor.org/rfc/rfc2046>.

[RFC2119]

Bradner, S., "Key words for use in RFCs to Indicate Requirement Levels", BCP 14, RFC 2119, DOI 10.17487/RFC2119, March 1997, <https://www.rfc-editor.org/rfc/rfc2119>.

[RFC8174]

Leiba, B., "Ambiguity of Uppercase vs Lowercase in RFC 2119 Key Words", BCP 14, RFC 8174, DOI 10.17487/RFC8174, May 2017, <https://www.rfc-editor.org/rfc/rfc8174>.

[RFC8615]

Nottingham, M., "Well-Known Uniform Resource Identifiers (URIs)", RFC 8615, DOI 10.17487/RFC8615, May 2019, <https://www.rfc-editor.org/rfc/rfc8615>.

[RFC9309]

Koster, M., Illyes, G., Zeller, H., and L. Sassman, "Robots Exclusion Protocol", RFC 9309, DOI 10.17487/RFC9309, September 2022, <https://www.rfc-editor.org/rfc/rfc9309>.

9.2. Informative References

[PURPOSE]

Illyes, G., "Robots Exclusion Protocol User Agent Purpose Extension", Work in Progress, Internet-Draft, draft-illyes-rep-purpose-00, 18 October 2024, <https://datatracker.ietf.org/doc/html/draft-illyes-rep-purpose-00>.