Robots Exclusion Protocol Extension for URL Level Control

3.1.1. HTTP Response Header

The robots-tag field is a List as defined in [STRUCTURED-FIELD-VALUES]. Each member of the List is an Item representing a product token. Rules applicable to a product token are defined as Parameters of that Item. For historical reasons, implementors SHOULD also support the experimental field name X-Robots-Tag.¶

The product token is either a specific string as defined in Section 2.2.1 of [ROBOTSTXT] or the global identifier *. All rules defined in this specification are restrictive. The absence of a rule for a specific instruction constitutes as no instruction.¶

If a product token appears multiple times in the field, or if rules are provided for both a specific product token and the global * identifier, the implementor MUST apply the union of all restrictive rules. A restriction applied to the global * token applies to all accessors regardless of whether a specific product token is present.¶

For example, the following response header field specifies a global nosnippet rule for all accessors, and an additional noindex rule specifically for ExampleBot:¶

Robots-Tag: *;nosnippet, ExampleBot;noindex

The structured field in the examples is deserialized into the following objects:¶

"*" = [
        ["nosnippet", true]
      ],
"ExampleBot" = [
        ["noindex", true],
        ["nosnippet", true]
      ]

Implementors SHOULD impose a parsing limit on the field value to protect their systems. The parsing limit MUST be at least 8 kibibytes [KiB].¶

For a robots-tag field that exceeds the implementor's parsing limit, the implementor MUST process the data up to that limit. Any complete and valid List members found within the processed bytes MUST be honored. Any partially transmitted or truncated List member at the limit, and all subsequent bytes in that field, MUST be ignored. This ensures consistency with the processing of robots.txt files as specified in Section 2.1.1 of [ROBOTSTXT].¶

3.1.2. HTML Meta Element

For historical reasons the robots-tag header values may be specified by HTML service owners as an HTML meta tag. In case of the meta tag, the name attribute is used to specify the product token, and the content attribute to specify the comma separated robots-tag rules.¶

As with the header, the product token may be a global token, robots, which signifies that the rules apply to all requestors, or a specific product token applicable to a single requestor. For example:¶

<meta name="robots" content="noindex">
<meta name="examplebot" content="nosnippet">

Multiple robots meta elements may appear in a single HTML document. The implementor MUST apply the union of all rules found in all applicable elements. This includes rules specified for the global robots token and rules specified for the accessor's specific product token.¶

Because all rules specified in this document are restrictive, they are inherently additive. An accessor cannot "opt-out" of a restriction defined in a global 'robots' tag by being mentioned in a specific tag without that restriction. If any applicable tag contains a restrictive rule, that rule MUST be honored.¶

3.1.3. Robots controls rules

The possible values of the rules are:¶

• noindex - instructs the parser to not store the served data in its publicly accessible index.¶

• nosnippet - instructs the parser to not reproduce any stored data as an excerpt snippet.¶

The values are case insensitive.¶

Implementors may support other rules as specified in Section 2.2.4 of [ROBOTSTXT].¶

3.1.4. Caching of values

Implementors SHOULD link the validity of robots-tag rules to the freshness of the associated resource. Rules extracted from an HTTP response header or an HTML meta tag remain in effect until the resource is recrawled or the cached representation expires.¶

Implementors SHOULD determine the freshness of the rules using standard HTTP cache directives, such as Cache-Control or Expires, as defined in [HTTP-SEMANTICS]. In the absence of explicit cache directives, implementors MAY use heuristics to determine the refresh interval, typically matching the scheduled recrawl frequency of the resource.¶

If a resource is not recrawled due to a lack of perceived change, the last known robots-tag rules MUST continue to be honored.¶