ICAP Trailers
draft-rousskov-icap-trailers-00

Network Working Group                                        A. Rousskov
Internet-Draft                                   The Measurement Factory
Updates: 3507 (if approved)                               August 4, 2016
Intended status: Informational
Expires: February 5, 2017

                             ICAP Trailers
                    draft-rousskov-icap-trailers-00

Abstract

   This document defines an ICAP trailer feature which allows ICAP
   agents to reliably send message metadata after the message body.  The
   ICAP trailer is independent from the HTTP trailer that might also be
   encapsulated in an ICAP message.  ICAP changes defined here are
   backward compatible and address a long-standing ICAP specification
   errata entry.

Status of This Memo

   This Internet-Draft is submitted in full conformance with the
   provisions of BCP 78 and BCP 79.

   Internet-Drafts are working documents of the Internet Engineering
   Task Force (IETF).  Note that other groups may also distribute
   working documents as Internet-Drafts.  The list of current Internet-
   Drafts is at http://datatracker.ietf.org/drafts/current/.

   Internet-Drafts are draft documents valid for a maximum of six months
   and may be updated, replaced, or obsoleted by other documents at any
   time.  It is inappropriate to use Internet-Drafts as reference
   material or to cite them other than as "work in progress."

   This Internet-Draft will expire on February 5, 2017.

Copyright Notice

   Copyright (c) 2016 IETF Trust and the persons identified as the
   document authors.  All rights reserved.

   This document is subject to BCP 78 and the IETF Trust's Legal
   Provisions Relating to IETF Documents
   (http://trustee.ietf.org/license-info) in effect on the date of
   publication of this document.  Please review these documents
   carefully, as they describe your rights and restrictions with respect
   to this document.  Code Components extracted from this document must
   include Simplified BSD License text as described in Section 4.e of

Rousskov                Expires February 5, 2017                [Page 1]
Internet-Draft                ICAP Trailers                  August 2016

   the Trust Legal Provisions and are provided without warranty as
   described in the Simplified BSD License.

Table of Contents

   1.  Motivation and Design Choices . . . . . . . . . . . . . . . .   2
   2.  Use Cases . . . . . . . . . . . . . . . . . . . . . . . . . .   4
   3.  Overall Operation . . . . . . . . . . . . . . . . . . . . . .   4
   4.  Notations . . . . . . . . . . . . . . . . . . . . . . . . . .   5
   5.  Extended Use of the Allow Header Field  . . . . . . . . . . .   5
   6.  Message Syntax  . . . . . . . . . . . . . . . . . . . . . . .   6
   7.  Trailer Field Syntax  . . . . . . . . . . . . . . . . . . . .   6
   8.  Client Requirements . . . . . . . . . . . . . . . . . . . . .   7
   9.  Server Requirements . . . . . . . . . . . . . . . . . . . . .   8
   10. Examples  . . . . . . . . . . . . . . . . . . . . . . . . . .   9
   11. Security Considerations . . . . . . . . . . . . . . . . . . .  11
   12. Normative References  . . . . . . . . . . . . . . . . . . . .  12
   Author's Address  . . . . . . . . . . . . . . . . . . . . . . . .  12

1.  Motivation and Design Choices

   ICAP [RFC3507] specification says that the Trailer header field is
   "defined in ICAP the same as [...] in HTTP".  Unfortunately, that
   phrase alone is not enough for trailer-related interoperability in
   the ICAP context because of the following conflicting
   interpretations, requirements, and needs:

   o  Both ICAP and HTTP message headers might contain a Trailer field.

   o  HTTP messages might contain HTTP trailers (that ICAP servers could
      be interested in receiving or even sending).  An HTTP trailer can
      be present with or without an HTTP Trailer header field.

   o  ICAP agents need to distinguish an HTTP trailer from an ICAP
      trailer.

   o  HTTP uses Chunked Transfer Coding [RFC7230] to transmit trailers.
      The chunked coding is applied to the entire HTTP message body.
      This choice places an HTTP trailer inside an HTTP message body.

   o  It is possible to interpret the ICAP specification as placing the
      ICAP trailer either inside the HTTP message body or inside the
      ICAP message body.

   o  ICAP does not chunk-encode ICAP message bodies.  Instead, ICAP
      message bodies contain a combination of zero, one, or two HTTP
      headers possibly followed by a chunked-encoded HTTP message body.

Rousskov                Expires February 5, 2017                [Page 2]
Internet-Draft                ICAP Trailers                  August 2016

   o  Chunked coding does not support multiple trailers: Chunked HTTP
      messages always contain exactly one (but possibly empty) trailer
      part.

   o  HTTP effectively restricts trailers usage to messages with bodies,
      presumably because, without a body, the information in the trailer
      can usually be placed in the message header.  In ICAP context, it
      is not obvious whether trailers ought to be restricted to messages
      with HTTP bodies (embedded in ICAP bodies) or to messages with
      ICAP bodies (that might only contain HTTP headers and no HTTP
      body).

   These problems led to a ban on ICAP trailers [Errata].

   Several designs were considered for introducing proper ICAP trailers
   support:

   1.  Extend chunked coding to support multiple trailers (one for HTTP
       and one for ICAP).  This option was rejected because many ICAP
       agents use existing HTTP-focused libraries to parse embedded HTTP
       bodies.  As anecdotal evidence related to the ICAP-only "ieof"
       chunk extension support shows, it would be difficult to extend
       those libraries to handle a complicated ICAP-only extension.
       Also, this design would make it difficult to send an ICAP trailer
       when processing large HTTP messages without bodies.

   2.  Embed ICAP trailer fields inside the chunked HTTP message body
       trailer, using an ICAP-specific field name prefix (e.g.,
       "ICAP-").  This option was rejected because it would either allow
       malicious HTTP messages to inject ICAP trailers or require ICAP
       clients to hide conflicting HTTP trailer fields from the ICAP
       server.  This design also badly violates layering boundaries by
       mixing HTTP- and ICAP-level information in the same protocol
       structure.

   3.  Extend Encapsulated header with a "trailer" token.  This option
       was rejected because the Encapsulated header describes embedded
       HTTP message parts and an ICAP trailer is not a part of any HTTP
       message.  In other words, ICAP trailers do not get encapsulated.

   4.  Clarify the ICAP Trailer semantics (and transfer mechanism)
       without introducing any new trailer support-negotiation
       mechanism.  This option was rejected because trailers affect
       message framing and many existing ICAP agent implementations
       cannot parse any form of trailers.

   5.  Add a new trailer support-negotiation mechanism (e.g., "Allow:
       trailers") and a new trailer presence-signaling mechanism (e.g.,

Rousskov                Expires February 5, 2017                [Page 3]
Internet-Draft                ICAP Trailers                  August 2016

       Trailer2) while leaving the poorly defined Trailer header
       semantics as is.  This option was narrowly rejected because a new
       trailer support-negotiation mechanism alone was deemed sufficient
       to resolve conflicts between this specification and any
       reasonable existing implementation of the poorly defined Trailer
       semantics.

   6.  Add a new trailer support-negotiation mechanism and only clarify
       Trailer header semantics (and transfer mechanism) upon successful
       negotiation, while reusing the well-known Trailer header field
       name as the trailer presence-signaling mechanism (requiring
       successful support negotiation).  This specification documents
       this design.

2.  Use Cases

   Trailers allow an ICAP agent to transmit metadata after the message
   body.  Such delayed transmission is useful when the same information
   was not available at the start of the message transmission.  For
   example:

   o  A client uses an ICAP trailer to relay the current HTTP/1.1
      connection or HTTP/2 stream status after transmitting a large HTTP
      message.  A server uses that information to optimize message
      analysis (e.g., skip or abort analysis of HTTP requests sent by
      already disconnected HTTP clients).

   o  A server uses an ICAP trailer to relay audit information about
      viruses present at the end of a large HTTP message.

   o  A server uses an ICAP trailer to relay prefetching information
      about HTML parts referenced from a large HTTP message.

3.  Overall Operation

   This section informally describes the overall feature operation.
   This description is deliberately imprecise and cannot be used to
   build compliant implementations.  The following sections contain
   actual protocol requirements.

   To announce feature support, ICAP agents exchange "Allow: trailers"
   settings during an OPTIONS transaction.  To send a trailer at the end
   of a particular REQMOD or RESPMOD transaction, the agent first sends
   both "Allow: trailers" and "Trailer" header fields.  The Trailer
   field lists header field names expected in the message trailer
   section.  After sending the entire ICAP message body, the agent sends
   the trailer section (a.k.a. "trailer").  The trailer section is
   syntactically equivalent to the ICAP message header section.  The

Rousskov                Expires February 5, 2017                [Page 4]
Internet-Draft                ICAP Trailers                  August 2016

   trailer section does not have to contain any of the promised fields
   and might even have no fields at all.

   A trailer makes sense only in an ICAP message with a body.  However,
   a trailer could be sent if the ICAP message body encapsulates just
   HTTP headers.

   If a trailer is sent, its bytes are always the last bytes sent during
   the entire ICAP transaction.  Thus, a client never sends a trailer at
   the end of Preview unless it sent the "ieof" chunk extension as well.
   Similarly, a server never a sends trailer with a 100 (Continue)
   control message.

4.  Notations

   The key words "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT",
   "SHOULD", "SHOULD NOT", "RECOMMENDED", "MAY", and "OPTIONAL" in this
   document are to be interpreted as described in [RFC2119].
   Conformance criteria and error handling considerations are defined in
   Section 2.5 of [RFC7230].

   This specification uses the Augmented Backus-Naur Form (ABNF)
   notation of [RFC5234] with a list extension defined in Section 7 of
   [RFC7230].  All syntax rules not explicitly defined in this
   specification (e.g., header-field and CRLF) are defined in (or
   included by reference from) [RFC7230].

   The "Allow/X" notation is defined in Section 5.

5.  Extended Use of the Allow Header Field

   The use of the Allow header field defined in sections 4.6 and 4.10.2
   of [RFC3507] is extended while preserving its original syntax and
   general semantics (i.e., a comma-separated list of tokens, each
   identifying an ICAP feature supported by the sending agent):

      Allow = 1#token

   [RFC3507] defines Allow usage in OPTIONS requests and responses.
   This specification extends Allow usage to other messages and adds
   "trailers" to the list of possible Allow value tokens.

   An agent MUST treat multiple Allow header fields as one Allow field
   with a comma-separated list of individual field value tokens,
   concatenated in the order of their appearance in the ICAP header.  An
   ICAP agent MUST ignore Allow header tokens it does not understand.
   This document does not specify the significance of Allow tokens order
   and impact of repeated tokens.

Rousskov                Expires February 5, 2017                [Page 5]
Internet-Draft                ICAP Trailers                  August 2016

   An agent MAY send an Allow header in any message.  Such header
   contains a list of ICAP features supported by the sending agent.  For
   example, a client could send an "Allow: trailers" in a REQMOD request
   and receive an "Allow: 204, trailers, 206" header field in response.
   The exact meaning of each Allow token in a context of its message is
   defined by the corresponding feature specification.  By sending
   Allow, the agent indicates compliance with each listed feature's
   specification but does not necessarily commit to offer or use any of
   the listed features during future transactions.

   To keep this specification succinct, we introduce "Allow/X" notation
   to mean an ICAP Allow header field which value contains an "X" token,
   possibly among other tokens.  For example, "Allow/trailers" stands
   for an Allow header field with a "trailers" token and possibly other
   tokens.  This notation covers multi-field Allow headers as well
   because they are equivalent to a combined single-field Allow header.

   The same approach to extending Allow header usage was successfully
   applied to the ICAP 206 extension (XXX: reference our expired ICAP
   icap-ext-partial-content draft).  This specification is compatible
   with the ICAP 206 extension.

6.  Message Syntax

   An ICAP message with a trailer is a concatenation of a regular ICAP
   message and an trailer section.  The trailer section syntax is
   identical to the ICAP header syntax:

         ICAP-message-without-trailer = <see [RFC3507]>
         ICAP-message-with-trailer    = ICAP-message-without-trailer
                                        trailer-section

         trailer-section = *( trailer-field CRLF )
                           CRLF
         trailer-field   = header-field

   Note that any trailer, even a trailer without fields, ends with CRLF.
   That terminating sequence is essential for proper message framing on
   persistent ICAP connections.

   A sender MUST NOT generate a trailer section that contains a field
   necessary for message framing (e.g., Encapsulated, Preview, and
   Trailer), routing (e.g., Host), or authentication.

7.  Trailer Field Syntax

   The ICAP Trailer header value syntax is identical to the HTTP Trailer
   syntax [RFC7230]:

Rousskov                Expires February 5, 2017                [Page 6]
Internet-Draft                ICAP Trailers                  August 2016

      Trailer = 1#field-name

   A Trailer header field sender SHOULD enumerate the names of all
   expected trailer fields.  This a priori knowledge of trailer fields
   might help the recipient with trailer processing (e.g., certain
   message annotation actions could be delayed in anticipation of the
   trailer section).  However, enumerating all expected trailer fields
   can be impractical or even impossible in some environments.  A
   Trailer sender MAY send a trailer section with a set of field names
   that differs from the set of field names listed in the Trailer header
   field.

   This specification does not place any restriction on the order of
   field names in the Trailer header field.  Senders SHOULD NOT generate
   duplicate names for the Trailer header field.

8.  Client Requirements

   A client compliant with this specification SHOULD send Allow/trailers
   in each OPTIONS request.  A non-authenticating server cannot be
   expected to mark an Allow/trailers-sending client specially, but this
   support announcement requirement is meant to minimize
   interoperability problems associated with servers sending Allow/
   trailers in OPTIONS responses.  Some servers that do not support
   trailers might not be able to ignore Allow/trailers in OPTIONS
   requests; therefore, a client SHOULD offer a configuration option or
   other means of disabling sending Allow/trailers in OPTIONS requests.

   An ICAP service sending Allow/trailers in OPTIONS response is called
   a trailers-supporting service.  That service designation, maintained
   by the client, starts upon receiving the OPTIONS service response
   carrying Allow/trailers and lasts until OPTIONS expiration or a new
   OPTIONS response from that service.

   A client compliant with this specification SHOULD send Allow/trailers
   in each request to a trailers-supporting service.  Doing so allows
   the service to respond with a trailer (and is also necessary for
   sending a client trailer, as detailed further below).

   A client receiving both a Trailer header field and Allow/trailers in
   the response MUST expect a trailer section in that response.  In all
   other cases, a client MUST use the usual trailer-free ICAP response
   syntax.  A client receiving a Trailer header field without Allow/
   trailers in a response MAY treat the response as syntactically
   malformed and MUST NOT reuse the connection for any other messages
   (including pending pipelined requests, if any).

Rousskov                Expires February 5, 2017                [Page 7]
Internet-Draft                ICAP Trailers                  August 2016

   A client MAY send a trailer in any request that satisfies all of
   these conditions:

   1.  the request is sent to a trailers-supporting service;

   2.  the request has a body.

   A client MUST NOT send a trailer in any other request.

   To send a trailer, the client MUST send Allow/trailers and a Trailer
   header field in the same request.  A client MUST NOT send a
   combination of those two header fields without sending a trailer.

9.  Server Requirements

   A server compliant with this specification SHOULD send Allow/trailers
   in each successful response to an OPTIONS request carrying Allow/
   trailers.  Although ICAP/1.0 [RFC3507] allows a list of features in
   the Allow header, some ICAP clients might not be able to handle an
   Allow header other than "Allow: 204"; therefore, a server SHOULD NOT
   send Allow/trailers in a response to a request without Allow/
   trailers.

   A server compliant with this specification MAY send Allow/trailers in
   a response without a trailer to a request with Allow/trailers.  The
   client receiving a no-trailer REQMOD or RESPMOD response with Allow/
   trailers ought to ignore Allow/trailers.  Nevertheless, the server is
   allowed to respond with Allow/trailers in this context because doing
   so might simplify server implementation and configuration.

   A server receiving both a Trailer header field and Allow/trailers in
   the request MUST expect a trailer in that request.  In all other
   cases, a server MUST use the usual trailer-free ICAP request syntax.
   A server receiving a Trailer header field without Allow/trailers in a
   request MAY treat the request as syntactically malformed and MUST NOT
   reuse the connection for any future requests.

   A server MAY send a trailer in any response that satisfies all of
   these conditions:

   1.  the response is for a request containing Allow/trailers;

   2.  the response has a body.

   A server MUST NOT send a trailer in any other response.

Rousskov                Expires February 5, 2017                [Page 8]
Internet-Draft                ICAP Trailers                  August 2016

   To send a trailer, the server MUST send Allow/trailers and a Trailer
   header field in the same response.  A server MUST NOT send a
   combination of those two header fields without sending a trailer.

10.  Examples

   The following examples illustrate trailer exchanges between ICAP
   agents compliant with this specification.  To show important details,
   all CRLF sequences after major message parts are shown as "\r\n"
   lines.  CRLF sequences at the end of other lines are implied.

   All REQMOD and RESPMOD messages in these examples share the context
   of the following OPTIONS handshake.

   Figure 1 shows an OPTIONS request and response when both the client
   and the server are compliant with this specification.  The client
   also supports the ICAP 206 extension, but the server does not.

      OPTIONS icap://example.net/sample-service ICAP/1.0
      Host: example.net
      User-Agent: Example-ICAP-Client/1.2
      Allow: 204, trailers, 206
      \r\n
      ICAP/1.0 200 OK
      Date: Sat, 13 Aug 2016 12:17:21 GMT
      Methods: RESPMOD
      Service: FOO Tech Server 1.0
      ISTag: "W3E4R7U9-L2E4-2"
      Encapsulated: null-body=0
      Max-Connections: 1000
      Options-TTL: 7200
      Allow: 204
      Allow: trailers
      Preview: 0
      Transfer-Complete: asp, bat, exe, com
      Transfer-Ignore: html
      Transfer-Preview: *
      \r\n

                        Figure 1: OPTIONS handshake

   The OPTIONS response in Figure 1 contains two Allow headers to
   illustrate one of several possible implementations.  A compliant
   server can also send a single Allow header with a list of values,
   just like the client does in the above example.

   Figure 2 is a RESPMOD request with an ICAP trailer that the client
   can send after receiving an OPTIONS response with Allow/trailers

Rousskov                Expires February 5, 2017                [Page 9]
Internet-Draft                ICAP Trailers                  August 2016

   shown in Figure 1.  Note that Trailer field value does not match the
   actual fields in the trailer.

      RESPMOD icap://example.net/sample-service ICAP/1.0
      Host: example.net
      User-Agent: Example-ICAP-Client/1.2
      Allow: 204, trailers
      Trailer: TBD
      Encapsulated: req-hdr=0, res-hdr=131, res-body=292
      \r\n
      GET /origin-resource HTTP/1.1
      Host: www.example.com
      Accept: text/html, text/plain, image/gif
      Accept-Encoding: gzip, compress
      \r\n
      HTTP/1.1 200 OK
      Date: Sat, 13 Aug 2016 12:17:22 GMT
      Server: Testserver/1.0 (Unix)
      ETag: "63840-1ab7-378d415b"
      Content-Type: text/html
      Content-Length: 24
      \r\n
      18
      Origin server sent this.
      0
      \r\n
      X-Client-Log-Lineno: 15612570
      X-Client-Status: disconnected (at 1470262108)
      \r\n

                         Figure 2: Request Trailer

   Figure 3 is a RESPMOD response with an HTTP and ICAP trailers.  The
   ICAP trailer contains X-Threat-Found and Connection field.  To
   respond with an ICAP trailer, the server ought to receive Allow/
   trailers in the corresponding ICAP request.

Rousskov                Expires February 5, 2017               [Page 10]
Internet-Draft                ICAP Trailers                  August 2016

      ICAP/1.0 200 OK
      Date: Sat, 13 Aug 2016 12:17:23 GMT
      Server: ICAP-Server-Software/1.0
      Connection: keep-alive
      ISTag: "W3E4R7U9-L2E4-2"
      Encapsulated: res-hdr=0, res-body=276
      Allow: trailers
      Trailer: X-ICAP-Log-Lineno, X-Threat-Found
      \r\n
      HTTP/1.1 200 OK
      Date: Sat, 13 Aug 2016 12:17:22 GMT
      Server: Testserver/1.0 (Unix)
      ETag: "63840-1ab7-378d415b"
      Content-Type: text/html
      Content-Length: 24
      Trailer: X-Content-Checksum
      \r\n
      18
      Origin server sent this.
      0
      X-Content-Checksum: sha-short=366bc24373b
      \r\n
      X-Threat-Found: Type=0; Resolution=1; Threat=plain-text
      Connection: close
      \r\n

                          Figure 3: Two Trailers

   XXX: Polish and validate offsets, lengths, etc.  TODO: Figure out how
   to keep figure descriptions and figures on the same page without
   inserting empty space into HTML rendering of the draft.

11.  Security Considerations

   Proper trailer support reduces old ICAP [RFC3507] security concerns
   because implementations unaware of trailer complexities are arguably
   more likely to misbehave when receiving HTTP or ICAP trailers.

   Sending request headers with multiple Allow tokens could crash poor-
   quality ICAP servers unaware of this specification.  Trailer support
   negotiation rules partially mitigate that risk by restricting unaware
   implementations exposure; such implementations are exposed only
   during OPTIONS exchanges.  Since OPTIONS transaction has to precede
   any HTTP message processing, and since virtually all ICAP client-
   server relationships are stable, most poor-quality implementations
   would be detected early and reliably.

Rousskov                Expires February 5, 2017               [Page 11]
Internet-Draft                ICAP Trailers                  August 2016

   A naive implementation of trailer interpretation logic might update
   an already "frozen" or "committed" (at header parsing time) state of
   the ICAP transaction or connection, resulting in crashes and other
   problems.  For example, such an implementation could panic after
   discovering a Connection trailer field value that contradicts the
   Connection header field value that has already been received and
   processed at the beginning of the same transaction.

12.  Normative References

   [RFC2119]  Bradner, S., "Key words for use in RFCs to Indicate
              Requirement Levels", BCP 14, RFC 2119, DOI 10.17487/
              RFC2119, March 1997,
              <http://www.rfc-editor.org/info/rfc2119>.

   [RFC3507]  Elson, J. and A. Cerpa, "Internet Content Adaptation
              Protocol (ICAP)", RFC 3507, DOI 10.17487/RFC3507, April
              2003, <http://www.rfc-editor.org/info/rfc3507>.

   [RFC5234]  Crocker, D., Ed. and P. Overell, "Augmented BNF for Syntax
              Specifications: ABNF", STD 68, RFC 5234, DOI 10.17487/
              RFC5234, January 2008,
              <http://www.rfc-editor.org/info/rfc5234>.

   [RFC7230]  Fielding, R., Ed. and J. Reschke, Ed., "Hypertext Transfer
              Protocol (HTTP/1.1): Message Syntax and Routing", RFC
              7230, DOI 10.17487/RFC7230, June 2014,
              <http://www.rfc-editor.org/info/rfc7230>.

   [Errata]   Various Authors, , "RFC 3507 Errata",
              <http://www.measurement-factory.com/std/icap/>.

Author's Address

   Alex Rousskov
   The Measurement Factory

   URI:   http://www.measurement-factory.com/

Rousskov                Expires February 5, 2017               [Page 12]