Internet Engineering Task Force G. Bertrand, Ed.
Internet-Draft E. Stephan
Intended status: Informational France Telecom - Orange
Expires: August 16, 2012 February 13, 2012
CDNI Logging Interface
draft-bertrand-cdni-logging-00
Abstract
This memo specifies the Logging interface between a downstream CDN
(dCDN) and an upstream CDN (uCDN). It introduces a framework, an
architecture design and a set of new requirements. Then it drafts an
information model.
Status of this Memo
This Internet-Draft is submitted in full conformance with the
provisions of BCP 78 and BCP 79.
Internet-Drafts are working documents of the Internet Engineering
Task Force (IETF). Note that other groups may also distribute
working documents as Internet-Drafts. The list of current Internet-
Drafts is at http://datatracker.ietf.org/drafts/current/.
Internet-Drafts are draft documents valid for a maximum of six months
and may be updated, replaced, or obsoleted by other documents at any
time. It is inappropriate to use Internet-Drafts as reference
material or to cite them other than as "work in progress."
This Internet-Draft will expire on August 16, 2012.
Copyright Notice
Copyright (c) 2012 IETF Trust and the persons identified as the
document authors. All rights reserved.
This document is subject to BCP 78 and the IETF Trust's Legal
Provisions Relating to IETF Documents
(http://trustee.ietf.org/license-info) in effect on the date of
publication of this document. Please review these documents
carefully, as they describe your rights and restrictions with respect
to this document. Code Components extracted from this document must
include Simplified BSD License text as described in Section 4.e of
the Trust Legal Provisions and are provided without warranty as
described in the Simplified BSD License.
Bertrand & Stephan Expires August 16, 2012 [Page 1]
Internet-Draft CDNI Logging February 2012
This document may contain material from IETF Documents or IETF
Contributions published or made publicly available before November
10, 2008. The person(s) controlling the copyright in some of this
material may not have granted the IETF Trust the right to allow
modifications of such material outside the IETF Standards Process.
Without obtaining an adequate license from the person(s) controlling
the copyright in such materials, this document may not be modified
outside the IETF Standards Process, and derivative works of it may
not be created outside the IETF Standards Process, except to format
it for publication as an RFC or to translate it into languages other
than English.
Table of Contents
1. Introduction . . . . . . . . . . . . . . . . . . . . . . . . . 4
1.1. Terminology . . . . . . . . . . . . . . . . . . . . . . . 4
1.2. Abbreviations . . . . . . . . . . . . . . . . . . . . . . 5
2. Logging Framework . . . . . . . . . . . . . . . . . . . . . . 6
3. Architecture . . . . . . . . . . . . . . . . . . . . . . . . . 6
4. Additional Requirements . . . . . . . . . . . . . . . . . . . 9
5. Rationale for Logging Interface . . . . . . . . . . . . . . . 9
5.1. Usages of CDNI Logging Information By uCDN . . . . . . . . 9
5.1.1. Maintenance/Debugging . . . . . . . . . . . . . . . . 9
5.1.2. Accounting . . . . . . . . . . . . . . . . . . . . . . 10
5.1.3. End-User Experience Management . . . . . . . . . . . . 10
5.1.4. Security . . . . . . . . . . . . . . . . . . . . . . . 10
5.2. Logging Information Views . . . . . . . . . . . . . . . . 11
5.3. Information Extracted From Logging Data . . . . . . . . . 11
6. Log Information Elements . . . . . . . . . . . . . . . . . . . 12
6.1. Core Information Elements . . . . . . . . . . . . . . . . 13
6.2. Information Elements for Content Delivery . . . . . . . . 14
6.3. Information Elements for Content Acquisition . . . . . . . 15
6.4. Log Extensibility . . . . . . . . . . . . . . . . . . . . 15
7. Core Logging Records . . . . . . . . . . . . . . . . . . . . . 15
7.1. Content Delivery . . . . . . . . . . . . . . . . . . . . . 15
7.2. Content Acquisition . . . . . . . . . . . . . . . . . . . 16
7.3. Content Purging . . . . . . . . . . . . . . . . . . . . . 16
7.4. Extended CoDRs . . . . . . . . . . . . . . . . . . . . . . 17
8. Logging Process . . . . . . . . . . . . . . . . . . . . . . . 17
8.1. Logging Aggregation . . . . . . . . . . . . . . . . . . . 17
8.1.1. Logging and Fragmented Objects . . . . . . . . . . . . 18
8.2. Logging Protection . . . . . . . . . . . . . . . . . . . . 18
8.2.1. Logging Signing . . . . . . . . . . . . . . . . . . . 18
8.3. Logging Filtering . . . . . . . . . . . . . . . . . . . . 18
8.4. Logging Update and Rectification . . . . . . . . . . . . . 19
9. Protocols for Logging . . . . . . . . . . . . . . . . . . . . 19
10. IANA Considerations . . . . . . . . . . . . . . . . . . . . . 20
Bertrand & Stephan Expires August 16, 2012 [Page 2]
Internet-Draft CDNI Logging February 2012
11. Security Considerations . . . . . . . . . . . . . . . . . . . 20
11.1. Privacy . . . . . . . . . . . . . . . . . . . . . . . . . 20
11.2. Non Repudiation . . . . . . . . . . . . . . . . . . . . . 20
12. Acknowledgments . . . . . . . . . . . . . . . . . . . . . . . 20
13. References . . . . . . . . . . . . . . . . . . . . . . . . . . 21
13.1. Normative References . . . . . . . . . . . . . . . . . . . 21
13.2. Informative References . . . . . . . . . . . . . . . . . . 21
Appendix A. Examples Log Format . . . . . . . . . . . . . . . . . 22
A.1. W3C Common Log File (CLF) Format . . . . . . . . . . . . . 22
A.2. W3C Extended Log File (ELF) Format . . . . . . . . . . . . 23
A.3. National Center for Supercomputing Applications (NCSA)
Common Log Format . . . . . . . . . . . . . . . . . . . . 24
A.4. NCSA Combined Log Format . . . . . . . . . . . . . . . . . 24
A.5. NCSA Separate Log Format . . . . . . . . . . . . . . . . . 24
A.6. Squid 2.0 Native Log Format for Access Logs . . . . . . . 25
Authors' Addresses . . . . . . . . . . . . . . . . . . . . . . . . 25
Bertrand & Stephan Expires August 16, 2012 [Page 3]
Internet-Draft CDNI Logging February 2012
1. Introduction
This memo specifies the Logging interface between a downstream CDN
(dCDN) and an upstream CDN (uCDN). It introduces a framework, an
architecture design and a set of new requirements. Then it drafts an
information model.
The reader should be familiar with the work of the CDNI WG:
o CDNI problem statement [I-D.ietf-cdni-problem-statement] and
framework [I-D.davie-cdni-framework] identify a Logging interface,
o Section 7 of [I-D.ietf-cdni-requirements] specifies a set of
requirements for Logging,
o [I-D.ietf-cdni-use-cases] outlines real world use-cases for
interconnecting CDNs. These use cases require the exchange of
Logging information between the dCDN and the uCDN.
The present document describes:
o The Logging framework (Section 2),
o The architecture (Section 3),
o The requirements (Section 4),
o Discussion on the monitoring and the reporting (Section 5)
o Log information (Section 6 and Section 7),
1.1. Terminology
We adopt the terminology described in
[I-D.ietf-cdni-problem-statement] and [I-D.davie-cdni-framework], and
extend it with the additional terms defined below.
For clarity, we use the word "Log" only for referring to internal CDN
logs and we use the word "Logging" for any inter-CDN information
exchange and processing operations related to CDNI Logging interface.
Log: CDN internal information collection and processing operations.
Logging: Inter-CDN information exchange and processing operations.
Small object: [Ed. Note: TBD]
Fragmented object: [Ed. Note: Tentative of a simple definition which
Bertrand & Stephan Expires August 16, 2012 [Page 4]
Internet-Draft CDNI Logging February 2012
fits with the current CDNi charter] Fragmented objects are pieces of
content provided by a CSP which are delivered individually through a
CDN interconnection. They differ from a simple object because the
delivery of the content to one user agent may be provided by more
than one Surrogate/CDN.
CDN Reporting: the process of providing the relevant information that
will be used to create a formatted content delivery report provided
to the CSP in differed time. Such information typically includes
aggregated data that can cover a large period of time (e.g., from
hours to several months). One of the usages of reporting is the
collection of charging data related to CDN services and the
computation of Key Performance Indicators (KPIs).
CDN Monitoring: the process of providing content delivery information
in real-time. The monitoring typically includes data in real time to
provide a vision of the deliveries in progress, for service operation
purposes. It presents a view of the global health of the services as
well as information on usage and performance, for network services
supervision and operation management. In particular, monitoring data
can be used to generate alarms.
Core log information: minimal information that has to be logged to
satisfy the Logging requirements
End-user experience management: study of Logging data using
statistical analysis to discover, understand, and predict user
behavior patterns.
Usage data: the usage data refers to all the information related to a
specific end-user session.
Delivery service: [Ed. Note: to be defined]
1.2. Abbreviations
[Ed. Note: List of abbreviations to be updated later]
o API: Application Programming Interface
o CDN: Content Delivery Network
o CDNP: Content Delivery Network Provider
o CoDR: Content Delivery Record
o CSP: Content Service Provider
Bertrand & Stephan Expires August 16, 2012 [Page 5]
Internet-Draft CDNI Logging February 2012
o DASH: Dynamic Adaptive Streaming over HTTP
o dCDN: downstream CDN
o FTP: File Transfer Protocol
o FTPS: FTP Secure
o HAS: HTTP Adaptive Streaming
o KPI: Key Performance Indicator
o PVR: Personal Video Recorder
o SNMP: Simple Network Management Protocol
o uCDN: upstream CDN
2. Logging Framework
The framework of the Logging interface is straightforward: dCDN logs
any information related to the completion of any task performed by a
dCDN on behalf of an uCDN and any exchange related to the management
of the contents that the said dCDN delivers on behalf of an uCDN, as
discussed in Section 7.1.
3. Architecture
Logging is a mandatory feature for a CDN, especially if the CDN is
interconnected to other CDNs. Logging provides the raw material for
some essential operations of a delivery service, such as monitoring,
reporting, billing, etc.
As stated in [I-D.ietf-cdni-problem-statement], "the CDNI Logging
interface enables details of logs or events to be exchanged between
interconnected CDNs".
Figure 1 provides an example of Logging information exchanges. uCDN
is connected to dCDN-1 and dCDN-2. Both dCDN-1, dCDN-2, and uCDN
deliver content for CSP. The Logging interface enables the uCDN to
obtain Logging data from dCDN-1 and dCDN-2. In the example, uCDN
uses the Logging data:
o to audit the performance of the delivery operated by the dCDNs and
to adjust its routing request as appropriate,
Bertrand & Stephan Expires August 16, 2012 [Page 6]
Internet-Draft CDNI Logging February 2012
o to provide reporting (non real-time) and monitoring (real-time)
information to CSP.
For instance, uCDN merges Logging data, extracts relevant KPIs, and
presents a formatted report to CSP, in addition to a bill for the
content delivered. uCDN may also provide Logging data as raw logs to
CSP, so that CSP uses its own Logging analysis tools.
+-----+
| CSP |
+-----+
^
|
| Reporting and monitoring data
| Billing
|
,--,--.
,-' `-.
CoDR ( uCDN ) CoDR
....>( )<....
| ( ) |
| ( RRi ) |
| `-. Tuning ,-' |
| -|-|-' |
| | | |
| | | |
,--|--. | | ,--|--.
,-' `-. | | ,-' `-.
( dCDN-1 <----+ + ---> dCDN-2 )
`-. ,-' `-. ,-'
`--'--' `--'--'
Figure 1: Exchange of Logging Information
Figure 2 presents the Logging Architecture. More details on the
Logging operations are provided in Section 8. A dCDN prepares the
CoDRs requested by the uCDN. This preparation involves operations
such as filtering, aggregating, anonymizing, and summarizing the
logs. The uCDN downloads the corresponding CoDRs and performs its
own reporting for the CSP.
Bertrand & Stephan Expires August 16, 2012 [Page 7]
Internet-Draft CDNI Logging February 2012
--------
/ \
| CSP |
\ /
--^-----
^
^ Reporting, Monitoring, Billing
^
---^--------------------- -------------------------
/ ^ Upstream CDN \ / Downstream CDN \
|+-----+ +-------------+ | | +-------------+ +-----+|
|| |**| Control | | | | Control |**| ||
|| | +-------------+ | | +-------------+ | I ||
|| I | | CoDR selection | | n ||
|| n | +-------------+ |----------------->| +-------------+ | t ||
|| t |<<| Logging | | | | Logging |<<| e ||
|| e | +-------------+ |<-----------------| +-------------+ | r ||
|| r | | CoDRs | | c L ||
|| c L | | | | o o ||
|| o o | +-------------+ | | +-------------+ | n g ||
|| n g |<<|Req-Routing | | | |Req-Routing |>>| n i ||
|| n i | +-------------+ | | +-------------+ | e c ||
|| e c | | | | c ||
|| c | +-------------+ | | +-------------+ | t ||
|| t |<<| Metadata | | | | Metadata |>>| i ||
|| i | +-------------+ | | +-------------+ | o ||
|| o | | | | n ||
|| n | +-------------+ | | +-------------+ | ||
|| |<<| Distribution| |******************| | Distribution|>>| ||
|+-----+ +-------------+ | Acquisition | +-------------+ +-----+|
\ / \ . * /
------------------------- ---------.-*-------------
. . *
. Request . * Delivery
. . *
. +--.-*--+
..................Request............| User |
| Agent |
+-------+
Figure 2: Logging Architecture
Logging Information elements may be captured at various stages during
the lifecycle of content distribution. The arrows (">>") of the
above Figure 2 represent the direction of information elements in the
Logging process. They illustrate several important aspects:
Bertrand & Stephan Expires August 16, 2012 [Page 8]
Internet-Draft CDNI Logging February 2012
o An Information element may be captured either by an uCDN or a
dCDN, or both;
o An Information element can be collected on another interface than
the Logging (e.g., uCDN's Request-Routing);
o Information elements can be collected before the exchange of
CoDRs.
These points are further discussed in Section 9.
4. Additional Requirements
Section 7 of [I-D.ietf-cdni-requirements], already specifies a set of
requirements for Logging (LOG-1 to LOG-16). Some security
requirements also affect Logging (e.g., SEC-4).
[Ed. Note: uCDN shall be able to select the type of events that a
dCDN should include in the Logging that the latter provides to the
uCDN.]
5. Rationale for Logging Interface
[I-D.davie-cdni-framework] and [I-D.ietf-cdni-problem-statement]
already introduce the rationale for the Logging interface as a means
for an uCDN to acquire some visibility on the contents the dCDN
delivers on behalf of the uCDN. dCDN provides the uCDN with elements
of information and CoDRs for operating the CDN interconnection and
reporting to the CSP. This section develops use cases that require
exchange of Logging information.
5.1. Usages of CDNI Logging Information By uCDN
This section presents the usage of the CoDRs by an uCDN. It does not
make any assumption on where the CoDRs are produced. CoDRs may be
produced either by the uCDN or a dCDN.
5.1.1. Maintenance/Debugging
Logging is useful to permit the detection (and limit the risk) of
content delivery failures. In particular, Logging facilitates the
resolution of false configuration issues.
To detect faults, Logging must enable the reporting of any CDN
operation success and failure, such as request redirection, content
acquisition, etc. Such information can be summarized into KPIs. For
Bertrand & Stephan Expires August 16, 2012 [Page 9]
Internet-Draft CDNI Logging February 2012
instance, Logging format should allow the computation of the number
of times during a given epoch, a content delivery related to a
specific service succeeds/fails.
This need is taken into account in the events triggering log entries,
which are listed in Section 7.
Logging is useful to analyze the performance of content delivery
services. This implies computing KPIs from the Logging data for
service quality analysis and monitoring (see Section 5.3).
Logging enables the CDN providers to evaluate the QoS level related
to a specific delivery service. For instance, one aspect of this QoS
level could be measured through the average delivery throughput
experienced by end-users in a given region for this specific service
over a period of time.
Logging enables the CDN providers to identify and troubleshoot
performance degradations. In particular, Logging enables the
communication of traffic data (e.g., the amount of traffic that has
been forwarded by a dCDN on behalf of an uCDN over a given period of
time), which is particularly useful for CDN and network planning
operations.
5.1.2. Accounting
Logging is essential for accounting, to permit inter-CDN billing, and
CSP billing by uCDN. For instance, Logging enables the uCDN to check
the total amount of traffic delivered by every dCDN and for every
delivery service, as well as the associated bandwidth usage (e.g.,
peak, 95th percentile), and the maximum number of simultaneous
sessions over a given period of time.
5.1.3. End-User Experience Management
The goal of end-user experience management is to gather any relevant
information to meter audience, analyze user behavior, etc. For
instance, Logging enables the CDN providers to report on content
consumption (e.g., delivered sessions per content) in a specific
geographic area.
5.1.4. Security
The goal of security is to prevent and monitor unauthorized access,
misuse, modification, and denial of access of a service. A set of
information is logged for security purposes. In particular, access
to content is usually collected to permit the CSP to detect
infringements of content delivery policies and other abnormal end-
Bertrand & Stephan Expires August 16, 2012 [Page 10]
Internet-Draft CDNI Logging February 2012
user behaviors.
5.2. Logging Information Views
Logging information is useful to the uCDN and potentially to the CSP.
Different views of the Logging information may be provided depending
on privacy, business, and scalability constraints. Some kind of
information format adaptation capability MAY be supported by an uCDN
to present some (e.g., filtered, aggregated) data in the appropriate
format (raw logs, reports) to the CSP. More details on these
operations are provided in Section 8.
We provide a non-exhaustive list and description of tools that can be
fed with Logging information.
o Tools used by the uCDN's operator: billing tools (information
system), customer experience intelligence, reporting tools,
security auditing tools, dimensioning tools, strategic planning
and investment...
o Tools used by CSPs: customer experience management tools,
reporting tools, security auditing tools...
5.3. Information Extracted From Logging Data
This section presents, for explanatory purposes, a non-exhaustive
list of information that can be extracted/produced from logs.
Depending on the inter-CDN agreement, this information may be
computed by the uCDN or by the dCDN.
CSPs require specific information, such as KPIs, about the delivery
of their content. The Logging data must contain appropriate
information to enable CSPs or the uCDN to extract the required KPIs.
In the present section, we list important examples of KPIs:
o Number of delivery requests received from end-users in a given
region for each piece of content, during a given period of time
(e.g., hour/day/week/month),
o Percentage of delivery successes / failures among the
aforementioned requests
o Number of failures listed by failure type (e.g., HTTP error code)
for requests received from end-users in a given region and for
each piece of content, during a given period of time (e.g., hour/
day/week/month),
Bertrand & Stephan Expires August 16, 2012 [Page 11]
Internet-Draft CDNI Logging February 2012
o Number and cause of delivery premature termination for end-users
in a given region and for each piece of content, during a given
period of time (e.g., hour/day/week/month),
o Maximum and mean number of simultaneous sessions established by
end-users in a given region, for a given delivery service, and
during a given period of time (e.g., hour/day/week/month),
o Volume of traffic delivered for sessions established by end-users
in a given region, for a given delivery service, and during a
given period of time (e.g., hour/day/week/month),
o Maximum, mean, and minimum delivery throughput for sessions
established by end-users in a given region, for a given delivery
service, and during a given period of time (e.g., hour/day/week/
month)
o Cache-hit and byte-hit ratios for requests received from end-users
in a given region for each piece of content, during a given period
of time (e.g., hour/day/week/month)
o Top 10 of the most popular requested content (with time
repartition into day/week/month),
o Terminal type (mobile, PC, STB, if this information can be
acquired from the browser type header, for example).
Additional KPIs can be computed from other sources of information
than the Logging, for instance, data collected by a content portal or
by specific client-side APIs.
6. Log Information Elements
CDNI must specify a set of Logging information elements to avoid log
format regeneration, which would affect the performance of the log
handling chain. A common set of Logging information element eases
the sharing of logs among the CDNs and the use of log processing
tools, for instance, to prepare reporting.
Existing CDNs Logging functions collect and consolidate logs
performed by their Surrogates. Surrogates usually store the logs
using a format derived from Web servers log standards such as W3C and
NCSA [ELF] [CLF]. In practice, these formats are adapted to cope
with CDN specifics. Appendix A presents the W3C and NCSA log
formats.
Bertrand & Stephan Expires August 16, 2012 [Page 12]
Internet-Draft CDNI Logging February 2012
6.1. Core Information Elements
This section describes a set of information elements that structure
log information generated by the dCDN. The section does not
prescribe a particular encoding (such as SNMP SMI or alternatives).
All fields in the log information are optional unless stated
otherwise.
+--------+----------------------------------------------------------+
| Name | Description |
+--------+----------------------------------------------------------+
| Time | A date and time associated with a logged event. For |
| | instance, the time that the server finished processing |
| | the request. |
| URI_lo | The requested URL path (e.g., |
| g | /cdn.csp.com/movies/potter.avi?param=11&user=toto if the |
| | full request URL was |
| | "http://node1.peer-a.op-b.net/cdn.csp.com/movies/potter. |
| | avi?param=11&user=toto"). The URI without hostname |
| | typically includes the "CDN domain" (ex.cdn.csp.com) - |
| | cf. [I-D.davie-cdni-framework]: it enables the |
| | identification of the CSP service agreed between the CS |
| | Pand the CDNP operating the uCDN. |
| Protoc | The protocol and protocol version of the message that |
| ol | triggered the log entry. |
| Reques | The protocol method of the request message that |
| t | triggered the log entry. |
| metho | |
| d | |
| Status | The protocol method of the reply message related to the |
| | log entry |
| Body | The number of bytes in the body of the reply message |
| size | related to the log entry. It does not include the size |
| | of the response headers. |
| Bytes | The number of bytes (headers + body) of the message that |
| receiv | triggered the log entry. |
| ed | |
| Header | Multiple header fields, such as User Agent or Referrer, |
| s | could be reproduced in the log entries. |
| Durati | The duration of an operation in milliseconds. For |
| on | instance, this field could be used to provide the time |
| | it took to the Surrogate to send the requested file to |
| | the end-user, or the time it took the Surrogate to |
| | acquire the file on a cache-miss event. |
| Operat | The kind of operation that is logged; for instance, |
| ion | Acquisition, Delivery, or Purging. |
+--------+----------------------------------------------------------+
Bertrand & Stephan Expires August 16, 2012 [Page 13]
Internet-Draft CDNI Logging February 2012
Table 1: Core Information Elements
Subsequent table illustrates the definition of the core information
elements. It provides examples using Apache log format strings
[apache] when they exist. The table is here for illustration and
does not prescribe a specific encoding.
+----------+-------------------+------------------------------------+
| Name | String | Example |
+----------+-------------------+------------------------------------+
| Time | %t | [10/Oct/2000:13:55:36 -0700] |
| URI_log | - | - |
| Protocol | %H | HTTP/1.0 |
| Request | %m | GET |
| method | | |
| Status | %>s | 200 |
| Body | %b | 2326 |
| size | | |
| Bytes | - | - |
| received | | |
| Header | \"%{Referer}i\" | "http://www.example.com/start.html |
| | \"%{User-agent}i\ | ""Mozilla/4.08 [en] (Win98; I |
| | " | ;Nav)" |
| Duration | - | - |
| Operatio | - | - |
| n | | |
+----------+-------------------+------------------------------------+
Table 2: Examples using Apache format
6.2. Information Elements for Content Delivery
+-------------+-----------------------------------------------------+
| Name | Definition |
+-------------+-----------------------------------------------------+
| uCDN | An element authenticating the operator of the uCDN |
| identifier | as the authority having delegated the request to |
| | the dCDN |
| End-user's | The IP address of the client making a content |
| IP address | delivery request (or of its proxy) |
| Cache bytes | The number of body bytes served from caches. This |
| | quantity permits the computation of the byte hit |
| | ratio. |
+-------------+-----------------------------------------------------+
Table 3: Delivery Information Elements
Bertrand & Stephan Expires August 16, 2012 [Page 14]
Internet-Draft CDNI Logging February 2012
6.3. Information Elements for Content Acquisition
+------------+------------------------------------------------------+
| Name | Definition |
+------------+------------------------------------------------------+
| dCDN | An element authenticating the operator of the dCDN |
| identifier | as the authority requesting the content to the uCDN |
+------------+------------------------------------------------------+
Table 4: Acquisition Information Elements
6.4. Log Extensibility
Future usages might introduce the need for additional Logging data.
In addition, some use-cases such as an Inter-Affiliate
Interconnection [I-D.ietf-cdni-use-cases], might take advantage of
extended Logging exchanges. Therefore, it is important to permit
CDNs to use additional Logging fields than the standard ones, if they
want.
7. Core Logging Records
This section defines a set of central events that a dCDN should
register and publish through the Logging interface. There are two
types of events. The fist category belongs to legacy Web servers'
access and errors logs. The second is directly tied to the auditing
of the CDN interconnection.
We classify the logged events depending on the CDN operation to which
they relate: content delivery, content acquisition, content purging,
etc.
Next versions of the memo will associate a CoDR to each event.
7.1. Content Delivery
Some CSPs pay a lot of attention to the protection of their content
(e.g., premium video CSPs). To fulfill the needs of these CSPs, a
CDN shall log all the details of the content delivery authorizations.
This means that a dCDN must be able to provide log detailing the
content delivery/content acquisition authorizations and denials as
well as information on why the request is authorized/denied.
The events triggering the generation of a log record include:
o Reception of a content request,
Bertrand & Stephan Expires August 16, 2012 [Page 15]
Internet-Draft CDNI Logging February 2012
The generated log record typically embeds information about:
o Denial of delivery (error or unauthorized request) for a request,
o Beginning of delivery (authorization) of a requested content,
o End of an authorized delivery (success),
o End of an authorized delivery (failure),
7.2. Content Acquisition
In case the uCDN require the dCDN to log acquisition related events,
the events triggering the generation of a log record include:
o Emission of a content acquisition request (first try or retry) for
a cache hit or a cache miss with content revalidation
The generated log record typically embeds information about:
o Reception of a reply indicating denial of delivery (error or
unauthorized request) for a content acquisition request,
o End of an authorized acquisition (success),
o End of an authorized acquisition (failure)
Note that a dCDN may acquire content only from the uCDN. It this
case, the uCDN can log the dCDN's content acquisition operations
itself, and thus, the uCDN typically does not require the dCDN to log
acquisition related events.
7.3. Content Purging
The purging of a piece of content is typically requested by the uCDN,
which can, therefore, log events related to purging. In case the
uCDN nevertheless requests a dCDN to log purging events, the events
triggering the generation of a log record include:
o Reception of a content purging request
The generated log record typically embeds information about:
o Denial of the purging request (error or unauthorized request),
o Beginning of purging (authorization) for a given content purging
request,
Bertrand & Stephan Expires August 16, 2012 [Page 16]
Internet-Draft CDNI Logging February 2012
o End of an authorized purging (success),
o End of an authorized purging (failure),
7.4. Extended CoDRs
The required Logging information may depend on the considered
services. For instance, static file delivery (e.g., pictures)
typically does not include any delivery restrictions. By contrast,
video delivery typically implies strong content delivery
restrictions, as explained in [I-D.ietf-cdni-use-cases], and Logging
could include information about the enforcement of these
restrictions. Therefore, to ease the support of different services
as well as future services, the Logging interface should support
optional log information.
8. Logging Process
We walk through a "day in the life" of a CDN interconnection to
present functions the two CDNs may require to exchange Logging
information. This will serve to illustrate many of the functions
that could be supported through CDNI Logging interface. We describe
capabilities, such as log aggregation, anonymization, and filtering,
that might be added to CDNI in a later stage, to optimize Logging
operations.
8.1. Logging Aggregation
CDNs typically handle millions of records per day. The processing of
these records to extract relevant monitoring and reporting
information is expensive in terms of CPU and time. Therefore, as
stated in [I-D.davie-cdni-framework], "a design tradeoff in the
Logging interface is the degree of aggregation or summarization of
data."
In particular, dCDNs aggregate the logs of their elements (e.g., the
Surrogates) to avoid both the complexity of distributing multiple log
files to the uCDN and to avoid disclosing information about dCDN's
internal topology. This aggregation alleviates the Logging
processing burden for the uCDN.
[Ed. Note: In a later version, the draft will propose methods to
optimize the amount of information transmitted: (e.g., transmit only
KPIs, use multiple levels of logs granularity such as in Apache
(debug, notice, etc.)]
Bertrand & Stephan Expires August 16, 2012 [Page 17]
Internet-Draft CDNI Logging February 2012
8.1.1. Logging and Fragmented Objects
Many situations lead to the delivery of fragments of content (DASH,
failure of delivery, partial delivery, PVR actions, etc.). A dCDN
may not publish a CoDR for each piece of content it delivers, because
this can lead to unacceptably large logs. In particular, a CoDR
could provide aggregated information about the delivery of several
content pieces. uCDN and dCDN must be able to agree on a level of
granularity for the CoDRs. This problem is well described for the
case of HTTP adaptive streaming in [I-D.davie-cdni-framework]:
"Most schemes to deliver HTTP-based adaptive bit- rate video use a
large number of relatively small HTTP requests (e.g., one request per
3-second chunk of video.) It may be desirable to aggregate Logging
information so that a single log entry is provided for the entire
video rather than for each chunk. Note however that such aggregation
requires a degree of application awareness in dCDN to recognize that
the many HTTP requests correspond to a single video."
8.2. Logging Protection
8.2.1. Logging Signing
CDNs need guarantees on logs Integrity. They want to know:
o who issued the Logging, and
o if the Logging has been modified by a third party.
This is extremely important, as the logs can provide a basis for
accounting/billing.
[Ed. note: propose a mechanism to authenticate the Logging origin]
[Ed. note: cf. requirements draft: "SEC-4 [MED] The CDNI solution
should be able to ensure that the Downstream CDN cannot spoof a
transaction log attempting to appear as if it corresponds to a
request redirected by a given Upstream CDN when that request has not
been redirected by this Upstream CDN. This ensures non-repudiation
by the Upstream CDN of transaction logs generated by the Downstream
CDN for deliveries performed by the Downstream CDN on behalf of the
Upstream CDN."]
8.3. Logging Filtering
The dCDN must be able to present only relevant information to the
uCDN, to avoid unnecessary log processing load for the uCDN. Hence,
the downstream CDN filters its logs, and passes the relevant records
Bertrand & Stephan Expires August 16, 2012 [Page 18]
Internet-Draft CDNI Logging February 2012
directly to each upstream CDN. This requires that the downstream CDN
can recognize the set of log entries that relate to each upstream
CDN, for instance thanks to the "uCDN identifier" information element
Table 3.
The dCDN must be able to filter some internal scope data such as
information related to its internal alarms (security, failures, load,
etc).
In some use cases described in [I-D.ietf-cdni-use-cases], the
interconnected CDNs do not want to disclose details on their internal
topology. The dCDN must be able to filter confidential data on the
dCDN's topology (number of servers, location, etc.). In particular,
information about the requests served by every Surrogate is
confidential. Therefore, the Logging information must be protected
so that data such as Surrogates hostnames is not disclosed to the
uCDN. In the "Inter-Affiliates Interconnection" use case, this
information may be disclosed to the uCDN because both the dCDN and
the uCDN are operated by entities of the same group.
8.4. Logging Update and Rectification
If Logging is generated periodically, it is important that the
sessions that start in one Logging period and end in another are
correctly reported. If they are reported in the starting period,
then the Logging of this period will be available only after the end
of the session, which delays the Logging generation.
A Logging rectification / update mechanism could be useful to reach a
good tradeoff between the Logging generation delay and the Logging
accuracy. Such mechanism would be particularly invaluable for real
time Logging, which must be provided rapidly and cannot wait for the
end of operations in progress.
9. Protocols for Logging
This section discusses the encoding and the protocols for
transporting Logging information.
CDNs usually store the logs in a format similar to the ones in use by
web servers, such as W3C, NCSA, and Squid's log format, which are
described in Appendix A.
As presented in [I-D.ietf-cdni-problem-statement], several protocols
already exist that could potentially be used to exchange CDNI Logging
between interconnected CDNs. The dCDN could publish non real-time
Logging on a server where the uCDN would retrieve it using FTP, FTPS,
Bertrand & Stephan Expires August 16, 2012 [Page 19]
Internet-Draft CDNI Logging February 2012
or Syslog. If the CDNs need to exchange real-time information
through the Logging interface, they could potentially rely on Web
APIs, syslog, SNMP... However, as explained in
[I-D.ietf-cdni-problem-statement], "SNMP traps pose scalability
concerns and SNMP does not support guaranteed delivery of Traps and
therefore could result in log records being lost and the consequent
CoDRs and billing records for that content delivery not being
produced as well as that content delivery being invisible to any
analytics platforms."
[Ed. Note: in a later version, this memo will include an analysis of
candidate protocols, based upon a set of (basic) requirements, such
as reliable transport mode, preservation of the integrity of the
information conveyed by the protocol, etc.]
10. IANA Considerations
This memo includes no request to IANA.
11. Security Considerations
11.1. Privacy
CDNs have the opportunity to collect detailed information about the
downloads performed by end users. The provision of this information
to another CDN introduces end-users privacy protection concerns.
11.2. Non Repudiation
Logging provides the raw material for charging. It permits the dCDN
to bill the uCDN for the content deliveries that the dCDN makes on
behalf of the uCDN. It also permits the uCDN to bill the CSP for the
content delivery service. Therefore, non-repudiation of Logging data
is essential.
12. Acknowledgments
The authors would like to thank Anne Marrec, Yannick Le Louedec, and
Christian Jacquenet for detailed feedback on early versions of this
document and for their input on existing Log formats.
The authors would like also to thank Fabio Costa, Yvan Massot, Renaud
Edel, and Joel Favier for their input and comments.
Finally, they thank the contributors of the EU FP7 OCEAN project for
Bertrand & Stephan Expires August 16, 2012 [Page 20]
Internet-Draft CDNI Logging February 2012
valuable inputs.
13. References
13.1. Normative References
[RFC2119] Bradner, S., "Key words for use in RFCs to Indicate
Requirement Levels", BCP 14, RFC 2119, March 1997.
13.2. Informative References
[CLF] A. Luotonen, "The Common Logfile Format, W3C (work in
progress)", 1995, <http://www.w3.org/pub/WWW/Daemon/User/
Config/Logging.html>.
[ELF] Phillip M. Hallam-Baker and Brian Behlendorf, "Extended
Log File Format, W3C (work in progress), WD-logfile-
960323", <http://www.w3.org/TR/WD-logfile.html>.
[I-D.bertrand-cdni-experiments]
Bertrand, G., Faucheur, F., and L. Peterson, "Content
Distribution Network Interconnection (CDNI) Experiments",
draft-bertrand-cdni-experiments-01 (work in progress),
August 2011.
[I-D.davie-cdni-framework]
Davie, B. and L. Peterson, "Framework for CDN
Interconnection", draft-davie-cdni-framework-01 (work in
progress), October 2011.
[I-D.ietf-cdni-problem-statement]
Niven-Jenkins, B., Faucheur, F., and N. Bitar, "Content
Distribution Network Interconnection (CDNI) Problem
Statement", draft-ietf-cdni-problem-statement-03 (work in
progress), January 2012.
[I-D.ietf-cdni-requirements]
Leung, K. and Y. Lee, "Content Distribution Network
Interconnection (CDNI) Requirements",
draft-ietf-cdni-requirements-02 (work in progress),
December 2011.
[I-D.ietf-cdni-use-cases]
Gilles, B., Emile, S., Watson, G., Burbridge, T., Eardley,
P., and K. Ma, "Use Cases for Content Delivery Network
Interconnection", draft-ietf-cdni-use-cases-03 (work in
progress), January 2012.
Bertrand & Stephan Expires August 16, 2012 [Page 21]
Internet-Draft CDNI Logging February 2012
[RFC3444] Pras, A. and J. Schoenwaelder, "On the Difference between
Information Models and Data Models", RFC 3444,
January 2003.
[RFC3466] Day, M., Cain, B., Tomlinson, G., and P. Rzewski, "A Model
for Content Internetworking (CDI)", RFC 3466,
February 2003.
[RFC3568] Barbir, A., Cain, B., Nair, R., and O. Spatscheck, "Known
Content Network (CN) Request-Routing Mechanisms",
RFC 3568, July 2003.
[apache] "Apache 2.2 log files documentation", Feb. 2012,
<http://httpd.apache.org/docs/current/logs.html>.
[squid] "Squid LogFormat documentation", Feb. 2012, <http://
wiki.squid-cache.org/Features/
LogFormat?highlight=%28\\bCategoryFeature\\b%29|%
28faqlisted.yes%29>.
Appendix A. Examples Log Format
This section provides example of log formats implemented in existing
CDNs, web servers, and caching proxies.
Web servers (e.g., Apache) maintain at least one log file for Logging
accesses to content (the Access Log). They can typically be
configured to log errors in a separate log file (the Error Log). The
log formats can be specified in the server's configuration files.
However, webmasters often use standard log formats to ease the log
processing with available log analysis tools.
A.1. W3C Common Log File (CLF) Format
The Common Log File (CLF) format defined by the World Wide Web
Consortium (W3C) working group is compatible with many log analysis
tools and is supported by the main web servers (e.g., Apache) Access
Logs.
According to [CLF], the common logfile format is as follows:
remotehost rfc931 authuser [date] "request" status bytes.
Example (from [apache]: 127.0.0.1 - frank [10/Oct/2000:13:55:36
-0700] "GET /apache_pb.gif HTTP/1.0" 200 2326
The fields are defined as follows [CLF]:
Bertrand & Stephan Expires August 16, 2012 [Page 22]
Internet-Draft CDNI Logging February 2012
+------------+------------------------------------------------------+
| Element | Definition |
+------------+------------------------------------------------------+
| remotehost | Remote hostname (or IP number if DNS hostname is not |
| | available, or if DNSLookup is Off. |
| rfc931 | The remote logname of the user. |
| authuser | The username that the user employed to authenticate |
| | himself. |
| [date] | Date and time of the request. |
| "request" | An exact copy of the request line that came from the |
| | client. |
| status | The status code of the HTTP reply returned to the |
| | client. |
| bytes | The content-length of the document transferred. |
+------------+------------------------------------------------------+
Table 5: Information elements in CLF format
A.2. W3C Extended Log File (ELF) Format
The Extended Log File (ELF) format defined by W3C extends the CLF
with new fields. This format is supported by Microsoft IIS 4.0 and
5.0.
The supported fields are listed below [ELF].
+------------+---------------------------------------------------+
| Element | Definition |
+------------+---------------------------------------------------+
| date | Date at which transaction completed |
| time | Time at which transaction completed |
| time-taken | Time taken for transaction to complete in seconds |
| bytes | bytes transferred |
| cached | Records whether a cache hit occurred |
| ip | IP address and port |
| dns | DNS name |
| status | Status code |
| comment | Comment returned with status code |
| method | Method |
| uri | URI |
| uri-stem | Stem portion alone of URI (omitting query) |
| uri-query | Query portion alone of URI |
+------------+---------------------------------------------------+
Table 6: Information elements in ELF format
Some fields start with a prefix (e.g., "c-", "s-"), which explains
which host (client/server/proxy) the field refers to.
Bertrand & Stephan Expires August 16, 2012 [Page 23]
Internet-Draft CDNI Logging February 2012
Example: date time s-ip cs-method cs-uri-stem cs-uri-query s-port cs-
username c-ip cs(User-Agent) sc-status sc-substatus sc-win32-status
time-taken
2011-11-23 15:22:01 x.x.x.x GET /file 80 y.y.y.y Mozilla/
5.0+(Windows;+U;+Windows+NT+6.1;+en-US;+rv:1.9.1.6)+Gecko/
20091201+Firefox/3.5.6+GTB6 200 0 0 2137
A.3. National Center for Supercomputing Applications (NCSA) Common Log
Format
This format for Access Logs offers the following fields:
o host rfc931 date:time "request" statuscode bytes
o x.x.x.x userfoo [10/Jan/2010:21:15:05 +0500] "GET /index.html
HTTP/1.0" 200 1043
A.4. NCSA Combined Log Format
The NCSA Combined log format is an extension of the NCSA Common log
format with three (optional) additional fields: the referral field,
the user_agent field, and the cookie field.
o host rfc931 username date:time request statuscode bytes referrer
user_agent cookie
o Example: x.x.x.x - userfoo [21/Jan/2012:12:13:56 +0500] "GET
/index.html HTTP/1.0" 200 1043 "http://www.example.com/" "Mozilla/
4.05 [en] (WinNT; I)" "USERID=CustomerA;IMPID=01234"
A.5. NCSA Separate Log Format
The NCSA Separate log format refers to a log format in which the
information gathered is separated into three separate files. This
way, every entry in the Access Log (in the NCSA Common log format) is
complemented with an entry in a Referral log and another one in an
Agent log. The format of the Referral log is as follows:
o date:time referrer
o Example: [21/Jan/2012:12:13:56 +0500]
"http://www.example.com/index.html"
The format of the Referral log is as follows:
o date:time agent
Bertrand & Stephan Expires August 16, 2012 [Page 24]
Internet-Draft CDNI Logging February 2012
o [21/Jan/2012:12:13:56 +0500] "Microsoft Internet Explorer - 5.0"
A.6. Squid 2.0 Native Log Format for Access Logs
Squid [squid] is a popular piece of open-source software for
transforming a Linux host into a caching proxy. Variations of Squid
log format are supported by some CDNs.
Squid log format is as follow: time elapsed remotehost code/status
bytes method URL rfc931 peerstatus/peerhost type
According to Squid 2.0 documentation [squid], these fields are
defined as follows:
+-----------+-------------------------------------------------------+
| Element | Definition |
+-----------+-------------------------------------------------------+
| time | Unix timestamp as UTC seconds with a millisecond |
| | resolution. |
| duration | The elapsed time in milliseconds the transaction |
| | busied the cache. |
| client | The client IP address. |
| address | |
| bytes | The size is the amount of data delivered to the |
| | client, including headers. |
| request | The request method to obtain an object. |
| method | |
| URL | The requested URL. |
| rfc931 | may contain the ident lookups for the requesting |
| | client (turned off by default) |
| hierarchy | The hierarchy information provides information on how |
| code | the request was handled (forwarding it to another |
| | cache, or requesting the content to the Origin |
| | Server). |
| type | The content type of the object as seen in the HTTP |
| | reply header. |
+-----------+-------------------------------------------------------+
Table 7: Information elements in Squid format
Bertrand & Stephan Expires August 16, 2012 [Page 25]
Internet-Draft CDNI Logging February 2012
Authors' Addresses
Gilles Bertrand (editor)
France Telecom - Orange
38-40 rue du General Leclerc
Issy les Moulineaux, 92130
FR
Phone: +33 1 45 29 89 46
Email: gilles.bertrand@orange.com
Stephan Emile
France Telecom - Orange
2 avenue Pierre Marzin
Lannion F-22307
France
Email: emile.stephan@orange.com
Bertrand & Stephan Expires August 16, 2012 [Page 26]