Internet-Draft Sub-codestream latency J2K over RTP October 2024
Lemieux & Taubman Expires 11 April 2025 [Page]
Workgroup:
Audio/Video Transport Core Maintenance
Internet-Draft:
draft-ietf-avtcore-rtp-j2k-scl-04
Published:
Intended Status:
Standards Track
Expires:
Authors:
P.-A. Lemieux, Ed.
Sandflow Consulting LLC
D. S. Taubman
University of New South Wales

RTP Payload Format for sub-codestream latency JPEG 2000 streaming

Abstract

This RTP payload format defines the streaming of a video signal encoded as a sequence of JPEG 2000 codestreams. The format allows sub-codestream latency, such that the first RTP packet for a given codestream can be emitted before the entire codestream is available.

Status of This Memo

This Internet-Draft is submitted in full conformance with the provisions of BCP 78 and BCP 79.

Internet-Drafts are working documents of the Internet Engineering Task Force (IETF). Note that other groups may also distribute working documents as Internet-Drafts. The list of current Internet-Drafts is at https://datatracker.ietf.org/drafts/current/.

Internet-Drafts are draft documents valid for a maximum of six months and may be updated, replaced, or obsoleted by other documents at any time. It is inappropriate to use Internet-Drafts as reference material or to cite them other than as "work in progress."

This Internet-Draft will expire on 11 April 2025.

1. Introduction

This RTP payload format specifies the streaming of a video signal encoded as a sequence of JPEG 2000 codestreams.

In addition to supporting a variety of frame scanning techniques (progressive, interlaced and progressive segmented frame) and image characteristics, the payload format includes the following features specifically designed for streaming applications:

  • the payload format allows sub-codestream latency such that the first RTP packet of a given codestream to be emitted before the entire codestream is available. Specifically, the payload format does not rely on the JPEG 2000 PLM and PLT marker segments for recovery after RTP Packet loss since these markers can only be written after the codestream is complete and are thus incompatible with sub-codestream latency. Instead, the payload format includes payload header fields (ORDH, ORDB, POS and PID) that indicates whether the RTP packet contains a resynchronization (resync) point and how a recipient can restart codestream processing from that resync point. This contrasts with [RFC5371], which also specifies an RTP payload format for JPEG 2000, but relies on codestream structures that cannot be emitted until the entire codestream is available.
  • as in [RFC4175], the payload header contains an extension (ESEQ) to the standard 16-bit RTP sequence number, enabling the payload format to accommodate high data rates without ambiguity. This is necessary as the standard sequence number will roll over very quickly for high data rates likely to be encountered in this application. For example, the standard sequence number will roll over in 0.5 seconds with a 1-Gbps video stream with RTP Packet sizes of at least 1000 octets, which can be a problem for detecting loss and out-of-order packets particularly in instances where the round-trip time is greater than the roll over period (0.5 seconds in this example).
  • the payload header optionally contains a temporal offset (PTSTAMP) relative to the first RTP Packet with the same value of RTP timestamp field (Section 5.2). The higher resolution of PTSTAMP compared to the timestamp allows receivers to recover the sender's clock more rapidly.

Finally, the payload format also makes use of the unique scalability features of JPEG 2000 to allow a network agent or recipient to discard resolutions and/or quality layers merely by inspecting payload headers (QUAL and RES fields), without having to parse the underlying codestream.

2. Requirements Language

The key words "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT", "SHOULD", "SHOULD NOT", "RECOMMENDED", "NOT RECOMMENDED", "MAY", and "OPTIONAL" in this document are to be interpreted as described in BCP 14 [RFC2119] [RFC8174] when, and only when, they appear in all capitals, as shown here.

3. Media format description

The following summarizes the structure of the JPEG 2000 codestream, which is specified in detail at [jpeg2000-1].

NOTE: as described at Section 6, a JPEG 2000 codestream allows capabilities defined in any part of the JPEG 2000 family of standards, including those specified in [jpeg2000-2] and [jpeg2000-15].

JPEG 2000 represents an image as one or more components, e.g., R, G and B, each uniformly sampled on a common rectangular reference grid. An image can be further divided into contiguous rectangular tiles that are each independently coded and decoded.

JPEG 2000 codes each image as a standalone codestream. Each codestream consists of (i) marker segments, which contain coding parameters and metadata, and (ii) coded data.

The codestream starts with an SOC marker segment and ends with an EOC marker segment. The main header of the codestream consists of marker segments between the SOC and first SOT marker segment and contains information that applies to the codestream in its entirety. It is generally impossible to decode a codestream without its main header.

The rest of the codestream consists of additional marker segments (tile-part headers) interleaved with coded image data.

The coded image data ultimately consists of code-blocks, each containing coded samples belonging to a rectangular (spatial) region within one resolution level of one component. Code-blocks are further collected into precincts, which, accordingly, represents code-blocks belonging to a spatial region within one resolution level of one component.

The image coded data can be arranged into several progression orders, which dictates which aspect of the image appears first in the codestream (in terms of byte offset). The progression orders are parameterized according to:

Position (P)
The first lines of the image come before the last lines of the image.
Component (C)
The first component of the image come before the last component of the image.
Resolution Layer (R)
The information needed to reconstruct the lower spatial resolutions of the image come before the information needed to reconstruct the higher spatial resolutions of the image.
Quality Layer (L)
The information needed to reconstruct the most-significant bits of each sample come before the information needed to reconstruct the least-significant bit of each sample.

For example, in the PRCL progression order, the information needed to reconstruct the first lines of the image come before that needed to reconstruct the last lines of the image and, within a collection of lines, the information needed to reconstruct the lower spatial resolutions of the image come before the information needed to reconstruct the higher spatial resolutions. This progression order is particular useful for sub-frame latency operations.

4. Video signal description

This RTP payload format supports three distinct video frame scanning techniques:

  • Progressive frame
  • Interlaced frame, where each frame consists of two fields. Field 1 occurs temporarily before Field 2. The height in lines of each field is half the height of the image.
  • Progressive segmented frame (PsF), where each frame consists of two segments. Segment 1 contains the odd lines (1, 3, 5, 7,...) of a frame and Segment 2 contains the even lines (2, 4, 6, 8,...) of the same frame, where lines from the top of the frame to the bottom of the frame are numbered sequentially starting at 1.

All frames are scanned left to right, top to bottom.

5. Payload Format

5.1. General 


        <----------- Codestream (image) --------->
        |                                        |
        < Extended Header >                      |
        |                 |                      |
        +-----+-----+-----+------------//--+-----+-----+---------
        | SOC | ... | SOD | .............. | EOC |  P  | SOC  ...
        +-----+-----+-----+------------//--+-----+-----+---------
        |                                              |
        |                                              |
        |                                              |
        +---------------------+------+-//--+-----------+---------
Packets |        Main         | Body | ... |    Body   | Main ...
        +---------------------+------+-//--+-----------+---------

        SOC = Start of codestream marker
        SOD = Start of data marker
        EOC = End of codestream marker
        P = (Optional) padding bytes
Figure 1: Packetization of a sequence of JPEG 2000 codestreams (not to scale).

Each RTP packet, as specified at [RFC3550], is either a Main Packet or a Body Packet.

A Main Packet consists of the following ordered sequence of structures concatenated without gaps:

  • the RTP Fixed Header;
  • a Main Packet Payload Header, as specified at Section 5.3; and
  • the payload, which consists of a JPEG 2000 codestream fragment.

A Body Packet consists of the following ordered sequence of structures concatenated without gaps:

  • the RTP Fixed Header;
  • a Body Packet Payload Header, as specified at Section 5.4; and
  • the payload, which consists of a JPEG 2000 codestream fragment.

When concatenated, the sequence of JPEG 2000 codestream fragments emitted by the sender MUST be a sequence of JPEG 2000 codestreams where two successive JPEG 2000 codestreams MAY be separated by one or more arbitrary padding bytes (see Figure 1).

The JPEG 2000 codestreams MUST conform to Section 6.

The padding bytes MUST be ignored by the recipient.

NOTE: Padding bytes can be used to achieve constant bit rate transmission.

A JPEG 2000 codestream consists of the bytes between, and including, the SOC and EOC markers, as defined in [jpeg2000-1].

A JPEG 2000 codestream fragment does not necessarily contain complete JPEG 2000 packets, as defined in [jpeg2000-1].

A JPEG 2000 codestream Extended Header consists of the bytes between, and including, the SOC marker and the first SOD marker.

The payload of a Body Packet MUST NOT contain any bytes of the JPEG 2000 codestream Extended Header.

The payload of a Main Packet MUST contain at least one byte of the JPEG 2000 codestream Extended Header and MAY contain bytes other than those of the JPEG 2000 codestream Extended Header.

A payload MUST NOT contain bytes from more than one JPEG 2000 codestream.

5.2. RTP Fixed Header Usage

The following RTP header fields have a specific meaning in the context of this payload format:

marker
1
The payload contains an EOC marker.
0
Otherwise
timestamp

The timestamp is the presentation time of the image to which the payload belongs.

The timestamp clock rate is 90 kHz.

The timestamp of successive progressive frames MUST advance at regular increments based on the instantaneous video frame rate.

The timestamp of Field 1 of successive interlaced frames MUST advance at regular increments based on the instantaneous video frame rate, and the Timestamp of Field 2 MUST be offset from the timestamp of Field 1 by one half of the instantaneous frame period.

The timestamp of both segments of a progressive segmented frame MUST be equal.

timestamp of all RTP packets of a given image MUST be equal.

sequence number

The low-order bits of the RTP sequence number.

The higher order bits of the RTP sequence number are contained in the ESEQ field, which is specified at Section 5.3.

The RTP sequence number is calculated as follows:

ESEQ * 65536 + sequence number

5.3. Main Packet Payload Header

Figure 2 specifies the structure of the payload header. Fields are interpreted as unsigned binary integers in network order.


 0                   1                   2                   3
 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
|MH | TP  |ORDH |P|XTRAC|        PTSTAMP        |     ESEQ      |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
|R|S|C| RSVD  |*|    PRIMS      |    TRANS      |      MAT      |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
|                              XTRAB                            |
|                               ...                             |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+

* RANGE
Figure 2: Structure of the Main Packet Payload Header
MH (Codestream Main Header Presence)
0
The RTP Packet is a Body Packet.
1
The RTP Packet is a Main Packet and the codestream has more than one Main Packet. The next RTP Packet is a Main Packet.
2
The RTP Packet is a Main Packet and the codestream has more than one Main Packet. The next RTP Packet is a Body Packet.
3
The RTP Packet is a Main Packet and the codestream has exactly one Main Packet.
TP (Image Type)

Indicates the scanning structure of the image to which the payload belongs.

0
Progressive frame.
1
Field 1 of an interlaced frame, where the first line of the field is the first line of the frame.
2
Field 2 of an interlaced frame, where the first line of the field is the second line of the frame.
3
Field 1 of an interlaced frame, where the first line of the field is the second line of the frame.
4
Field 2 of an interlaced frame, where the first line of the field is the first line of the frame.
5
Segment 1 of a progressive segmented frame, where the first line of the image is the first line of the frame.
6
Segment 2 of a progressive segmented frame, where the first line of the image is the second line of the frame.
7
Extension value. See Section 8.6 and Section 7.8.
ORDH (Progression Order [Main Packet])

Specifies the progression order used by the codestream and whether resync points are signaled.

0
Resync points are not necessarily signaled. The progression order can vary over the codestream.
1
The progression order is LRCP for the entire codestream. The first resync point is specified in every Body Packet that contains one or more resync points.
2
The progression order is RLCP for the entire codestream. The first resync point is specified in every Body Packet that contains one or more resync points.
3
The progression order is RPCL for the entire codestream. The first resync point is specified in every Body Packet that contains one or more resync points.
4
The progression order is PCRL for the entire codestream. The first resync point is specified in every Body Packet that contains one or more resync points.
5
The progression order is CPRL for the entire codestream. The first resync point is specified in every Body Packet that contains one or more resync points.
6
The progression order is PRCL for the entire codestream. The first resync point is specified in every Body Packet that contains one or more resync points.
7
The progression order can vary over the codestream. The first resync point is specified in every Body Packet that contains one or more resync points.

ORDH MUST be 0 is the codestream consists of more than one tile.

NOTE: Only ORDH = 4 and ORDH = 6 allow sub-codestream latency streaming.

NOTE: Progression order PRCL is defined in [jpeg2000-2]. The other progression orders are specified in [jpeg2000-1].

P (Precision Timestamp Presence)
0
PTSTAMP is not used.
1
PTSTAMP is used.
XTRAC (Extension Payload Length)
Length, in multiples of 4 bytes, of the XTRAB field.
PTSTAMP (Precision Timestamp)

PTSTAMP = (timestamp + TOFF) mod 4096, if P = 1 in the Main Packet of this codestream.

TOFF is the transmission time of this RTP Packet, in the timebase of the timestamp clock and relative to the first packet with the same timestamp value.

TOFF = 0 in the first RTP Packet with the same timestamp value.

PTSTAMP = 0, if P = 0 in the Main Packet of this codestream.

NOTE: As described at Section 7.4 and Section 8.1, PTSTAMP is intended to improve clock recovery at the receiver and only applies when the transmission time of two consecutive RTP packets with identical timestamp fields differ by no more than 45 ms = 4095/90,000. [RFC5450] provides addresses the general case when a RTP packet is transmitted at a time other than its nominal transmission time.

ESEQ (Extended Sequence Number)

The high order bits of the RTP sequence number.

Section 5.2 specifies the low-order bits of the RTP sequence number and the formula to compute the RTP sequence number

R (Codestream Main Header Reuse)

Determines whether Main Packet and codestream header information can be reused across codestreams.

1

All Main Packets in this stream, as identified by its SSRC value:

  • MUST have identical Main Packet Payload Headers, with the exception of their TP, MH, ESEQ and PTSTAMP fields;
  • MUST contain the same codestream main header information, with the exception of the SOT and COM marker segments, and any pointer marker segments; and
  • MUST NOT contain bytes other than Extended Header bytes.
0
Otherwise
S (Parameterized Colorspace Presence)
0

Component colorimetry is not specified, and left to the session or the application.

PRIMS, TRANS and MAT and RANGE MUST be zero.

1

Component colorimetry is specified by the PRIMS, TRANS and MAT and RANGE fields.

The codestream components MUST conform to one of the combinations at Table 1.

Table 1: Mapping of codestream components to color channels
Combination name Component index
0 1 2 3
Y Y
YA Y A
RGB R G B
RGBA R G B A
YCbCr Y CB CR
YCbCrA Y CB CR A
The channel A is an opacity channel. The minimum sample value (0) indicates a completely transparent sample, and the maximum sample value (as determined by the bit depth of the codestream component) indicates a completely opaque sample. The opacity channel MUST map to a component with unsigned samples.
C (Code-block Caching Usage)
0
Code-block caching is not in use.
1

Code-block caching is in use.

R MUST be equal to 1.

RSVD (Reserved)
Reserved value. See Section 8.5 and Section 7.7.
RANGE (Video Full Range Usage)
Value of the VideoFullRangeFlag specified in [rec-itu-t-h273]
PRIMS (Color Primaries)
One of the ColourPrimaries values specified in [rec-itu-t-h273]
TRANS (Transfer Characteristics)
One of the TransferCharacteristics values specified in [rec-itu-t-h273]
MAT (Color Matrix Coefficients)
One of the MatrixCoefficients values specified in [rec-itu-t-h273]
XTRAB (Extension Payload)
Allows the contents of the Main Packet Payload Header to be extended in the future. See Section 8.4 and Section 7.6.

5.4. Body Packet Payload Header

Figure 3 specifies the structure of the Body Packet Payload Header. Fields are interpreted as unsigned binary integers in network order.


 0                   1                   2                   3
 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
|MH | TP  |RES  |*|QUAL |       PTSTAMP         |     ESEQ      |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
|         POS           |                  PID                  |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+

* ORDB
Figure 3: Structure of the Body Packet Payload Header
MH
See Section 5.3.
TP
See Section 5.3.
RES (Resolution Layers)
0
The payload can contribute to all resolution layers.
Otherwise
The payload contains at least one byte of one JPEG 2000 packet belonging to resolution level (NL + RES - 7) but does not contain any byte of any JPEG 2000 packet belonging to lower resolution levels. NL is the number of decomposition levels of the codestream.
ORDB (Progression Order [Body Packet]
0
No resync point is specified for the payload.
1
The payload contains a resync point.

ORDB MUST be 0 is the codestream consists of more than one tile.

QUAL (Quality Layers)
0
The payload can contribute to all quality layers.
Otherwise
The payload contributes only to quality layer index QUAL or above.
PTSTAMP
See Section 5.3.
ESEQ
See Section 5.3.
POS (Resync Point Offset)

Byte offset from the start of the payload to the first byte of the resync point belonging to the precinct identified by PID.

POS MUST be 0 if ORDB = 0.

PID (Precinct Identifier)

Unique identifier of the precinct of the resync point.

PID = c + s * num_components

where:

  • c is the index (starting from 0) of the image component to which the precinct belongs;
  • s is a sequence number which identifies the precinct within its tile-component; and
  • num_components is the number of components of the codestream.

If PID is present, the payload MUST NOT contain codestream bytes from more than one precinct.

PID MUST be 0 if ORDB = 0.

NOTE: PID is identical to precinct identifier I specified in [jpeg2000-9].

6. JPEG 2000 codestream requirements

6.1. General

The JPEG 2000 codestream MAY include capabilities beyond those specified at [jpeg2000-1], including those specified in [jpeg2000-2] and [jpeg2000-15].

NOTE: The Rsiz parameter and CAP marker segments of each JPEG 2000 codestream contain detailed information on the capabilities necessary to decode the codestream.

NOTE: The caps media type parameter defined in Section 9.2 allows applications to signal required device capabilities.

NOTE: The block coder specified at [jpeg2000-15] improves throughput and reduces latency compared to the original arithmetic block coder defined in [jpeg2000-1].

For interlaced or progressive segmented frames, the height specified in the JPEG 2000 main header MUST be the height in lines of the field or the segment, respectively.

If any decomposition level involves only horizontal decomposition then no decomposition level MUST involve only vertical decomposition; and conversely, if any decomposition level involves only vertical decomposition then no decomposition level MUST involve only horizontal decomposition.

7. Sender requirements

7.1. Main Packet

Only Main Packets MAY contain bytes of the JPEG 2000 codestream Extended Header.

The sender MUST either emit a single Main Packet with MH = 3, or one or more Main Packets with MH = 1 followed by a single Main Packet with MH = 2.

The Main Packet Payload Headers fields MUST be identical in all Main Packet of a given codestream, with the exception of:

  • MH;
  • ESEQ; and
  • PTSTAMP.

7.2. RTP Packet filtering

A network agent MAY strip out RTP Packet from a codestream that are of no interest to a particular client, e.g., based on a resolution or a spatial region of interest. Such a network agent SHOULD include a CSRC identifier to identify the SSRC field of the original source from which content was stripped.

7.3. Resync point

A resync point is the first byte of JPEG 2000 packet header data for a precinct and for which PID < 224.

NOTE: Resync points cannot be specified if the codestream consists of more than one tile (ORDB and ORDH are both equal to zero).

NOTE: A resync point can be used by a receiver to process a codestream even if earlier packets in the codestream have been corrupted, lost or deliberately discarded by a network agent. As a corollary, resync points can be used by a network agent to discard packets that are not relevant to a given rendering resolution or region of interest. Resync points play a role similar to pointer marker segments, albeit tailored for high bandwidth low latency streaming applications.

7.4. PTSTAMP field

A sender SHOULD set P = 1, but only if it can generate PTSTAMP accurately.

PTSTAMP can be derived from the same clock that is used to produce the 32-bit timestamp field in the RTP fixed header. Specifically, a sender maintains, at least conceptually, a 32-bit counter that is incremented by a 90kHz clock. The counter is sampled at the point when each RTP Packet is or SHOULD be at least notionally transmitted and the 12 LSBs of the sample are stored in the PTSTAMP field.

If P = 1, then the transmission time TOFF (as defined at Section 5.3) for two consecutive RTP packets with identical timestamp fields MUST NOT differ by more than 4095.

7.5. RES field

A sender SHOULD set RES > 0 whenever possible.

NOTE: While a sender can always safely set RES = 0, this makes it more difficult to discard packets based on resolution, as described at Section 8.3.

7.6. Extra information

The sender MUST set the value of XTRAC to 0.

Future edition of this specification can permit other values.

7.7. Reserved values

The sender MUST set reserved values to 0.

Future edition of this specification can specify other values such that these values can be ignored by receivers that conform to this specification.

7.8. Extension values

A sender MUST NOT use an extension value.

7.9. Code-block caching

This section applies only if C = 1.

A sender can improve bandwidth efficiency by only occasionally transmitting code-blocks corresponding to static portions of the video and otherwise transmitting empty code-blocks. When C = 1, and as described at Section 8.7, a receiver maintains a simple cache of previously received code-blocks, which it uses to replace empty code-blocks.

A sender alone determines which and when code-blocks are replaced with empty code-blocks.

The sender cannot however determine with certainty the state of the receiver's cache: some code-blocks might have been lost in transit, the sender doesn't know exactly when the receiver started processing the stream, etc.

A code-block is empty if:

  • it does not contribute code-bytes as specified in the parent JPEG 2000 packet header; or
  • if the code-block conforms to [jpeg2000-15], contains an HT cleanup segment and the first two bytes of the Magsgn byte-stream are between 0xFF80 and 0xFF8F.

NOTE: the last condition allows the encoder to insert padding bytes to achieve a constant bit rate even when a code-block does not contribute code-bytes, as suggested at [jpeg2000-15], F.4.

8. Receiver

8.1. PTSTAMP

Receivers can use PTSTAMP values to accelerate sender clock recovery since PTSTAMP typically updates more regularly than timestamp.

8.2. QUAL

A receiver can discard packets where QUAL > N if it is interested in reconstructing an image that only incorporates quality layers N and below.

8.3. RES

The JPEG 2000 coding process decomposes an image using a sequence of discrete wavelet transforms (DWT) stages.

Table 2: Optional discarding of Body Packets based on the value of the RES field when decoding a reduced resolution image, in the case where NL = 5 and all DWT stages consist of both horizontal and vertical transforms. The image has nominal width and height of W x H.
Decomposition level Resolution level Sub-bands Keep all Body Packets with RES equal to or less than this value... ... to decode an image with at most these dimensions
1 5 HL1,LH1,HH1 7 W x H
2 4 HL2,LH2,HH2 6 (W/2) x (H/2)
3 3 HL3,LH3,HH3 5 (W/4) x (H/4)
4 2 HL4,LH4,HH4 4 (W/8) x (H/8)
5 1 HL5,LH5,HH5 3 (W/16) x (H/16)
5 0 LL5 2 (W/32) x (H/32)

Table 2 illustrates the case where each DWT stage consists of both horizontal and vertical transforms, which is the only mode supported in [jpeg2000-1]. The first stage transforms the image into (i) the image at half-resolution (LL1 sub-bands) and (ii) residual high-frequency data (HH1, LH1, HL1 sub-bands). The second stage transforms the image at half-resolution (LL1 sub-bands) into the image at quarter resolution (LL2 sub-bands) and residual high-frequency data (HH2, LH2, HL2 sub-bands). This process is repeated NL times, where NL is the number of decomposition levels as defined in the COD and COC marker segments of the codestream.

The decoding process reconstructs the image by reversing the coding process, starting with the lowest resolution image stored in the codestream (LLNL).

As a result, it is possible to reconstruct a lower resolution of the image by stopping the decoding process at a selected stage. For example, in order to reconstruct the image at quarter resolution (LL2), only sub-bands with index greater than 2, e.g., HL3, LH3, HH3, HL4, LH4, HH4, etc., are necessary. In other words, a receiver that wishes to reconstruct an image at quarter resolution could discard all packets where RES >= 6 since those packets can only contribute to HL1, LH1, HH1, HL2, LH2 and HH2 sub-bands.

In the case where all DWT stages consist of both horizontal and vertical transforms, the maximum decodable resolution is reduced by a factor of 27 - N if all Body Packets where RES > N are discarded.

Table 3: Optional discarding of Body Packets based on the value of the RES field when decoding a reduced resolution image, in the case where NL = 5 and some DWT stages consist of only horizontal transforms. The image has nominal width and height of W x H.
Decomposition level Resolution level Sub-bands Keep all Body Packets with RES equal to or less than this value... ... to decode an image with at most these dimensions
1 5 HL1,LH1,HH1 7 W x H
2 4 HL2,LH2,HH2 6 (W/2) x (H/2)
3 3 HX3 5 (W/4) x (H/2)
4 2 HX4 4 (W/8) x (H/2)
5 1 HX5 3 (W/16) x (H/2)
5 0 LX5 2 (W/32) x (H/2)

Table 3 illustrates the case where some of DWT stage consist of only horizontal transforms, as specified at Annex F of [jpeg2000-2].

A receiver can therefore discard all Body Packets where RES is greater than some threshold value if it is interested in decoding an image with its resolution reduced by a factor determined by the threshold value, as illustrated in Table 2 and Table 3.

8.4. Extra information

The receiver MUST accept values XTRAC other than 0 and MUST ignore the value of XTRAB, whose length is given by XTRAC.

Future edition of this specification can specify XTRAB contents such that this content can be ignored by receivers that conform to this specification.

8.5. Reserved values

The receiver MUST ignore the value of reserved values.

8.6. Extension values

The receiver MUST discard an RTP packet that contains any extension value.

8.7. Code-block caching

This section applies only if C = 1.

When C = 1, and as specified in Section 7.9, the sender can improve bandwidth efficiency by only occasionally transmitting code-blocks corresponding to static portions of the video and otherwise transmitting empty code-blocks, as defined at Section 7.9.

When decoding a codestream, and for each code-block in the codestream:

  • if the code-block in the codestream is empty, the receiver MUST replace it with a matching code-block from the cache, if one exists; or
  • if the code-block in the codestream is not empty, the receiver MUST replace any matching code-block from the cache with the code-block in the codestream.

Two code-blocks are matching if the following characteristics are identical for both: spatial coordinates, resolution level, component, sub-band and value of the TP field of the parent RTP packet.

9. Media Type

9.1. General

This RTP payload format is identified using the media type defined at Section 9.2, which is registered in accordance with [RFC4855] and using the template of [RFC6838].

9.2. Definition

Type name
video
Subtype name
jpeg2000-scl
Required parameters
None
Optional parameters
pixel

Specifies the pixel format used by the video sequence.

The parameter MUST be a URI-reference as specified in [RFC3986].

If the parameter is a relative-ref as specified in [RFC3986], then it MUST be equal to one of the pixel formats specified in Table 4 and the RTP header and payload MUST conform with the characteristics of that pixel format.

If the parameter is not a relative-ref, the specification of the pixel format is left to the application that defined the URI.

If the parameter is not specified, the pixel format is unspecified.

sample

Specifies the format of the samples in each component of the codestream.

The parameter MUST be a URI-reference as specified in [RFC3986].

If the parameter is a relative-ref as specified in [RFC3986], then it MUST be equal to one of the formats specified in Appendix C and the stream MUST conform with the characteristics of that format.

If the parameter is not a relative-ref, the specification of the sample format is left to the application that defined the URI.

If the parameter is not specified, the sample format is unspecified.

width

Maximum width in pixels of each image. Integer between 0 and 4,294,967,295.

The parameter MUST be a sequence of 1 or more digits.

If the parameter is not specified, the maximum width is unspecified.

height

Maximum height in pixels of each image. Integer between 0 and 4,294,967,295.

The parameter MUST be a sequence of 1 or more digits.

If the parameter is not specified, the maximum height is unspecified.

signal

Specifies the sequence of image types.

The parameter MUST be a URI-reference as specified in [RFC3986].

If the parameter is a relative-ref as specified in [RFC3986], then it MUST be equal to one of the signal formats specified in Appendix B and the image sequence MUST conform to that signal format.

If the parameter is not a relative-ref, the specification of the pixel format is left to the application that defined the URI.

If the parameter is not specified, the stream consists of an arbitrary sequence of image types.

caps

The parameters contains a list of sets of constraints to which the stream conforms, with each set of constraints identified using an absolute-URI defined by an application.

The parameter MUST conform to the uri-list syntax expressed using ABNF ([RFC5234]):

  uri-list = absolute-URI *(";" absolute-URI)

Each absolute-URI MUST NOT contain any ";" character.

The application that defines the absolute-URI MUST associate it with a set of constraints to which the stream conforms. Such constraints can, for example, include the maximum height and width of images.

If the parameter is not specified, constraints, beyond those specified in this document, are unspecified.

cache

The value of the parameter MUST be either false or true.

If the parameter is true, the field C MAY be 0 or 1; otherwise the field C MUST be 0.

If the parameter is not specified, then the parameter is equal to false.

Encoding considerations
This media type is framed and binary, see Section 4.8 of [RFC6838].
Security considerations
See Section 12.
Interoperability considerations
The RTP stream is a sequence of JPEG 2000 images. An implementation that conforms to the family of JPEG 2000 standards can decode and attempt to display each image.
Published specification
This document
Applications that use this media type
video streaming and communication
Person and email address to contact for further information
Pierre-Anthony Lemieux <pal@sandflow.com>
Intended usage
COMMON
Restrictions on Usage
This media type depends on RTP framing, and hence is only defined for use with RTP as specified at [RFC3550]. Transport within other framing protocols is not defined at the time.
Author
Pierre-Anthony Lemieux
Change controller
IETF Audio/Video Transport Core Maintenance Working Group delegated from the IESG.

10. Mapping to the Session Description Protocol (SDP)

The mapping of the payload format media type and its parameters to SDP, as specified in [RFC8866] MUST be done according to Section 3 of [RFC4855].

11. IANA Considerations

This memo requests that IANA registers the content type specified at Section 9.

12. Security considerations

RTP packets using the payload format specified in this document are subject to the security considerations discussed in [RFC3550] , and in any applicable RTP profile such as [RFC3551], [RFC4585], [RFC3711], [RFC5124]. However, as [RFC7202] discusses, it is not an RTP payload format's responsibility to discuss or mandate what solutions are used to meet the basic security goals like confidentiality, integrity, and source authenticity for RTP in general. This responsibility lays on anyone using RTP in an application. They can find guidance on available security mechanisms and important considerations in [RFC7201]. Applications SHOULD use one or more appropriate strong security mechanisms. The rest of this Security Considerations section discusses the security impacting properties of the payload format itself.

This RTP payload format and its media decoder do not exhibit any significant non-uniformity in the receiver-side computational complexity for RTP Packet processing, and thus are unlikely to pose a denial-of-service threat due to the receipt of pathological data. Nor does the RTP payload format contain any active content.

Security considerations related to the JPEG 2000 codestream contained in the payload are discussed at Section 3 of [RFC3745].

13. References

13.1. Normative References

[jpeg2000-1]
ITU-T, "Recommendation ITU-T T.800, JPEG 2000 image coding system: Core coding system", .
[jpeg2000-2]
ITU-T, "Recommendation ITU-T T.801, JPEG 2000 image coding system: Extensions", .
[jpeg2000-15]
ITU-T, "Recommendation ITU-T T.814, JPEG 2000 image coding system: High-throughput JPEG 2000", .
[rec-itu-t-h273]
ITU-T, "Recommendation ITU-T H.273, Coding-independent code points for video signal type identification", .
[jpeg2000-9]
ITU-T, "JPEG 2000 image coding system: Interactivity tools, APIs and protocols", .
[RFC3550]
Schulzrinne, H., Casner, S., Frederick, R., and V. Jacobson, "RTP: A Transport Protocol for Real-Time Applications", STD 64, RFC 3550, DOI 10.17487/RFC3550, , <https://www.rfc-editor.org/info/rfc3550>.
[RFC8866]
Begen, A., Kyzivat, P., Perkins, C., and M. Handley, "SDP: Session Description Protocol", RFC 8866, DOI 10.17487/RFC8866, , <https://www.rfc-editor.org/info/rfc8866>.
[RFC4855]
Casner, S., "Media Type Registration of RTP Payload Formats", RFC 4855, DOI 10.17487/RFC4855, , <https://www.rfc-editor.org/info/rfc4855>.
[RFC3986]
Berners-Lee, T., Fielding, R., and L. Masinter, "Uniform Resource Identifier (URI): Generic Syntax", STD 66, RFC 3986, DOI 10.17487/RFC3986, , <https://www.rfc-editor.org/info/rfc3986>.
[RFC5234]
Crocker, D., Ed. and P. Overell, "Augmented BNF for Syntax Specifications: ABNF", STD 68, RFC 5234, DOI 10.17487/RFC5234, , <https://www.rfc-editor.org/info/rfc5234>.
[RFC2119]
Bradner, S., "Key words for use in RFCs to Indicate Requirement Levels", BCP 14, RFC 2119, DOI 10.17487/RFC2119, , <https://www.rfc-editor.org/info/rfc2119>.
[RFC8174]
Leiba, B., "Ambiguity of Uppercase vs Lowercase in RFC 2119 Key Words", BCP 14, RFC 8174, DOI 10.17487/RFC8174, , <https://www.rfc-editor.org/info/rfc8174>.

13.2. Informative References

[RFC5371]
Futemma, S., Itakura, E., and A. Leung, "RTP Payload Format for JPEG 2000 Video Streams", RFC 5371, DOI 10.17487/RFC5371, , <https://www.rfc-editor.org/info/rfc5371>.
[RFC4175]
Gharai, L. and C. Perkins, "RTP Payload Format for Uncompressed Video", RFC 4175, DOI 10.17487/RFC4175, , <https://www.rfc-editor.org/info/rfc4175>.
[RFC6838]
Freed, N., Klensin, J., and T. Hansen, "Media Type Specifications and Registration Procedures", BCP 13, RFC 6838, DOI 10.17487/RFC6838, , <https://www.rfc-editor.org/info/rfc6838>.
[RFC3551]
Schulzrinne, H. and S. Casner, "RTP Profile for Audio and Video Conferences with Minimal Control", STD 65, RFC 3551, DOI 10.17487/RFC3551, , <https://www.rfc-editor.org/info/rfc3551>.
[RFC4585]
Ott, J., Wenger, S., Sato, N., Burmeister, C., and J. Rey, "Extended RTP Profile for Real-time Transport Control Protocol (RTCP)-Based Feedback (RTP/AVPF)", RFC 4585, DOI 10.17487/RFC4585, , <https://www.rfc-editor.org/info/rfc4585>.
[RFC3711]
Baugher, M., McGrew, D., Naslund, M., Carrara, E., and K. Norrman, "The Secure Real-time Transport Protocol (SRTP)", RFC 3711, DOI 10.17487/RFC3711, , <https://www.rfc-editor.org/info/rfc3711>.
[RFC5124]
Ott, J. and E. Carrara, "Extended Secure RTP Profile for Real-time Transport Control Protocol (RTCP)-Based Feedback (RTP/SAVPF)", RFC 5124, DOI 10.17487/RFC5124, , <https://www.rfc-editor.org/info/rfc5124>.
[RFC7201]
Westerlund, M. and C. Perkins, "Options for Securing RTP Sessions", RFC 7201, DOI 10.17487/RFC7201, , <https://www.rfc-editor.org/info/rfc7201>.
[RFC7202]
Perkins, C. and M. Westerlund, "Securing the RTP Framework: Why RTP Does Not Mandate a Single Media Security Solution", RFC 7202, DOI 10.17487/RFC7202, , <https://www.rfc-editor.org/info/rfc7202>.
[RFC3745]
Singer, D., Clark, R., and D. Lee, "MIME Type Registrations for JPEG 2000 (ISO/IEC 15444)", RFC 3745, DOI 10.17487/RFC3745, , <https://www.rfc-editor.org/info/rfc3745>.
[RFC5450]
Singer, D. and H. Desineni, "Transmission Time Offsets in RTP Streams", RFC 5450, DOI 10.17487/RFC5450, , <https://www.rfc-editor.org/info/rfc5450>.

Appendix A. Pixel formats

Table 4 defines pixel formats.

Table 4: Defined pixel formats
NAME SAMP COMPS TRANS PRIMS MAT VFR Mapping in Table 1
rgb444sdr 4:4:4 RGB 1 1 0 0, 1 RGB
rgb444wcg 4:4:4 RGB 1 9 0 0, 1 RGB
rgb444pq 4:4:4 RGB 16 9 0 0, 1 RGB
rgb444hlg 4:4:4 RGB 18 9 0 0, 1 RGB
ycbcr420sdr 4:2:0 YCbCr 1 1 1 0 YCbCr
ycbcr422sdr 4:2:2 YCbCr 1 1 1 0 YCbCr
ycbcr422wcg 4:2:2 YCbCr 1 9 9 0 YCbCr
ycbcr422pq 4:2:2 YCbCr 16 9 9 0 YCbCr
ycbcr422hlg 4:2:2 YCbCr 18 9 9 0 YCbCr

Each pixel format is characterized by the following:

NAME
Identifies the pixel format
COMPS
RGB
Each codestream contains exactly three components, associated with the R, G and B color channels, in order.
YCbCr
Each codestream contains exactly three components, associated with the Y, Cb and Cr color channels, in order.
SAMP
4:2:0
The Cb and Cr color channels are subsampled horizontally and vertically by 1/2.
4:2:2
The Cb and Cr color channels are subsampled horizontally by 1/2.
4:4:4
No color channels are sub-sampled.
TRANS

Identifies the transfer characteristics allowed by the pixel format, as defined at [rec-itu-t-h273]

PRIMS

Identifies the color primaries allowed by the pixel format, as defined at [rec-itu-t-h273]

MAT

Identifies the matrix coefficients allowed by the pixel format, as defined at [rec-itu-t-h273]

VFR

Allows values of the VideoFullRangeFlag defined at [rec-itu-t-h273]

Appendix B. Signal formats

prog
The stream MUST only consist of a sequence of progressive frames.
psf
Progressive segmented frame (PsF) stream. The stream MUST only consist of an alternating sequence of first segment and second segment.
tff
Interlaced stream. The stream MUST only consist of an alternating sequence of first field and second field, where the first line of the first field is the first line of the frame.
bff
Interlaced stream. The stream MUST only consist of an alternating sequence of first field and second field, where the first line of the first field is the second line of the frame.

Appendix C. Sample formats

8
All components consist of unsigned 8-bit integer samples.
10
All components consist of unsigned 10-bit integer samples.
12
All components consist of unsigned 12-bit integer samples.
16
All components consist of unsigned 16-bit integer samples.

Appendix D. Summary of Changes (Informative)

D.1. Introduction

This Appendix summarizes substantive changes across revisions of this specification. This summary is informative and not intended to be exhaustive.

D.2. Changes from draft-ietf-avtcore-rtp-j2k-scl-00

  • Allow multi-tile images in a single stream, in addition to allowing multi-tile images to be transmitted as multiple single-tile streams.
  • Fix incorrect TRANS values.

D.3. Changes from draft-ietf-avtcore-rtp-j2k-scl-01

  • Removed signaling for the transmission of multi-tile images as multiple single-tile image streams (the tile media type parameter).

D.4. Changes from draft-ietf-avtcore-rtp-j2k-scl-02

  • Removed request for registration in the deprecated IANA registry for RTP Payload Format MIME types.

Authors' Addresses

Pierre-Anthony Lemieux (editor)
Sandflow Consulting LLC
San Mateo, CA
United States of America
David Scott Taubman
University of New South Wales
Sydney
Australia