Internet Draft S. Wenger Document: draft-wenger-avt-rtp-jvt-01.txt M. Hannuksela Expires: December 2002 T. Stockhammer June 2002 Expires December 2002 RTP payload Format for JVT Video Status of this Memo This document is an Internet-Draft and is in full conformance with all provisions of Section 10 of RFC2026. Internet-Drafts are working documents of the Internet Engineering Task Force (IETF), its areas, and its working groups. Note that other groups may also distribute working documents as Internet-Drafts. Internet-Drafts are draft documents valid for a maximum of six months and may be updated, replaced, or obsoleted by other documents at any time. It is inappropriate to use Internet-Drafts as reference material or to cite them other than as "work in progress." The list of current Internet-Drafts can be accessed at http://www.ietf.org/1id-abstracts.txt The list of Internet-Draft Shadow Directories can be accessed at http://www.ietf.org/shadow.html Abstract This memo describes an RTP Payload format for the JVT codec. This codec is designed as a joint project of the ITU-T SG 16 VCEG, and the ISO/IEC JTC1/SC29/WG11 MPEG groups. The most up-to-date draft of the video codec was specified in early May 2002, is due for revision in late July 2002, and is available for public review [2]. Wenger et. al. Expires December 2002 [Page 1]
Internet Draft 10 June 2002 1. The JVT codec This memo specifies an RTP payload specification for a new video codec that is currently under development by the Joint Video Group (JVT), which is formed of video coding experts of MPEG and the ITU- T. After the likely approval by the two parent bodies, the codec specification will have the status of an ITU-T Recommendation (likely H.264) and become part of the MPEG-4 specification (ISO/IEC 14496 Part 10). The current project timeline of the JVT project is such that a technically frozen specification (pending bug fixes) is expected in July 2002 in the form of an ISO/IEC Final Committee Draft (FCD). Before JVT was formed in late 2001, this project used the ITU-T project name H.26L and the JVT project inherited all the technical concepts of the H.26L project. The JVT video codec has a very broad application range that covers the whole range from low bit rate Internet Streaming applications to HDTV broadcast and Digital Cinema applications with near loss-less coding. Most, if not all, relevant companies in all of these fields (including TV broadcast) have participated in the standardization, which gives hope that this wide application range is more than an illusion and may materialize, probably in a relatively short time frame. The overall performance of the JVT codec is as such that bit rate savings of 50% or more, compared to the current state of technology, are reported. Digital Satellite TV quality, for example, was reported to be achievable at 1.5 Mbit/s, compared to the current operation point of MPEG 2 video at around 3.5 Mbit/s [1]. The codec specification [2] itself distinguishes between a video coding layer (VCL), and a network abstraction layer (NAL). The VCL contains the signal processing functionality of the codec, things such as transform, quantization, motion search/compensation, and the loop filter. It follows the general concept of most of today's video codecs, a macroblock based coder that utilized inter picture prediction with motion compensation, and transform coding of the residual signal. The output of the VCL are slices: a bit string that contains the macroblock data of an integer number of macroblocks, and the information of the slice header (containing the spatial address of the first macroblock in the slice, the initial quantization parameter, and similar). Macroblocks in slices are ordered in scan order unless a different macroblock allocation is specified, using the so-called Flexible Macroblock Ordering syntax. In-picture prediction is used only within a slice. The NAL encapsulates the slice output of the VCL into Network Abstraction Layer Units (NALUs), which are suitable for the transmission over packet networks or the use in packet oriented multiplex environments. JVT's Annex B defines an encapsulation process to transmit such NALUs over byte-stream oriented networks. In the scope of this memo Annex B is not relevant. Neither VCL nor NAL are claimed to be media or network independent - the VCL needs to know transmission characteristics in order to Wenger et. al. Expires December 2002 [Page 2]
Internet Draft 10 June 2002 appropriately select the error resilience strength, slice size, etc., whereas the NAL needs information like the importance of a bit string provided by the VCL to select the appropriate application layer protection. Internally, the NAL uses NAL Units or NALUs. A NALU consists of a one-byte header and the payload byte string. The header co-serves as the RTP payload header and indicates the type of the NALU, the (potential) presence of bit errors in the NALU payload, and information whether this NALU is required for maintaining the synchronicity of the encoder/decoder loops. This RTP payload specification is designed to be unaware of the bit string in the NALU payload. One of the main properties of the JVT codec is the possibility of the use of Reference Picture Selection. For each macroblock the reference picture to be used can be selected independently. The reference pictures may be used in a first-in, first-out fashion, but it is also possible to handle the reference picture buffers explicitly. A consequence of this new feature (it was available before only in H.263++ [3]) is the complete decoupling of the transmission time, the decoding time, and the sampling or presentation time of slices and pictures. For this reason, the handling of the RTP timestamp requires some special considerations for those NALUs for which the sampling or presentation time is not defined, or, at transmission time, unknown. 2. Status of JVT, and Changes relative to the -00 version [This section will be removed in a future version of this draft.] 2.1. Status of the JVT standardization, and recent changes to JVT Since the last draft, JVT has met and a new JVT working draft was produced. This JVT working draft is currently in the first stage of the ISO/IEC approval process, the ballot on the so-called Committee Draft. Procedural provisions are taken by interested ISO/IEC members to ensure that changes relative to this draft are still possible, even after the ballot. The meeting brought a lot of changes in the VCL, which do not have a direct influence to this memo. However, there were also numerous changes introduced to the NAL. They somehow break the clean design of the NAL as it was presented at the Minneapolis IETF, in favor to save bits in a byte stream environment. This memo reflects the current JVT working draft, but please see the following section on our expectations regarding future changes of the NAL design. The main changes of the JVT NAL relative to the pre-Fairfax design are as follows: - Introduction of a picture header - A means to carry redundant copies of the picture header - Adding of a "Disposable Flag" to the NALU type. Wenger et. al. Expires December 2002 [Page 3]
Internet Draft 10 June 2002 - Adding many more slice types to the NALU type (were 8, now 30) The next JVT meeting will take place in the week after the Japan IETF in Klagenfurt, Austria. This will be the last meeting in which significant changes (anything but bug fixes) can be done. 2.2. Authors' comments and expectation regarding JVT NAL design The authors deem many of the changes to the NAL as technically problematic, and are working within JVT to fix the freshly introduced and, from the RTP point-of-view, problematic features. The re-introduction of the picture header concept will lead to an undesirable overhead in packet network environments, by making mechanisms such as header repetition necessary. It also breaks the clean Parameter Set concept, making it easier for people to take shortcuts. We know that we can show that the number of bits that can be saved in a byte stream environment through the picture header concept is negligible, and insignificant when compared to the problems the packet world has with this concept. We are confident that we can replace the picture header mechanism with something like a hierarchical Parameter Set concept. If we can convince JVT to go back to the clean JVT NAL design, the number of NALU types (30, plus one for the aggregation packets now) would go down to something more reasonable and freeing codepoint space for future extensions. Otherwise, the draft will require language that recommends the amount of redundant picture header data to be sent. 2.3. Changes relative to draft-wenger-avt-rtp-jvt-00.txt This memo reflects the current JVT WD, and hence required alignment with this draft. In addition to editorial changes (mostly to reflect the changed terminology in the JVT draft), the discussion of the NAL unit types was aligned. As a response to the last IETF meeting's request, the RTP timestamp is now the sampling/presentation timestamp. (It is unclear to us how to distinguish between the two). The RTP clock is now fixed at 90 kHz. Compound Packets are renamed to Aggregation Packets. Since the timestamp now carries vital information, a second type of an aggregation packet is necessary. The compound packet of draft- wenger-avt-rtp-jvt-00.txt can now be used only to aggregate packets that share the same RTP timestamp, and is now called Single-Time Aggregation Packet (STAP). Usually, this packet type can only be used to aggregate packets belonging to the same picture. The second aggregated packet type adds a 16-bit timestamp offset to the Wenger et. al. Expires December 2002 [Page 4]
Internet Draft 10 June 2002 aggregated packet data structure for each of the aggregated NALUs, and is called Multi-Time Aggregation Packet (MTAP). At 90 kHz clock this packet type allows to aggregate NALUs that are roughly 2/3rd's of a second apart. It is believed that such a distance is a good compromise between the requirements of the streaming industry (they want to packetize NALUs belonging to more than one picture into one packet) and the overhead constraints (16 bits per NALU). See section 11 (Open issues) for a more flexible concept. In the JVT meeting a "Disposable Flag" was introduced in the NALU header. That bit is documented here as well. 3. Scope This payload specification can only be used to carry the "naked" JVT NALU stream over RTP. Likely, the first applications of a Standard Track RFC resulting from this draft will be in the conversational multimedia field, video telephone or video conference. The draft is not intended for the use in conjunction with the Byte Stream format of Annex B of the JVT working draft, the MPEG 4 system layer [4] or other multiplexing schemes. 4. NAL basics Tutorial information on the NAL design can be found in [5] and [6]. For the precise definition of the NAL it is referred to [2]. This section tries to provide a very short overview of the concepts used. 4.1. Parameter Set Concept One very fundamental design concept of the JVT codec is to generate self-contained packets, to make mechanisms such as the header duplication of RFC2429 [7] or MPEG-4's HEC [8] unnecessary. (Please see section 2.2 regarding the authors' opinion re the Picture header.) The way how this was achieved is to decouple information that is relevant for more than one slice from the media stream. This higher layer meta information should be sent reliably and asynchronously from the RTP packet stream that contains the slice packets. The combination of the higher level parameters is called a Parameter Set. The Parameter Set contains information such as o picture size, o display window, o optional coding modes employed, o and others. In order to be able to change picture parameters (such as the picture size), without having the need to transmit Parameter Set updates synchronously to the slice packet stream, the encoder and decoder can maintain a list of more than one Parameter Set. Each slice header contains a codeword that indicates the Parameter Set to be used. Wenger et. al. Expires December 2002 [Page 5]
Internet Draft 10 June 2002 This mechanism allows to decouple the transmission of the Parameter Sets from the packet stream, and transmit them by external means, e.g. as a side effect of the capability exchange, or through a (reliable or unreliable) control protocol. It may even be possible that they get never transmitted but are fixed by an application design specification. Although, conceptually, the Parameter Set updates are not designed to be sent in the synchronous packet stream, this memo contains a means to convey them in the RTP packet stream. 4.2. Network Abstraction Layer Packet (NALU) Types All NALUs consist of a single NALU Type octet, which also serves as the payload header. The payload of a NALU follows immediately. The NALU type octet has the following format: +---------------+ |0|1|2|3|4|5|6|7| +-+-+-+-+-+-+-+-+ |E| Type |P|D| +---------------+ E: 1 bit The Error Indication bit, when cleared assures a bit-error free payload of the NALU and of the NALU type octet. When set, the decoder is advised that bit errors may be present in the payload or in the NALU type octet. A prudent reaction of decoders that are incapable of handling bit errors is to discard such packets. Type: 5 bits The NAL Unit payload type as defined in table 8.2 of [2]. P: 1 bit Picture Header Flag. Indicates the presence of a Picture Header at the beginning of the payload. D: 1 bit The Disposable Flag indicates that the payload of the NALU, after decoding, will not be used for future prediction. Hence, the decoder and/or media aware network elements can discard such packets without hurting the codec performance or start error propagation due to predicted coding. However, the user experience will suffer (most likely due to lower frame rates). For a reference of all currently defined NALU types and their semantics please see section 8.2 in [2]. Because we anticipate significant changes to this table, only a few remarks on those NALU types shall be provided here. Wenger et. al. Expires December 2002 [Page 6]
Internet Draft 10 June 2002 NAL Units of the type X Picture Header (where X is Intra, Inter, B, SI, or SP) indicate a payload that consists of a picture header of the indicated type. All NAL Unit types called X slice contain exactly one coded slice of the specified type. In some cases it is also assured that not only this slice, but also all other slices of the coded picture are of the same slice type. This can help the resource allocation process at the decoder. An instantaneous decoder refresh picture (IDER picture) is an I or SI picture that can be used as a random access point. The NAL unit of the types DPB and DPC carry Data Partitions consisting only of Intra and Inter CPBs and coefficients. The Supplemental Enhancement Information type (SEI) is used to carry metadata that is not necessary to keep the loops in encoder and decoder synchronized. A prime example for SEI information is the presentation time in such networks that do not have a time property comparable to the RTP timestamp. Parameter Set Information NALUs (PSIs) are used to carry new Parameter Sets or updates to previous Parameter Sets. Normally, the transmission and update of Parameter Sets is a function of a control protocol and, hence, PSIs SHOULD NOT be used in such systems where adequate protocol support is available. However, there are applications where the packet stream has to be self-contained. In such cases PSIs MAY be used. Severe synchronization problems between the RTP stream containing PSIs and control protocol messages can occur if PSIs and control protocol messages are used in the same RTP session. For this reason, PSIs MUST NOT be used in an RTP session whose Parameter Sets were already changed by control protocol messages during the lifetime of the RTP session. Similarly, control protocol messages MUST NOT be used that affect any RTP session on which at least one PSI was sent. The Parameter Set mechanism is designed to decouple the transmission of picture/GOP/sequence header information from the picture data that is composed of the other NALU types. To successfully decode a picture, all Parameter Sets (referenced by the slice Header) need to be available. Hence, the PSIs (when used) SHOULD be conveyed significantly before their content is first referenced. 4.3. Aggregation Packets Aggregation packets are the packet aggregation scheme of this payload specification. The scheme is introduced to reflect the dramatically different MTU sizes of two target networks -- wireline IP networks (with an MTU size that is often limited by the Ethernet MTU size -- roughly 1500 bytes), and IP or non-IP (e.g. H.324/M) based wireless networks with preferred transmission unit sizes of 254 bytes or less. In order to prevent media transcoding between the two worlds, and to avoid undesirable packetization overhead, a packet aggregation scheme is introduced. Two types of Aggregation packets are defined by this specification: Wenger et. al. Expires December 2002 [Page 7]
Internet Draft 10 June 2002 o Single-Time Aggregation Packet (STAP) aggregate NALUs with identical NALU-time. o Multi-Time Aggregation Packet (MTAP) aggregate NALUs with potentially differing NALU-time. The term NALU-time is defined as the value the RTP timestamp would have if that NALU would be transported in its own RTP packet. MTAP and STAP share the following packetization rules: The disposable flag MUST be set if it is set in all aggregated NALUs, otherwise it MUST be cleared. The Type field of the NALU type octet MUST be zero. The E bit MUST be cleared if all E bits of the aggregated NALUs are zero, otherwise it MUST be set. For MTAPs and STAPs (identified by type = 0 in the NALU type byte) the Picture Header flag is overloaded with a new semantic. A zero in the Picture Header flag indicates a STAP, a one indicates an MTAP. The Marker bit in the RTP header MUST be set to the value the marker bit of the last NALU of the aggregated packet would have if it were transported in its own RTP packet. The NALU Payload of an aggregation packet consists of one or more aggregation units. See section 4.3.1 and 4.3.2 for the two different types of aggregation units. An aggregation packet can carry as many aggregation units as necessary, however the total amount of data in an aggregation packet obviously MUST fit into an IP packet, and the size SHOULD be chosen such that the resulting IP packet is smaller than the MTU size. 4.3.1. Single-Time Aggregation Packet Single-Time Aggregation Packet (STAP) SHOULD be used when aggregating NALUs that share the same NALU-time. The Picture Header Flag MUST be set to zero in order to distinguish an STAP from an MTAP. The NALU payload of an STAP consists of Single-Picture Aggregation units. A Single-Picture Aggregation Unit consists of 16-bit unsigned size information that indicates the size of the following NALU in bytes (excluding these two octets, but including the NALU type octet of the NALU), followed by the NALU itself including its NALU type byte. 4.3.2. Multi-Time Aggregation Packet (MTAP) An MTAP has a similar architecture as an STAP. It consists of the NALU header byte and one or more Multi-Picture Aggregation Units. The Picture Header flag in the MTAP NALU type byte is set to 1 to distinguish an MTAP from an STAP. Wenger et. al. Expires December 2002 [Page 8]
Internet Draft 10 June 2002 This Memo does not specify how the NALUs within an MTAP are ordered. In most cases, the natural "decoding order" SHOULD be used, in particular in conjunction with bi-predicted pictures that use a forward reference picture. However, all other NALU ordering schemes that are legal in JVT video MAY be used as well. A Multi-Picture Aggregation Unit consists of 16 bits unsigned size information of the following NALU (same as the size information of in the STAP). These 16 bits are followed by 16 bits of timing information for this NALU. The timing information field MUST be set so that the RTP timestamp of an RTP packet of each NALU in the MTAP (the NALU-time) can be generated by subtracting the timing information from the RTP timestamp of the MTAP. For the "latest" multi-picture Aggregation Unit in an MTAP the timing offset MUST be zero. Hence, the RTP timestamp of the MTAP itself is identical to the latest NALU-time. 5. RTP Packetization Process The RTP packetization process of the JVT codec is straightforward and follows the general principles outlined in RFC1889. When using one NALU per RTP packet, the RTP payload consists of the bit buffer containing the NALU. The RTP payload (and the settings for some RTP header bits) for aggregation packets were already defined in section 4.3 above. There is no specific RTP payload header -- the NALU type byte double-functions in this task. The RTP header information is set as follows: Timestamp: 32 bits The RTP timestamp is set to the presentation/sampling timestamp of the content. If the NALU has no own timing properties (e.g. PSIs, SEI), or if the presentation/sampling time is unknown, the RTP timestamp is set to the RTP timestamp of the last transmitted RTP packet in the session. The setting of the RTP Timestamp for MTAPs is defined in section 4.3.2 above. Marker bit (M): 1 bit Set for the very last packet of the picture indicated by the RTP timestamp, in line with the normal use of the M bit and to allow an efficient playout buffer handling. Decoders MAY use this bit as an early indication of the last packet of a coded picture, but MUST not rely on this property because the last packet of the picture may get lost, and because the use of MTAPs does not always preserve the M bit. Sequence No (Seq): 16 bit Increased by one for each sent packet. Set to a random value during startup as per RFC1889 Version (V): 2 bits set to 2 Wenger et. al. Expires December 2002 [Page 9]
Internet Draft 10 June 2002 Padding (P): 1 bit set to 0 Extension (X): 1 bit set to 0 Payload Type (PT): 8 bits established dynamically during connection establishment All other RTP header fields are set as per RFC1889. 6. Packetization Rules Two cases of packetization rules have to be distinguished by the possibility to put packets belonging to more than a single picture into a single aggregated packet (using STAPs or MTAPs). 6.1. Unrestricted Mode (Multiple Picture Model) This mode MAY be supported by some receivers. Usually, the capability of a receiver to support this mode is indicated by one of the profiles of the JVT codec (this is not yet defined in [2]). The following packetization rules MUST be enforced by the sender: o Single slice packets belonging to the same picture (and hence share the same RTP timestamp value) MAY be sent in any order, although, for delay critical systems, they SHOULD be sent in their original coding order to minimize the delay. Note that the coding order is not necessarily the scan order, but the order the NAL packets become available to the RTP stack. o Both MTAPs and STAPs MAY be used. o SEI packets MAY be sent anytime. o PSIs MUST NOT be sent in an RTP session whose Parameter Sets were already changed by control protocol messages during the lifetime of the RTP session. If PSIs are allowed by this condition, they MAY be sent at any time. o All NALU types MAY be mixed freely, provided that above rules are obeyed. In particular, it is allowed to mix slices in data-partitioned and single-slice mode. o Network elements MAY convert multiple RTP packets carrying individual NALUs into one aggregated RTP packet, convert an aggregated RTP packet into several RTP packets carrying individual NALUs, or mix both concepts. However, when doing so they SHOULD take into account at least the following parameters: path MTU size, unequal protection mechanisms (e.g. through packet duplication, packet-based FEC carried by RFC2198, especially for header and Type A Data Partitioning packets), bearable latency of Wenger et. al. Expires December 2002 [Page 10]
Internet Draft 10 June 2002 the system, and buffering capabilities of the receiver. o NALUs of all types MAY be conveyed as aggregation units of an STAP or MTAP rather than individual RTP packets. Special care SHOULD be taken (particularly in gateways) to avoid more than a single copy of identical NALUs in a single STAP/MTAP in order to avoid unnecessary data transfers without any improvements of QoS. 6.2. Restricted Mode (Single Picture Model) This mode MUST be supported by all receivers. It is primarily intended for low delay applications. Its main difference from the Unrestricted Mode is to forbid the packetization of data belonging to more than one picture in a single RTP packet. Hence, MTAPs MUST NOT be used. The following packetization rules MUST be enforced by the sender: o All rules of the Unrestricted Mode above, with the following additions o only STAPs MAY be used, MTAPs MUST NOT be used. This implies that aggregated packets MUST NOT include slices or data partitions belonging to different pictures. 7. De-Packetization Process The de-packetization process is implementation dependent. Hence, the following description should be seen as an example of a suitable implementation. Other schemes MAY be used as well. Optimizations relative to the described algorithms are likely possible. The general concept behind these de-packetization rules is to collect all packets belonging to a picture, bringing them into a reasonable order, discard anything that is unusable, and pass the rest to the decoder. Aggregation packets are handled by unloading their payload into individual RTP packets carrying NALUs. Those NALUs are processed as if they were received in separate RTP packets, in the order they were arranged in the Aggregation Packet. The following de-packetization rules MAY be used to implement an operational JVT de-packetizer: o NALUs are presented to the JVT decoder in the order of the RTP sequence number. o NALUs carried in an Aggregation Packet are presented in their order in the Aggregation packet. All NALUs of the Aggregation packet are processed before the next RTP packet is processed. o Intelligent RTP receivers (e.g. in Gateways) MAY identify lost DPAs. If a lost DPA is found, the Gateway MAY decide not to send the DPB and DPC partitions, as their information is meaningless for the JVT Decoder. In this way a network element can reduce network load by discarding useless packets, without parsing a Wenger et. al. Expires December 2002 [Page 11]
Internet Draft 10 June 2002 complex bit stream o Intelligent receivers MAY discard all packets that have the Disposable Flag set. However, they SHOULD process those packets if possible, because the user experience may suffer if the packets are discarded. 8. MIME Considerations This section is to be completed later. 9. Security Considerations So far, no security considerations beyond those of RFC1889 have been identified. Currently, the JVT CD does not allow carrying any type of active payload. However, the inclusion of a "user data" mechanism is under consideration, which could potentially be used for mechanisms such as remote software updates of the video decoder and similar tasks. 10. Informative Appendix: Application Examples This payload specification is very flexible in its use, to cover the extremely wide application space that is anticipated for the JVT codec. However, such a great flexibility also makes it difficult for an implementer to decide on a reasonable packetization scheme. Some information how to apply this specification to real-world scenarios is likely to appear in the form of academic publications and a Test Model in the near future. However, some preliminary usage scenarios should be described here as well. 10.1. Video Telephony, no Data Partitioning, no packet aggregation The RTP part of this scheme is implemented and tested (though not the control-protocol part, see below). In most real-world video telephony applications, the picture parameters such as picture size or optional modes never change during the lifetime of a connection. Hence, all necessary Parameter Sets (usually only one) are sent as a side effect of the capability exchange/announcement process. An example for such a capability exchange with an SDP-like syntax can be found in [9], but other schemes such as ASN.1 are possible as well. Since all necessary Parameter Set information is established before the RTP session starts, there is no need for sending any PSIs. Data Partitioning is not used either. Hence, the RTP packet stream consists basically of NALUs that carry single slices of video information. The size of those single-slice NALUs is chosen by the encoder such that they offer the best performance. Often, this is done by Wenger et. al. Expires December 2002 [Page 12]
Internet Draft 10 June 2002 adapting the coded slice size to the MTU size of the IP network. For small picture sizes this may result in a one-picture-per-one- packet strategy. The loss of packets and the resulting drift- related artifacts are cleaned up by Intra refresh algorithms. 10.2. Video Telephony, Interleaved Packetization using Packet Aggregation This scheme allows better error concealment and is widely used in H.263 based designed using RFC2429 packetization. It is also implemented and good results were reported [5]. The source picture is coded by the VCL such that all MBs of one MB line are assigned to one slice. All slices with even MB row addresses are combined into one STAP, and all slices with odd MB row addresses into another STAP. Those STAPs are transmitted as RTP packets. The establishment of the Parameter Sets is performed as discussed above. Note that the use of STAPs is essential here, because the high number of individual slices (18 for a CIF picture) would lead to unacceptably high IP/UDP/RTP header overhead (unless the source coding tool FMO is used, which is not assumed in this scenario). Furthermore, some wireless video transmission systems, such as H.324M and the IP-based video telephony specified in 3GPP, are likely to use relatively small transport packet size. For example, a typical MTU size of H.223 AL3 SDU is around 100 bytes [10]. Coding individual slices according to this packetization scheme provides a further advantage in communication between wired and wireless networks, as individual slices are likely to be smaller than the preferred maximum packet size of wireless systems. Consequently, a gateway can convert the STAPs used in a wired network to several RTP packets with only one NALU that are preferred in a wireless network and vice versa. 10.3. Video Telephony, with Data Partitioning This scheme is implemented and was shown to offer good performance especially at higher packet loss rates [5]. Data Partitioning is known to be useful only when some form of unequal error protection is available. Normally, in single-session RTP environments, even error characteristics are assumed -- statistically, the packet loss probability of all packets of the session is the same. However, there are means to reduce the packet loss probability of individual packets in an RTP session. One simple way is known as Packet Duplication: simply send the to-be- protected packet twice, with the same sequence number. If both packets survive, the receiver will assume a packet duplication by UDP and discard one of the two packets. Other means of unequal protection within the same RTP session include the use of RFC 2198 [11] (for this application it is essentially a packet duplication process as well, with some saved bytes for the second RTP header), or packet-based Forward Error Correction [12] carried in RFC2198. Wenger et. al. Expires December 2002 [Page 13]
Internet Draft 10 June 2002 The implemented software uses the simple packet duplication process to increase the probability of all DPA NALUs. The incurred overhead is substantial, but in the same order of magnitude as the number of bits that have otherwise be spent for intra information. However, this mechanism is not adding any delay to the system. Again, the complete Parameter Set establishment is performed through control protocol means. 10.4. MPEG-2 Transport to RTP Gateway This example is not implemented completely, but the basic mechanisms are part of the interim file format the JVT group uses and, hence, well tested. When using JVT video in satellite/cable broadcast environments, there is no control protocol available that can be used for the transmission of Parameter Sets. Furthermore, a receiver has to be able to "tune" into an ongoing packet stream at any time, without much delay and artifacts. For this reason, PSIs that contain all Parameter Set information are included in the packet stream at any Instantaneous Decoder Refresh Point (which are similar to Key Frames in earlier coding standards). IDERP packets are used to signal these "key frames" so that a decoder can most easily determine where to start in its decoding process. Since the byte stream format used in satellite/cable broadcast environments does not include timing information in the video stream, the gateway needs to use external timing information (e.g. from the MPEG-2 system layer) to generate the RTP timestamp. Please note that this timestamp is also a 90 kHz clock -- hence, in most cases, the conversion should be relatively simple. The simplest possible MPEG-2 transport to RTP gateway could take the NALUs as they come from the MPEG-2 transport stream (after de- framing), and send them, each NALU in one RTP packet, with increasing RTP sequence numbers. However, less than perfect packet loss rates would lead to a very poor performance of such a system. However, a Gateway could use the protection mechanisms discussed above to unequally protect the most important packets, e.g. all PSIs (very strong protection) IDERPs (weak protection), and transmit everything else best effort. The Gateway can do this without parsing the bit stream, by simply using the NALU type byte. A more sophisticated Gateway may be able to combine some small NALUs to a big STAP or MTAP in order to save the bytes used for the IP/UDP/RTP headers. A similar mechanism is, of course, also possible in H.320 to RTP gateways. Here, however, the system environment does not include any timing information, and exact presentation timing is carried in the form of SEIs. Hence, in the H.320 to IP data path, the gateway has the additional duty to filter out SEIs containing timing information and setting the RTP timestamp of the following video Wenger et. al. Expires December 2002 [Page 14]
Internet Draft 10 June 2002 packets accordingly. In the reverse direction, SEIs need to be generated using the RTP timestamp as a guideline. 10.5. Low-Bit-Rate Streaming This scheme has been implemented with H.263 and gave good results [13]. There is no technical reason why similarly good results could not be achievable using the JVT codec. In today's Internet streaming, some of the offered bit-rates are relatively low in order to allow terminals with dial-up modems to access the content. In wired IP networks, relatively large packets, say 500 - 1500 bytes, are preferred to smaller and more frequently occurring packets in order to reduce network congestion. Moreover, use of large packets decreases the amount of RTP/UDP/IP header overhead. For low-bit-rate video, the use of large packets means that sometimes up to few pictures should be encapsulated in one packet. However, loss of such a packet would have drastic consequences in visual quality, as there is practically no other way to conceal a loss of an entire picture than to repeat the previous one. One way to construct relatively large packets and maintain possibilities for successful loss concealment is to construct MTAPs that contain slices from several pictures in an interleaved manner. An MTAP should not contain spatially adjacent slices from the same picture or spatially overlapping slices from any picture. If a packet is lost, it is likely that a lost slice is surrounded by spatially adjacent slices of the same picture and spatially corresponding slices of the temporally previous and succeeding pictures. Consequently, concealment of the lost slice is likely to succeed relatively well. 11. Open Issues There are several open issues on which the authors would like to receive opinions. They are listed below. MTAPs: are they efficient enough? And, is 16 bit unsigned offset to a 90 kHz timestamp enough? Need input from the streaming industry. One solution would be to create five different xTAP, with 0, 8, 16, 24, and 32 bit timestamps per aggregation unit. Another option would be a more complex payload header that signals presence (and size) of the timing information per aggregation unit. Since JVT will likely be approved as the advanced video codec of MPEG-4, it may be desirable to align this payload specification with other payload specifications for MPEG 4. The authors of this I-D and some authors of the MPEG-4 packetization I-Ds are discussing the issue, and there is a chance that in the future changes to this I-D will be proposed to AVT to reflect the outcome of these discussions. 12. Full Copyright Statement Wenger et. al. Expires December 2002 [Page 15]
Internet Draft 10 June 2002 Copyright (C) The Internet Society (2002). All Rights Reserved. This document and translations of it may be copied and furnished to others, and derivative works that comment on or otherwise explain it or assist in its implementation may be prepared, copied, published and distributed, in whole or in part, without restriction of any kind, provided that the above copyright notice and this paragraph are included on all such copies and derivative works. However, this document itself may not be modified in any way, such as by removing the copyright notice or references to the Internet Society or other Internet organizations, except as needed for the purpose of developing Internet standards in which case the procedures for copyrights defined in the Internet Standards process must be followed, or as required to translate it into languages other than English. The limited permissions granted above are perpetual and will not be revoked by the Internet Society or its successors or assigns. This document and the information contained herein is provided on an "AS IS" basis and THE INTERNET SOCIETY AND THE INTERNET ENGINEERING TASK FORCE DISCLAIMS ALL WARRANTIES, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO ANY WARRANTY THAT THE USE OF THE INFORMATION HEREIN WILL NOT INFRINGE ANY RIGHTS OR ANY IMPLIED WARRANTIES OF MERCHANTABILITY OR FITNESS FOR A PARTICULAR PURPOSE. 13. Bibliography [1] P. Borgwardt, "Handling Interlaced Video in H.26L", VCEG-N57r2, available from ftp://standard.pictel.com/video- site/0109_San/VCEG-N57r2.doc, September 2001 [2] JVT Joint Committee Draft, available from ftp://ftp.imtc- files.org/jvt-experts/2002_05_Fairfax/JVT-C167.doc [3] ITU-T Recommendation H.263-2000 [4] ISO/IEC IS 14496-1 [5] S. Wenger, "H.26L over IP", IEEE Transaction on Circuits and Systems for Video technology, to appear (April 2002) [6] S. Wenger, "H.26L over IP: The IP Network Adaptation Layer", Proceedings Packet Video Workshop 02, April 2002, to appear. [7] C. Borman et. Al., "RTP Payload Format for the 1998 Version of ITU-T Rec. H.263 Video (H.263+)", RFC 2429, October 1998 [8] ISO/IEC IS 14496-2 [9] S. Wenger, T. Stockhammer, "H.26L over IP and H.324 Framework", VCEG-N52, available from ftp://standard.pictel.com/video- site/0109_San/VCEG-N52.doc, September 2001 [10] ITU-T Recommendation H.223 (1999) [11] C. Perkins et. al., "RTP Payload for Redundant Audio Data", RFC 2198, September 1997 Wenger et. al. Expires December 2002 [Page 16]
Internet Draft 10 June 2002 [12] J. Rosenberg, H. Schulzrinne, "An RTP Payload Format for Generic Forward Error Correction", RFC 2733, December 1999 [13] V Varsa, M. Karczewicz, "Slice interleaving in compressed video packetization", Packet Video Workshop 2000 Author's Addresses Stephan Wenger Phone: +49-172-300-0813 TU Berlin / Teles AG Email: stewe@cs.tu-berlin.de Franklinstr. 28-29 D-10587 Berlin Germany Thomas Stockhammer Phone: +49-89-28923474 Institute for Communications Eng. Email: stockhammer@ei.tum.de Munich University of Technology D-80290 Munich Germany Miska M. Hannuksela Phone: +358 40 5212845 Nokia Corporation Email: miska.hannuksela@nokia.com P.O. Box 68 33721 Tampere Finland Wenger et. al. Expires December 2002 [Page 17]