Internet Engineering Task Force Audio/Video Transport Working Group INTERNET DRAFT Balabanian-Nortel draft-ietf-avt-rtp-mpeg4-dmif-00.txt March 11,1998 Expires: September 16,1998 The Role of DMIF in Support of RTP MPEG-4 Payloads STATUS OF THIS MEMO This document is an Internet-Draft. Internet-Drafts are working documents of the Internet Engineering Task Force (IETF), its areas, and its working groups. Note that other groups may also distribute working documents as Internet-Drafts. Internet-Drafts are draft documents valid for a maximum of six months and may be updated, replaced, or made obsolete by other documents at any time. It is inappropriate to use Internet-Drafts as reference material or to cite them other than as ``work in progress''. To learn the current status of any Internet-Draft, please check the ``1id-abstracts.txt'' listing contained in the Internet-Drafts Shadow Directories on ftp.is.co.za (Africa), nic.nordu.net (Europe), munnari.oz.au (Pacific Rim), ds.internic.net (US East Coast), or ftp.isi.edu (US West Coast). Distribution of this document is unlimited. ABSTRACT This memorandum describes how RTP carrying MPEG-4 payloads interacts with the MPEG-4 Access Unit layer through the MPEG (Delivery Multimedia Integration Framework) DMIF. DMIF is used to pass information essential for the packing and unpacking of MPEG-4 streams into RTP as well as for adjusting the MPEG-4 AL-PDU lengths to be within the path MTU. DMIF interprets the RTCP reports by comparing its statistics to the requested MPEG-4 media based QoS. If the statistics fail to meet the requested QoS then action is taken to either continue with the impaired perforamce, upgrade the network service class, scale down the stream or delete the stream. This specification is a product of the Audio/Video Transport working group within the Internet Engineering Task Force. Comments are solicited and should be addressed to the working group's mailing list at rem-conf@es.net and/or the authors. Balabanian [Page 1]
Internet Draft Role of DMIF with RTP MPEG-4 Payloads March 11,1998 1 Introduction MPEG-4 is a recent standard from ISO/IEC for the coding of natural and synthetic audio-visual data in the form of audiovisual objects that are arranged into an audiovisual scene by means of a scene description [1] [2][3][4]. This memorandum specifies how DMIF is used to facilitate the operation of the RTP MPEG-4 payloads [5][6]. This memorandum provides a solution for discussion in IETF AVT and ISO/IEC MPEG technical communities in order to identify issues in using of MPEG-4/DMIF with RTP and incorporate the results. This would lead to the finalization of the specification on RTP use of MPEG-4 with DMIF. 1.1 Overview of MPEG-4 End-System Architecture Figure 1 below shows the general architecture of MPEG-4 terminals. The Compression Layer processes individual audio-visual media streams without regard to delivery technologies. The compression schemes in MPEG-4 achieve efficient encoding over a wide range from Kbps to multiple Mbps. The MPEG-4 compression schemes are defined in the ISO/IEC specifications 14496-2 and -3 [2][3]. The media content at this layer is organized in Elementary Streams. The MPEG-4 Systems specification, ISO/IEC 14496-1 [1], defines the concepts needed to describe the relations between Elementary Streams in a way that allows to create distributed, yet integrated, content presentations and to synchronize the streams. This part of the specification is both media unaware and delivery technology unaware. The Delivery Layer in MPEG-4 consists of the Delivery Multimedia Integration Framework defined in ISO/IEC 14496-6 [4]. This layer is media unaware but delivery technology aware. It provides transparent access to and delivery of content irrespective of the technologies used. The interface between the AU Layer and DMIF is called DMIF Application Interface (DAI). It offers content location independent procedures for establishing MPEG-4 sessions and access to transport channels. Balabanian [Page 2]
Internet Draft Role of DMIF with RTP MPEG-4 Payloads March 11,1998 media aware +-----------------------------------------+ delivery unaware | COMPRESSION LAYER | 14496-2 Visual |streams from as low as Kbps to multi-Mbps| 14496-3 Audio +-----------------------------------------+ Elementary Stream =============================================================Interface (ESI) +-------------------------------------------+ media and | SYSTEMS LAYER | delivery unaware | manages elementary streams, their synch- | 14496-1 Systems | ronization and hierarchical relations | +-------------------------------------------+ DMIF Application ==============================================================Interface (DAI) +-------------------------------------------+ delivery aware | DELIVERY LAYER | media unaware |provides transparent access to and delivery| 14496-6 DMIF | of content irrespective of delivery | | technology | +-------------------------------------------+ Figure 1: General MPEG-4 terminal architecture 1.2 The DMIF Model DMIF as an integration framework uses a uniform procedure at the DAI interface to access the MPEG-4 content irrespective whether the content is broadcast, stored on a local file or obtained through interaction with a remote end-system. The model is shown in Figure 2 below. The specific instance of interest in this memorandum is the interaction with a remote end-system. For this case DMIF uses internal (informative) DMIF-Network Interface(DNI)to map the controls obtained from the application through DAI into the various signaling appropriate to the various networks. Balabanian [Page 3]
Internet Draft Role of DMIF with RTP MPEG-4 Payloads March 11,1998 ! +---+ +-----------+ +-----------+ +-----------+ ! | | |Local DMIF | |Remote DMIF| |Remote App.| +-----+ ! | D | |for Brd'cst| |(locally | |(locally | Brd'cst | | ! | M | | | | emulated) | | emulated) |<-----source |Local| ! | I | +-----------+ +-----------+ +-----------+ | | ! | F | +-----------+ +-----------+ +-----------+ |App | ! | | |Local DMIF | |Remote DMIF| |Remote App.| File | | ! | F | |for Local | |(locally | |(locally |<-----source | | ! | i | | Files | | emulated) | | emulated) | | | ! | l | +-----------+ +-----------+ +-----------+ | | ! | t | +-----------+ ! +---+ / ---- \ | | ! | e | |Local DMIF | ! |Sig| | --- ---- | +-----+ ! | r | |for Remote | ! |map|<->( Network ) |Outside the ! | | | Service | ! | | | ---- ----- |scope of DMIF ! +---+ +-----------+ ! +---+ | ----- | DAI DNI | ^ | \_____|_________/ | | | +---+!+------+!+------+ | |Sig|!|Remote|!|Remote| +>|map|!| DMIF |!| App. | | |!|(real)|!|(real)| +---+!+------+!+------+ DNI DAI Balabanian [Page 4]
Internet Draft Role of DMIF with RTP MPEG-4 Payloads March 11,1998 1.3 Mapping between MPEG-4/DMIF and RTP Figure 3 below draws the correspondence between RTP and MPEG-4/DMIF. It is noted that DAI signaling allows the establishment of an MPEG-4 Service e.g., Video on Demand, the request of channels to carry MPEG-4 Elementary Streams for that service and the reading of Elementary Stream data when received. RTP MPEG-4/DMIF +-----------------------------+ +---------------------------+ / DATA TRANSPORT CONTROL \ / DATA TRANSPORT \ (RTP) (RTCP) | (Elementary Stream) Audio, Video SR, RR, SDES . Audio, Video Simulated Data BYE, APP | Simulated Data . ^ ^ | ^ I | . I CONTROL I | | I (Scene, Object I v . I Descriptor(OD)) I Corresponds to | I ^ I DAI SIGNALING . I | I | I | I . I / I | I / I . I / | I / In addition to . I / SIGNALING Audio, Video | I/ (Service, Channel, Simulated Data . I Data) to include Scene | I ^ and OD streams . ====I==========|=== DAI | I | . I v | Elementary Streams . in AL-PDU fragments Figure 3: Drawing some correspondence between MPEG-4/DMIF and RTP The AL-PDUs can be mapped to RTP-PDUs as follows [6]: RTP-PDU 1:1 AL-PDU RTP-PDU 1:N AL-PDU RTP-PDU N:1 AL-PDU The selection from above choices is based on the size of the AL-PDU with respect to the RTP-PDU MTU (IP) size. The first choice can use MPEG-4 single stream RTP payload type. The second case can use MPEG-4 FlexMux RTP payload type. The last choice will also use MPEG-4 FlexMux RTP payload type. The last situation occurs when MPEG-4 Access Unit is not able to adjust the MPEG-4 AL-PDU lengths to be within the path MTU. Balabanian [Page 5]
Internet Draft Role of DMIF with RTP MPEG-4 Payloads March 11,1998 1.4 MPEG-4 Media Based RTP QoS The following Media based QoS can be derived for the different mappings. 1.4.1 The case RTP-PDU 1:1 or N:1 AL-PDU +=====================+=====================================+ | RTP Media | Derivation from the | | QoS_QualifierTag | ES Media transport-QoS | +=====================+=====================================+ | MAX_DELAY of RTP PDU|Maximum delay per AU (microseconds) | | |measured over 1 sec. | +---------------------+-------------------------------------+ | AVG_DELAY of RTP PDU|Average delay per AU allowed | | |(microseconds) measured over 1 min | +---------------------+-------------------------------------+ | Dejitter Buffer | Adjusted for the overhead of the | | for the RTP stream | RTP PDU | +---------------------+-------------------------------------+ | LOSS_PROB of RTP PDU|Probability of loss of any single AU | | |(Fraction (0.00 - 1.00) over 1 sec. | +---------------------+-------------------------------------+ | MAX_GAP_LOSS |Maximum allowable consecutive AU loss| | of RTP PDUs |measured over 1 sec | +===========================================================+ | MAX_RTP_SIZE |Maximum size of an AU (Bytes) | | |Plus AL-PDU and RTP overhead | +---------------------+-------------------------------------+ | MAX_RTP_RATE |Maximum arrival rate of AUs | | |(RTP-PDU/second) | +---------------------+-------------------------------------+ | AVG_RTP_SIZE |Average size of AUs (Bytes) | | |Plus AL-PDU and RTP overhead | +---------------------+-------------------------------------+ Balabanian [Page 6]
Internet Draft Role of DMIF with RTP MPEG-4 Payloads March 11,1998 1.4.2 The case RTP-PDU 1:N AL-PDU +=====================+=====================================+ | RTP Media | Derivation from the | | QoS_QualifierTag | ES Media QoS | +=====================+=====================================+ | MAX_DELAY of RTP PDU|Least Maximum delay per AU from | | |among the N AL-PDUs(microseconds) | | |measured over 1 sec. | +---------------------+-------------------------------------+ | AVG_DELAY of RTP PDU|Average delay per AU allowed | | |(microseconds) measured over 1 min. | +---------------------+-------------------------------------+ | Dejitter Buffer | Total of deitter buffers adjusted | | for the RTP stream | for the overhead of the RTP PDU | +---------------------+-------------------------------------+ | LOSS_PROB of RTP PDU|Least Probability of loss of any | | |single AU from the N AL-PDUs | | |(Fraction (0.00 - 1.00) over 1 sec. | +---------------------+-------------------------------------+ | MAX_GAP_LOSS |Least Maximum allowable consecutive | | of RTP PDUs |AU loss for any one of the N AL-PDUs | | | measured over 1 sec | +===========================================================+ | MAX_RTP_SIZE |Sum of the MAX_AU_SIZEs of from | | |each of the N AL-PDUs Plus AL-PDU | | |and RTP overhead | +---------------------+-------------------------------------+ | MAX_RTP_RATE |Highest MAX_AU_RATE of AUs from each | | |of the N AL-PDUs (RTP-PDU/second) | +---------------------+-------------------------------------+ | AVG_RTP_SIZE |Sum of Average size of AUs from | | |each of the N AL-PDUs Plus | | |AL-PDU and RTP overhead (Bytes) | +---------------------+-------------------------------------+ Note all the Streams chosen for encapsulation with RTP belong to the same priority level. 2 Operation of the RTP MPEG-4 payloads with DMIF Figure 4 and 5 show the conceptual operation of the MPEG-4/DMIF with RTP. The DAI signaling shall be used to set up the MPEG-4 session. When the RTP is used A local (or remote) DMIF entity could be used to start the RTP session with its corresponding RTP data transport (carrying one or more MPEG-4 Elementary Streams) and RTCP for control. Figure 4 shows that in case of a single MPEG-4 stream payload type, the ALConfigDescriptor is being received at the sender side and being used both at the sender and the receiver for efficient packing of the stream into the RTP transport [6]. Balabanian [Page 7]
Internet Draft Role of DMIF with RTP MPEG-4 Payloads March 11,1998 ^ | | +---+ SIGNALING |AL | (MTU + ALConfigDescriptor) +---+ ^ ^ | | ===================|=================================|===DAI=== | v +------------------------------------+ +---------------+ | Regenerate AL-PDUs |<------| DMIF | +------------------------------------+ . | | MPEG-4 ^ Time ^ Sequence^ . +---------------+ Payloads | Stamps | Numbers| ALConfig ^ ^ | | | Descriptor | | +-----------------------+ v | | RTP Parser | +-----------+| +-----------------------+ |RTCP Parser|| ^ +-----------+| I ^ | I | | I v v ~~~~~~~~DNI Figure 4: Conceptual view of the operation of DMIF with MPEG-4 single stream RTP Payload type Figure 5 shows that in the case of MPEG-4 FlexMux RTP payload type, information for the MTU and ALConfig is not required. The FlexMux decoder however needs the MuxCode information which is generated at the sending end by the FlexMux muxing code and passed to the receiver through the DMIF signaling. Balabanian [Page 8]
Internet Draft Role of DMIF with RTP MPEG-4 Payloads March 11,1998 Audio, Video, Simulated Data, Scene, ODs ^ ^ ^ ^ | | | | | | | | +---+ +---+ +---+ +---+ |AL | |AL | |AL |. . . . . . |AL | +---+ +---+ +---+ +---+ SIGNALING ^ ^ ^ ^ ^ | | | | | ==|====|======|=====================|====================|===DAI=== | | | | v +--------------------------------------+ +---------------+ | FlexDemux |<------| DMIF | +--------------------------------------+ . | | MPEG-4 ^ Time ^ Sequence^ . +---------------+ Payloads | Stamps | Numbers| . ^ ^ | | | MuxCode | | +-----------------------+ Descriptor v | | RTP Parser | +-----------+| +-----------------------+ |RTCP Parser|| ^ +-----------+| I ^ | I | | I v v ~~~~~~~~DNI Figure5: Conceptual view of the operation of DMIF with MPEG-4 FlexMux RTP Payload type 3 RTCP Sender and Receiver Reports RTP receivers provide reception quality feedback using RTCP report packets which may take one of two forms depending upon whether or not the receiver is also a sender. These reports shall be used by DMIF in the case of MPEG-4/DMIF-RTP to readjust the demand put on the network based on a predefined policy which may involve a decision to be made by the user. The sender report packet consists of three sections, possibly followed by a fourth profile-specific extension section if defined (none has been specified so far for MPEG-4 RTP payloads). The third section contains zero or more reception report blocks depending on the number of other sources heard by this sender since the last report. Each reception report block conveys statistics on the reception of RTP packets from a single synchronization source. SSRC_n (source identifier): The SSRC identifier of the source to which the information in this reception report block pertains. This SSRC may either relate to an MPEG-4 single or FelxMux RTP payload. Balabanian [Page 9]
Internet Draft Role of DMIF with RTP MPEG-4 Payloads March 11,1998 fraction lost: The fraction of RTP data packets from source SSRC_n lost since the previous SR or RR packet was sent, expressed as a fixed point number. This fraction is compared to the LOSS_PROB. It is important that the duration over which this metric is measured corresponds to the same duration used to express the LOSS_PROB. If the statistics consistently exceeds the LOSS_PROB then the policy enforcer is brought into action. As a result the load on the RTP stream is reduced. Note: RTCP does not provide a measure for consecutive RTP-PDU loss in order to be able to calculate the GAP_LOSS. interarrival jitter: An estimate of the statistical variance of the RTP data packet interarrival time, measured in timestamp units and expressed as an unsigned integer. The jitter calculation in RTCP is based on the variation of consecutive interarrival times: If Si is the RTP timestamp from packet i, and Ri is the time of arrival in RTP timestamp units for packet i, then for two packets i and j, D may be expressed as D(i,j)=(Rj-Ri)-(Sj-Si)=(Rj-Sj)-(Ri-Si) The interarrival jitter is calculated continuously as each data packet i is received from source SSRC_n, using this difference D for that packet and the previous packet i-1 in order of arrival (not necessarily in sequence), according to the formula J=J+(|D(i-1,i)|-J)/16 Whenever a reception report is issued, the current value of J is sampled. The J value must be converted to msecs J J must be <= to 500*Dejitter_Buffer/( MAX_RTP_SIZE* MAX_RTP_RATE) If this value is exceeded consistently for some time then the QoS policy enforcer is brought into action. As a result the load on the RTP stream is reduced. Round trip Delay: This delay is calculated by measuring the time sending a sender report and receiving the associated receiver report and subtracting the delay it took to send the receiver report at the receiver. Delay must be <= .5*MAX_DELAY converted to seconds from microseconds Average Delay over 1 minute <= .5*AVG_DELAY converted to seconds from microseconds Balabanian [Page 10]
Internet Draft Role of DMIF with RTP MPEG-4 Payloads March 11,1998 If either of the above values is exceeded consistently for some time then the QoS policy enforcer is brought into action. As a result the load on the RTP stream is reduced. 4. BYE: Goodbye RTCP packet When a single stream in case of the MPEG-4 single stream RTP payload, a BYE packet is sent along with the DS_ChannelDelete using DMIF-DMIF signaling. At the receiver either a BYE packet or DS_ChannelDelete signal will cause DAI to pass DA_ChannelDelete to the application. When a FlexMux stream is used, the BYE packet is generated when no longer any MPEG-4 streams are carried on the RTP session. This means that DS_ChannelDeletes have already been sent for all the channels carried on the RTP session and the application has been notified by DA_ChannelDelete(s) across the DAI. A BYE message is followed by a DS_TransMuxDelete which at the reception will allow both the sending and receiving DMIF sides to reuse the RTP/IP ports. A Authors' Addresses Vahe Balabanian Nortel P.O.Box 3511, St. C Ottawa, Ontario Canada K1Y 4H7 Email: balabani@nortel.ca B Bibliography [1] ISO/IEC 14496-1 CD MPEG-4 Systems Oct. 1997 [2] ISO/IEC 14496-2 CD MPEG-4 Visual Oct. 1997 [3] ISO/IEC 14496-3 CD MPEG-4 Audio Oct. 1997 [4] ISO/IEC 14496-6 CD Delivery Multimedia Integration Framework,Oct. 1997 [5] Schulzrinne, Casner, Frederick, Jacobson RTP: A Transport Protocol for Real Time Applications RFC 1889,Internet Engineering Task Force, Jan. 1996. [6] Carsten, Balabanian, Basso, Civanlar, hoffman, Speer, Schulzrinne, RTP payload format for MPEG-4 Elementary Streams draft-ietf-avt-rtp-mpeg4, Internet Engineering Task Force, March 1998. Internet Engineering Task Force Audio/Video Transport Working Group INTERNET DRAFT Balabanian-Nortel draft-ietf-avt-rtp-mpeg4-dmif-00.txt March 11,1998 Expires: September 16,1998 Balabanian [Page 11]
