Network Working Group C. Jennings
Internet-Draft cisco
Intended status: Standards Track R. Logan
Expires: 27 April 2023 Cisco
24 October 2022
Game State over Real Time Protocol
draft-jennings-dispatch-game-state-over-rtp-01
Abstract
This specification defines an Real Time Protocol (RTP) payload to
send game moves and the state of game objects over RTP. This is
useful for games as well collaboration systems that use augment or
virtual reality.
RTP provide a way to synchronize game state between players with
robust technique for recovery from network packet loss while still
having low latency.
Status of This Memo
This Internet-Draft is submitted in full conformance with the
provisions of BCP 78 and BCP 79.
Internet-Drafts are working documents of the Internet Engineering
Task Force (IETF). Note that other groups may also distribute
working documents as Internet-Drafts. The list of current Internet-
Drafts is at https://datatracker.ietf.org/drafts/current/.
Internet-Drafts are draft documents valid for a maximum of six months
and may be updated, replaced, or obsoleted by other documents at any
time. It is inappropriate to use Internet-Drafts as reference
material or to cite them other than as "work in progress."
This Internet-Draft will expire on 27 April 2023.
Copyright Notice
Copyright (c) 2022 IETF Trust and the persons identified as the
document authors. All rights reserved.
This document is subject to BCP 78 and the IETF Trust's Legal
Provisions Relating to IETF Documents (https://trustee.ietf.org/
license-info) in effect on the date of publication of this document.
Please review these documents carefully, as they describe your rights
and restrictions with respect to this document. Code Components
Jennings & Logan Expires 27 April 2023 [Page 1]
Internet-Draft Game State over RTP October 2022
extracted from this document must include Simplified BSD License text
as described in Section 4.e of the Trust Legal Provisions and are
provided without warranty as described in the Simplified BSD License.
Table of Contents
1. Overview . . . . . . . . . . . . . . . . . . . . . . . . . . 3
2. Goals: . . . . . . . . . . . . . . . . . . . . . . . . . . . 4
3. Primitives . . . . . . . . . . . . . . . . . . . . . . . . . 4
3.1. Location . . . . . . . . . . . . . . . . . . . . . . . . 4
3.2. Scale . . . . . . . . . . . . . . . . . . . . . . . . . . 4
3.3. Normal . . . . . . . . . . . . . . . . . . . . . . . . . 5
3.4. TextureUV . . . . . . . . . . . . . . . . . . . . . . . . 5
3.5. Rotation . . . . . . . . . . . . . . . . . . . . . . . . 5
3.6. Child Transform . . . . . . . . . . . . . . . . . . . . . 6
3.7. Texture URL . . . . . . . . . . . . . . . . . . . . . . . 6
3.8. Mesh URL . . . . . . . . . . . . . . . . . . . . . . . . 6
3.9. Texture Stream . . . . . . . . . . . . . . . . . . . . . 6
3.10. Timestamp . . . . . . . . . . . . . . . . . . . . . . . . 7
4. Objects . . . . . . . . . . . . . . . . . . . . . . . . . . . 7
4.1. Common Objects . . . . . . . . . . . . . . . . . . . . . 7
4.1.1. Generic Game Object . . . . . . . . . . . . . . . . . 7
4.1.2. Player Head . . . . . . . . . . . . . . . . . . . . . 8
4.1.3. Mesh . . . . . . . . . . . . . . . . . . . . . . . . 8
4.1.4. External Mesh . . . . . . . . . . . . . . . . . . . . 9
4.1.5. Player Hand . . . . . . . . . . . . . . . . . . . . . 9
4.2. Common Controls . . . . . . . . . . . . . . . . . . . . . 11
4.2.1. Three Degree of Freedom (3DOF) Controller . . . . . . 11
4.2.2. Six Degree of Freedom (6DOF) Controller . . . . . . . 11
4.2.3. Game Controller . . . . . . . . . . . . . . . . . . . 11
5. Encoding . . . . . . . . . . . . . . . . . . . . . . . . . . 13
5.1. Tag . . . . . . . . . . . . . . . . . . . . . . . . . . . 13
5.2. Float . . . . . . . . . . . . . . . . . . . . . . . . . . 13
5.3. Boolean . . . . . . . . . . . . . . . . . . . . . . . . . 14
5.4. Integer . . . . . . . . . . . . . . . . . . . . . . . . . 14
5.5. String . . . . . . . . . . . . . . . . . . . . . . . . . 14
5.6. Blob . . . . . . . . . . . . . . . . . . . . . . . . . . 14
6. Full Intra Request . . . . . . . . . . . . . . . . . . . . . 15
7. RTP and SDP . . . . . . . . . . . . . . . . . . . . . . . . . 15
8. IANA . . . . . . . . . . . . . . . . . . . . . . . . . . . . 15
8.1. Media Type Registration . . . . . . . . . . . . . . . . . 15
8.2. Game State Tag Registry . . . . . . . . . . . . . . . . . 16
9. Security . . . . . . . . . . . . . . . . . . . . . . . . . . 17
10. Normative References . . . . . . . . . . . . . . . . . . . . 17
11. Informative References . . . . . . . . . . . . . . . . . . . 18
Appendix A. Acknowledgments . . . . . . . . . . . . . . . . . . 18
Appendix B. Implementations . . . . . . . . . . . . . . . . . . 18
Appendix C. Test Vectors . . . . . . . . . . . . . . . . . . . . 18
Jennings & Logan Expires 27 April 2023 [Page 2]
Internet-Draft Game State over RTP October 2022
C.1. Head Location . . . . . . . . . . . . . . . . . . . . . . 18
Appendix D. Encode API . . . . . . . . . . . . . . . . . . . . . 19
Appendix E. Decode API . . . . . . . . . . . . . . . . . . . . . 19
Appendix F. EBNF . . . . . . . . . . . . . . . . . . . . . . . . 20
Authors' Addresses . . . . . . . . . . . . . . . . . . . . . . . 24
1. Overview
Many real time applications, such as games, want to to share state
about 3D objects across the network. This specification allows an
application to define objects with state, and the current values of
that state over RTP [RFC3550].
The conceptual model is each RTP sender has a small number of objects
with state that needs to be synchronized to the other side. The
current values are periodically sent over RTP. They MAY be sent when
the values change, but they MUST also be sent periodically so that
any lost updates are eventually received and state is consistent
between the sender and receiver.
The state sent can include a time stamp and rate change estimates
that allow the receiver to estimate the current state values even at
a point in the future. An application that receives a state update
can apply it immediately (often called immediate based), wait a fixed
delay and then apply state changes (often called delay based), or
apply a predicted value based on overwriting any previous predictions
(often called rollback based).
In many cases the state does not have any units but if does, SI units
SHOULD be used. Unless otherwise defined by the application, the
default coordinate system SHOULD be a left handed coordinate system
where Y points up and X points to the right.
Applications can define their own objects or use some of the
predefined common objects. Each object is identified by a type and
identifier number to uniquely identify the object within the scope of
that sender's RTP stream. Multiple updates and objects can be
combined in a single RTP packet so that they are guaranteed to be
fate shared and either atomically delivered at the same time or not
delivered at all to the receiver.
Jennings & Logan Expires 27 April 2023 [Page 3]
Internet-Draft Game State over RTP October 2022
The objects are defined as a series of primitives that define common
types. The objects and updates to state are encoded with Tag Length
Value (TLV) style encoding so that receivers can skip objects they do
not understand. The Objects in an single RTP packet MUST be
processed in order. This allows a sender to write state in an old
format followed by a new format, allowing the new format to override
values in the old format. This allows for easy upgrade of the
protocol with backwards compatibility.
2. Goals:
* Support 2D and 3D
* Support delay and rollback based synchronization
* Relatively compact, simple encoding
* Extensible for applications to send custom data
* Support for forward and backwards compatibility
3. Primitives
This section defines primitives that are useful in defining objects.
The definition are in W3C style EBNF [https://www.w3.org/TR/2010/REC-
xquery-20101214/#EBNFNotation (https://www.w3.org/TR/2010/REC-xquery-
20101214/#EBNFNotation)].
3.1. Location
Loc1 ::=
Float32 /* x */
Float32 /* y */
Float32 /* z */
Loc1 is simply a 3D location of a point stored in Float32;
Loc2 ::=
Float32 /* x */
Float32 /* y */
Float32 /* z */
Float16 /* vx */
Float16 /* vy */
Float16 /* vz */
Loc2 has a location as Float32 followed by the rate of change in
location per second as Float16.
3.2. Scale
Scale1 ::=
Float16 /* all dimensions */
Jennings & Logan Expires 27 April 2023 [Page 4]
Internet-Draft Game State over RTP October 2022
Scale1 will scale the object in all dimensions equally.
Scale2 ::=
Float32 /* x */
Float32 /* y */
Float32 /* z */
Float16 /* vx */
Float16 /* vy */
Float16 /* vz */
Scale2 has a scale in each axis as Float32 followed by the rate of
change in the scale per second as Float16.
3.3. Normal
Norm1 ::=
Float16 /* nx */
Float16 /* ny */
Float16 /* nz */
Normal vector for a point.
3.4. TextureUV
TextureUV1 ::=
Float32 /* u */
Float32 /* v */
Location in texture map for a point.
3.5. Rotation
Rot1 ::=
Float16 /* i */
Float16 /* j */
Float16 /* k */
The non-real parts of a normalized rotation quaternion. The real
part can be computed based on how it is normalized.
Rot2 ::=
Float16 /* s.i */
Float16 /* s.j */
Float16 /* s.k */
Float16 /* e.i */
Float16 /* e.j */
Float16 /* e.k */
Jennings & Logan Expires 27 April 2023 [Page 5]
Internet-Draft Game State over RTP October 2022
Rot2 defines the s, the current rotation, and e, an estimated value
of the rotation in one second. The rotation e is chosen such that
rotation of the object will follow a rotation along the great circle
path from s to e with an constant angular rotation rate such that it
would reach e in 1 second.
This representation of rate of change of the rotation allows the
receiver to use an algorithm such as SLERP
[https://en.wikipedia.org/wiki/Slerp (https://en.wikipedia.org/wiki/
Slerp)] to estimate the current rotation for the object for short
periods of time into the future. The representation cannot represent
something that is rotating faster than one revolution ever 2 seconds.
Open Issue: Would there be a better way to represent angular
velocity?
3.6. Child Transform
Transform1 ::=
Float16 /* tx */
Float16 /* ty */
Float16 /* tz */
Defines a linear translation of a child object from a base object.
3.7. Texture URL
TextureUrl1 ::= String
URL of image with texture map. JPEG images SHOULD be supported.
3.8. Mesh URL
MeshUrl1 ::= String
URL of external mesh.
Open Issue: Any mandatory to implements mesh formats?
3.9. Texture Stream
In some cases it is desirable to provide a separate RTP video stream
which might have the texture used be the frame from the video stream
with the corresponding time to the game object that uses the texture
map. To do this there needs to be a way to identity the other RTP
video stream. One way to do that is use the "Payload Type" value
used for the RTP packets for that video stream.
Jennings & Logan Expires 27 April 2023 [Page 6]
Internet-Draft Game State over RTP October 2022
TextureRtpPT1 ::= UInt8 /* pt */
RTP "Payload Type" value of RTP video stream to use as a texture map.
Open Issue: Is there a better way to identity video stream?
3.10. Timestamp
Time1 ::= UInt16 /* time in ms */
Lower 16 bits of number in milliseconds since 00:00:00 UTC on 1
January 1970, not counting leap seconds. The assumption is these
timestamps are accurate to about the level of an NTP synchronized
clock. In C++20 this can be found with:
duration_cast<milliseconds>(
system_clock::now().time_since_epoch()
).count() % 65536
4. Objects
All objects must start with a unique tag that defines the object
type, a VarUInt length, a VarUInt objectID, and the data for object.
Applications can reserve tags for objects in the registry defined in
the IANA section. Objects can be made extensible by adding a section
that contains optional tag, length, value tuples.
4.1. Common Objects
4.1.1. Generic Game Object
It is common to need to describe the location, rotation, scale, and
parent for objects in a scene.
Object1 ::= tagObject1 Length ObjectID Time1
Loc1
Rot1
Scale1
Boolean /* Active */
( tagParent1 Length ObjectID )? /* Optional Parent */
Object1 contains a simple 3D location, rotation, scale and an
optional ID of a parent object.
Jennings & Logan Expires 27 April 2023 [Page 7]
Internet-Draft Game State over RTP October 2022
Object2 ::= tagObject2 Length ObjectID Time1
Loc2
Rot2
Scale2
Boolean /* Active */
( tagParent1 Length ObjectID )? /* Optional Parent */
Object2 has the same information but with the ability to scale
differently in each dimension and derivatives that describe how all
the parameters change over time.
4.1.2. Player Head
Head1 ::= tagHead1 Length ObjectID Time1
Loc2 Rot2
( tagHeadIpd1 Length Float16 /* IPD */ )?
Defines location and rotation of head with optional interpupillary
distance (IPD).
4.1.3. Mesh
Object with the following variable length arrays:
Mesh1 ::= tagMesh1 Length ObjectID
( TextureUrl1 | TextureRtpPT1 )
VarUInt /* num Vertexes */
Loc1+ /* vertexes */
VarUInt /* numNormals */
Norm1* /* normals */
VarUInt /* numTextureCoord */
TextureUV1* /* textureCoord */
VarUInt /* numTrianglesIndex */
VarUInt+ /* trianglesIndex */
The vertex is an array of at least 3 locations that defines the
vertex of a triangle mesh. The normals array can either be empty or
the same size as the vertex and defines the normal for each vertex.
The uv array must be empty or the same size as vertex array and have
the u,v coordinate in the texture map for the vertex.
The texture can be defined by a URL that may refer to some local
resource or a resource retrieved over the network. Alternatively,
the texture can reference a local RTP video stream, in which case the
most recently received frame of video is used as the texture and
texture updates with new frames of video.
Jennings & Logan Expires 27 April 2023 [Page 8]
Internet-Draft Game State over RTP October 2022
The triangles array can be of a different size from the vertex array.
Each entry defines one triangle in the mesh and contains the index of
the three vertexes in the vertexes array. Vertexes MUST be in
counter clockwise order.
An important limitation to note is that objects cannot span RTP
packets so the Mesh needs to be small enough that it size is less
that the MTU. A typical limit might be as low as 50 triangles.
4.1.4. External Mesh
The Mesh2 object allows a mesh to be loaded from an external URL and
then moved, rotated, and scaled. An optional texture map may be
used.
Mesh2::= tagMesh2 Length ObjectID
Loc2 Rot2 Scale2
MeshUrl1
( TextureUrl1 | TextureRtpPT1 )? /* Optional Texture Map */
( tagParent1 Length ObjectID )? /* Optional Parent */
4.1.5. Player Hand
Hand1 ::= tagHand1 Length ObjectID Time1
Boolean /* left */
Loc2 Rot2
The Hand1 identifies a location and rotation of a hand. The left is
true for the left hand, false for a right hand.
Jennings & Logan Expires 27 April 2023 [Page 9]
Internet-Draft Game State over RTP October 2022
Hand2 ::= tagHand2 Length ObjectID Time1
Boolean /* left */
Loc2 Rot2
Transform1 /* wrist */
Transform1 /* thumbTip */
Transform1 /* thumbIP */
Transform1 /* thumbMCP */
Transform1 /* thumbCMC */
Transform1 /* indexTip */
Transform1 /* indexDIP */
Transform1 /* indexPIP */
Transform1 /* indexMCP */
Transform1 /* indexCMC */
Transform1 /* middleTip */
Transform1 /* middleDIP */
Transform1 /* middlePIP */
Transform1 /* middleMCP */
Transform1 /* middleCMC */
Transform1 /* ringTip */
Transform1 /* ringDIP */
Transform1 /* ringPIP */
Transform1 /* ringMCP */
Transform1 /* ringCMC */
Transform1 /* pinkyTip */
Transform1 /* pinkyDIP */
Transform1 /* pinkyPIP */
Transform1 /* pinkyMCP */
Transform1 /* pinkyCMC */
Hand2 represents a wired skeletal hand. The boolean is true for the
left hand. The location should represent the location of the palm
and rotation is from the palm facing the x axis.
The transform points represent the relative location to the main
joints in the fingers from the hand location. The location of the
wrist is followed by finger joints. The fingers are ordered by
thumb, index, middle, ring, then pinky. The joints are ordered by
tip of finger (TIP), distal interphalangeal joint (DIP), proximal
interphalangeal joint (PIP), metacarpophalangeal joint (MCP), then
carpometacarpal joint (CMC). Note the thumb has no middle phalange
so the PIP and DIP joint just become the interphalangeal joint (IP).
Note: The Microsoft documentation calls the MCP "knuckle" for the
fingers and PIP "middle joint" and the CMC is called "Metacarpal",
which is very confusing since this is not the metacarpophalangeal
joint. The MCP for the thumb gets called "proximal", not knuckle,
and the IP is "middle".
Jennings & Logan Expires 27 April 2023 [Page 10]
Internet-Draft Game State over RTP October 2022
Names of the joints are explained in [https://en.wikipedia.org/wiki/
Interphalangeal_joints_of_the_hand (https://en.wikipedia.org/wiki/
Interphalangeal_joints_of_the_hand)]
This is about 175 bytes, so at a 5Hz update rate this will be around
10 Kbps. [TODO: Check.]
4.2. Common Controls
4.2.1. Three Degree of Freedom (3DOF) Controller
Defines the location and orientation of a 6DOF controller.
ThreeDOF1 ::= tag3DOF1 Length ObjectID Time1
bool isLeft
Rot2
For systems with two controls, the isLeft indicates if it is the left
or right controller.
4.2.2. Six Degree of Freedom (6DOF) Controller
Defines the location and orientation of a 6DOF controller.
SixDOF1 ::= tag6DOF1 Length ObjectID Time1
bool isLeft
Loc2 Rot2
( tag6DOFpointer1 Loc1 /* interrsection point */ )?
For systems with two controls, the isLeft indicates if it is the left
or right controller. The intersection point can be used for or
systems where the controller can indicate the location that a
controller is pointing at.
4.2.3. Game Controller
Transfers the current values and time the value last changed for a
left and right analog stick with x and y values ranging from -1.0 to
1.0 and the state of several buttons along with the time a button
state last changed.
Jennings & Logan Expires 27 April 2023 [Page 11]
Internet-Draft Game State over RTP October 2022
+=====+==================+
| Bit | Button |
+=====+==================+
| 1 | Menu |
+-----+------------------+
| 2 | View |
+-----+------------------+
| 3 | A |
+-----+------------------+
| 4 | B |
+-----+------------------+
| 5 | X |
+-----+------------------+
| 6 | Y |
+-----+------------------+
| 7 | DPadUp |
+-----+------------------+
| 8 | DPadDown |
+-----+------------------+
| 9 | DPadLeft |
+-----+------------------+
| 10 | DPadRight |
+-----+------------------+
| 11 | LeftShoulder |
+-----+------------------+
| 12 | RightShoulder |
+-----+------------------+
| 13 | LeftStickButton |
+-----+------------------+
| 14 | RightStickButton |
+-----+------------------+
| 15 | LeftTrigger |
+-----+------------------+
| 16 | RightTrigger |
+-----+------------------+
| 17 | LeftShoulder |
+-----+------------------+
| 18 | RightShoulder |
+-----+------------------+
| 19 | Z |
+-----+------------------+
| 20 | Pause |
+-----+------------------+
Table 1
Jennings & Logan Expires 27 April 2023 [Page 12]
Internet-Draft Game State over RTP October 2022
Sony style buttons for Square, Triangle ,Cross, Circle map to X,Y,A,B
and R1,R2,L1,L2 map to RightShoulder, RightTrigger, LeftShoulder,
LeftTrigger.
GameControl1 ::= tagGameControl1 Length ObjectID Time1
VarInt Time1 /* buttons state and last time state changed */
Float16 Float16 /* left stick x,y */
Float16 Float16 /* right stick x,y */
5. Encoding
Each RTP payload will contain one or more objects. An object cannot
be split across two RTP packets. The general design is that if the
decoder has not been coded to understand a given object type, the
decode can skip over the object to the next object but will not be
able to provide any information and the internal format of the data.
The objects are defined such that they always start with a tag that
indicates the type followed by a length of the object (so it can be
skipped). Any optional or variable parts of the object also use tags
so that the decoder can always be implemented as a LL(1) parser.
In general, network byte order encoding is used on the wire.
A length field is encoded to represent the number of bytes following
the length field and does not include the size of the length field or
information before it.
5.1. Tag
Constant values of tags can be found in the IANA section. They are
encoded as VarUInt.
5.2. Float
Float16, Float32, and Float64 are encoded as IEEE 754 half, single,
and double precisions respectively.
The half precision are often useful for things where only a few
significant digits are needed such as normals. The internal
representation of them will often be single precession (four bytes)
in memory but they can be reduced to two bytes when encoded on the
wire.
Note there is an example decode for a single precision float at
[https://datatracker.ietf.org/doc/html/rfc7049#appendix-D
(https://datatracker.ietf.org/doc/html/rfc7049#appendix-D)]
Jennings & Logan Expires 27 April 2023 [Page 13]
Internet-Draft Game State over RTP October 2022
5.3. Boolean
Encoded as byte with 0 for false or 1 for true.
5.4. Integer
UInt8, Int8, UInt16, Int16, UInt32, Int32, UInt16, Int64 encoded as
1, 2, 4, or 8 bytes.
VarInt is encoded as:
* Top bits of first byte is 0, then 7 bit signed integer (-64 to 63)
* Top bits of first byte is 10, then 6+8 bit signed integer (-8192
to 8191)
* Top bits of first byte is 110, then 5+16 bit signed integer
(1,048,576 to 1,048,575)
* Top bits of first byte is 1110,0001 then next 4 bytes 32 bit
signed integer
* Top bits of first byte is 1110,0010 then next 8 bytes 64 bit
signed integer
VarUInt is encoded as:
* Top bits of first byte is 0, then 7 bit unsigned integer
* Top bits of first byte is 10, then 6+8 bit unsigned integer
* Top bits of first byte is 110, then 5+16 bit unsigned integer
* Top bits of first byte is 1110,0001 then next 4 bytes 32 bit
unsigned integer
* Top bits of first byte is 1110,0010 then next 8 bytes 64 bit
unsigned integer
5.5. String
Strings are encoded as a VarUInt length in bytes (not characters)
followed by a UTF-8 representation of the string.
5.6. Blob
Blobs are encoded as a VarUInt length in bytes followed by the binary
data that goes in the blob.
Jennings & Logan Expires 27 April 2023 [Page 14]
Internet-Draft Game State over RTP October 2022
6. Full Intra Request
RTP supports a Full Intra Request (FIR) Feedback Control feedback
messages. When an RTP sender receives a FIR, it SHOULD send a copy
of all the relevant game state.
7. RTP and SDP
The media type is application/gamestate and defined in
Section Section 8.1. There are no optional or required parameters.
The RTP marker bit is not used. The RTP clock MUST be 90 kHz.
An example m= and a= line in SDP could look like:
m=application 49170 RTP/AVP 98
a=rtpmap:98 gamestate/90000
Multiple Objects as defined in this specification can be concatenated
into one RTP payload.
No storage format is defined.
8. IANA
8.1. Media Type Registration
This specification defines a new media type using the template
defined in [RFC4288] and following the guidance provided in
[RFC4855].
Jennings & Logan Expires 27 April 2023 [Page 15]
Internet-Draft Game State over RTP October 2022
Type name: application
Subtype name: gamestate
Required parameters: None.
Optional parameters: None.
Encoding considerations: This media is sent as binary data.
Security considerations:
(IANA point to security section of this RFC)
Interoperability considerations: None.
Published specification:
(IANA to replace with RFC number for this specification)
Applications that use this media type: N/A
Fragment identifier considerations: N/A
Additional information: None
Person & email address to contact for further information:
Cullen Jennings <fluffy@iii.ca>
Intended usage: COMMON
Restrictions on usage: N/A
Author: Cullen Jennings <email:fluffy@iii.ca>
Change controller: IESG
Provisional registration? (standards tree only): No
8.2. Game State Tag Registry
The specification defines a new IANA registry called "Game State Type
Registry" for tag values. All values MUST be greater than zero.
Values 1-127 are assigned by "IETF Review" as defined in [RFC8126],
and should only be used when size is critical, the object is small,
and will be used frequently.
Values 127-16383 are assigned by "Specification Required" as defined
in [RFC8126].
Values 16384 to 2,097,151 are "First Come First Served" as defined in
[RFC8126].
Initial assignments are:
Jennings & Logan Expires 27 April 2023 [Page 16]
Internet-Draft Game State over RTP October 2022
+=================+=======+
| TagName | Value |
+=================+=======+
| tagInvalid | 0 |
+-----------------+-------+
| tagHead1 | 1 |
+-----------------+-------+
| tagHand1 | 2 |
+-----------------+-------+
| tagObject1 | 3 |
+-----------------+-------+
| tagParent1 | 4 |
+-----------------+-------+
| tagMesh1 | 128 |
+-----------------+-------+
| tagHand2 | 129 |
+-----------------+-------+
| tagHeadIPD1 | 130 |
+-----------------+-------+
| tagObject2 | 131 |
+-----------------+-------+
| tagMesh2 | 132 |
+-----------------+-------+
| tagGameControl1 | 133 |
+-----------------+-------+
| tag3DOF1 | 134 |
+-----------------+-------+
| tag6DOF1 | 135 |
+-----------------+-------+
| tag6DOFpointer1 | 136 |
+-----------------+-------+
Table 2
9. Security
Like most things in RTP, the data can be personal identifying
information. For example, the Hand2 type of data when generated by
tracking a persons hand might identify that user.
10. Normative References
[RFC3550] Schulzrinne, H., Casner, S., Frederick, R., and V.
Jacobson, "RTP: A Transport Protocol for Real-Time
Applications", STD 64, RFC 3550, DOI 10.17487/RFC3550,
July 2003, <https://www.rfc-editor.org/info/rfc3550>.
Jennings & Logan Expires 27 April 2023 [Page 17]
Internet-Draft Game State over RTP October 2022
[RFC4288] Freed, N. and J. Klensin, "Media Type Specifications and
Registration Procedures", RFC 4288, DOI 10.17487/RFC4288,
December 2005, <https://www.rfc-editor.org/info/rfc4288>.
11. Informative References
[RFC4855] Casner, S., "Media Type Registration of RTP Payload
Formats", RFC 4855, DOI 10.17487/RFC4855, February 2007,
<https://www.rfc-editor.org/info/rfc4855>.
Appendix A. Acknowledgments
Thanks to Paul Jones for comments and writing an implementation.
Appendix B. Implementations
A C++ open source implementation is available at:
https://github.com/cisco/gse
Appendix C. Test Vectors
C.1. Head Location
Head Location type 1 with head at location 1.1, 0.2, 30.0, no
rotation (so quaternion 0, 0, 0, 1) and not rotating, an objectID of
4, a time of 5 ms after epoch and an IPD of 0.056.
Jennings & Logan Expires 27 April 2023 [Page 18]
Internet-Draft Game State over RTP October 2022
+=========+=========+=======+============+
| Field | Type | Value | Hex |
+=========+=========+=======+============+
| head1 | Tag | 1 | 0x01 |
+---------+---------+-------+------------+
| len | VarInt | 33 | 0x21 |
+---------+---------+-------+------------+
| objID | VarInt | 0 | 0x00 |
+---------+---------+-------+------------+
| time | UInt16 | 5 | 0x0500 |
+---------+---------+-------+------------+
| loc.x | Float32 | 1.1 | 0x3F8CCCCD |
+---------+---------+-------+------------+
| loc.y | Float32 | 0.2 | 0x3E4CCCCD |
+---------+---------+-------+------------+
| loc.z | Float32 | 30.0 | 0x41F00000 |
+---------+---------+-------+------------+
| loc.vx | Float16 | 0.0 | 0x0000 |
+---------+---------+-------+------------+
| loc.vy | Float16 | 0.0 | 0x0000 |
+---------+---------+-------+------------+
| loc.vz | Float16 | 0.0 | 0x0000 |
+---------+---------+-------+------------+
| rot.s.i | Float16 | 0.0 | 0x0000 |
+---------+---------+-------+------------+
| rot.s.j | Float16 | 0.0 | 0x0000 |
+---------+---------+-------+------------+
| rot.s.k | Float16 | 0.0 | 0x0000 |
+---------+---------+-------+------------+
| rot.e.i | Float16 | 0.0 | 0x0000 |
+---------+---------+-------+------------+
| rot.e.j | Float16 | 0.0 | 0x0000 |
+---------+---------+-------+------------+
| rot.e.k | Float16 | 0.0 | 0x0000 |
+---------+---------+-------+------------+
Table 3
Appendix D. Encode API
API that take a high level representation of each object where types
are all float or int and returns a memory buffer.
Appendix E. Decode API
API that takes binary data and set up objects and updates the objects
and returns object ids that were updated.
Jennings & Logan Expires 27 April 2023 [Page 19]
Internet-Draft Game State over RTP October 2022
For each object, an API to get the predicted values at a given time.
Appendix F. EBNF
Head1 ::= tagHead1 Length ObjectID Time1 Loc2 Rot2
( tagHeadIpd1 Length Float16 /* IPD */ )?
Mesh1 ::= tagMesh1 Length ObjectID
( TextureUrl1 | TextureRtpPT1 )
VarUInt /* num Vertexes */
Loc1+ /* vertexes */
VarUInt /* numNormals */
Norm1* /* normals */
VarUInt /* numTextureCoord */
TextureUV1* /* textureCoord */
VarUInt /* numTrianglesIndex */
VarUInt+ /* trianglesIndex */
Mesh2::= tagMesh2 Length ObjectID
Loc2 Rot2 Scale2
MeshUrl1
( TextureUrl1 | TextureRtpPT1 )? /* Optional Texture Map */
( tagParent1 Length ObjectID )? /* Optional Parent */
Object1 ::= tagObject1 Length ObjectID Time1
Loc1
Rot1
Scale1
Boolean /* Active */
( tagParent1 Length ObjectID )? /* Optional Parent */
Object2 ::= tagObject2 Length ObjectID Time1
Loc2
Rot2
Scale2
Boolean /* Active */
( tagParent1 Length ObjectID )? /* Optional Parent */
Hand1 ::= tagHand1 Length ObjectID Time1
Boolean /* left */
Loc2 Rot2
Hand2 ::= tagHand2 Length ObjectID Time1
Boolean /* left */
Loc2 Rot2
Transform1 /* wrist */
Transform1 /* thumbTip */
Transform1 /* thumbIP */
Jennings & Logan Expires 27 April 2023 [Page 20]
Internet-Draft Game State over RTP October 2022
Transform1 /* thumbMCP */
Transform1 /* thumbCMC */
Transform1 /* indexTip */
Transform1 /* indexDIP */
Transform1 /* indexPIP */
Transform1 /* indexMCP */
Transform1 /* indexCMC */
Transform1 /* middleTip */
Transform1 /* middleDIP */
Transform1 /* middlePIP */
Transform1 /* middleMCP */
Transform1 /* middleCMC */
Transform1 /* ringTip */
Transform1 /* ringDIP */
Transform1 /* ringPIP */
Transform1 /* ringMCP */
Transform1 /* ringCMC */
Transform1 /* pinkyTip */
Transform1 /* pinkyDIP */
Transform1 /* pinkyPIP */
Transform1 /* pinkyMCP */
Transform1 /* pinkyCMC */
ThreeDOF1 ::= tag3DOF1 Length ObjectID Time1
bool isLeft
Rot2
SixDOF1 ::= tag6DOF1 Length ObjectID Time1
bool isLeft
Loc2 Rot2
( tag6DOFpointer1 Loc1 /* intersection point */ )?
GameControl1 ::= tagGameControl1 Length ObjectID Time1
VarInt Time1 /* buttons state and last time state changed */
Float16 Float16 /* left stick x,y */
Float16 Float16 /* right stick x,y */
Tag ::= VarUInt
tagInvalid ::= #x00
tagHead1 ::= #x01
tagHand1 ::= #x02
tagObject1 ::= #x03
tagParent1 ::= #x04
tagMesh1 ::= #x80 #x00
tagHand2 ::= #x80 #x01
tagHeadIpd1 ::= #x80 #x02
tagObject2 ::= #x80 #x03
Jennings & Logan Expires 27 April 2023 [Page 21]
Internet-Draft Game State over RTP October 2022
tagMesh2 ::= #x80 #x04
tagGameControl1 ::= #x80 #x05
tag3DOF1 ::= #x80 #x06
tag6DOF1 ::= #x80 #x07
tag6DOFpointer1 ::= #x80 #x08
ObjectID ::= VarUInt
Length ::= VarUInt
Loc1 ::=
Float32 /* x */
Float32 /* y */
Float32 /* z */
Loc2 ::=
Float32 /* x */
Float32 /* y */
Float32 /* z */
Float16 /* vx */
Float16 /* vy */
Float16 /* vz */
Scale1 ::=
Float16 /* all dimensions */
Scale2 ::=
Float32 /* x */
Float32 /* y */
Float32 /* z */
Float16 /* vx */
Float16 /* vy */
Float16 /* vz */
Norm1 ::=
Float16 /* x */
Float16 /* y */
Float16 /* z */
TextureUV1 ::=
Float32 /* u */
Float32 /* v */
Rot1 ::=
Float16 /* i */
Float16 /* j */
Float16 /* k */
/* w computed based on quaternion is normalized */
Jennings & Logan Expires 27 April 2023 [Page 22]
Internet-Draft Game State over RTP October 2022
Rot2 ::=
Float16 /* s.i */
Float16 /* s.j */
Float16 /* s.k */
Float16 /* e.i */
Float16 /* e.j */
Float16 /* e.k */
Transform1 ::=
Float16 /* tx */
Float16 /* ty */
Float16 /* tz */
TextureUrl1 ::= String
MeshUrl1 ::= String
TextureRtpPT1 ::= UInt8 /* pt */
Time1 ::= UInt16 /* time in ms */
Tag ::= VarUInt
Boolean ::= #x00 | #x01
String ::= VarUInt byte*
Blob ::= VarUInt byte*
Float16 ::= byte byte
Float32 ::= byte byte byte byte
Float64 ::= byte byte byte byte byte byte byte byte
Int8 ::= byte
Int16 ::= byte byte
Int32 ::= byte byte byte byte
Int64 ::= byte byte byte byte byte byte byte byte
UInt8 ::= byte
UInt16 ::= byte byte
UInt32 ::= byte byte byte byte
UInt64 ::= byte byte byte byte byte byte byte byte
VarUInt ::=
( [#x0-#x7F] ) |
( [#x80-#x87] byte ) |
( [#x88-#x8B] byte byte ) |
( #xE1 UInt32 ) |
Jennings & Logan Expires 27 April 2023 [Page 23]
Internet-Draft Game State over RTP October 2022
( #xE2 UInt64 )
VarInt ::=
( [#x0-#x7F] ) |
( [#x80-#x87] byte ) |
( [#x88-#x8B] byte byte ) |
( #xE1 Int32 ) |
( #xE2 Int64 )
byte ::= [#x00-#xFF]
Authors' Addresses
Cullen Jennings
cisco
Canada
Email: fluffy@iii.ca
Rich Logan
Cisco
United Kingdom
Email: rilogan@cisco.com
Jennings & Logan Expires 27 April 2023 [Page 24]
