Skip to main content

The 'haptics' Top-level Media Type
draft-ietf-mediaman-haptics-03

The information below is for an old version of the document.
Document Type
This is an older version of an Internet-Draft whose latest revision state is "Active".
Authors Yeshwant K Muthusamy , Chris Ullrich
Last updated 2023-07-18 (Latest revision 2023-02-08)
Replaces draft-muthusamy-dispatch-haptics
RFC stream Internet Engineering Task Force (IETF)
Formats
Reviews
Additional resources Mailing list discussion
Stream WG state In WG Last Call
Associated WG milestone
Jul 2023
Registration of the "haptics" top-level type to the IESG for approval (Proposed Standard)
Document shepherd (None)
IESG IESG state AD is watching
Consensus boilerplate Unknown
Telechat date (None)
Responsible AD Murray Kucherawy
Send notices to (None)
draft-ietf-mediaman-haptics-03
MEDIAMAN                                                 Y. K. Muthusamy
Internet-Draft                                                          
Intended status: Standards Track                              C. Ullrich
Expires: 12 August 2023                                  8 February 2023

                   The 'haptics' Top-level Media Type
                     draft-ietf-mediaman-haptics-03

Abstract

   This memo serves to register and document the 'haptics' top-level
   media type, under which subtypes for representation formats for
   haptics may be registered.  This document also serves as a
   registration application for a set of intended subtypes, which are
   representative of some existing subtypes already in use.

Status of This Memo

   This Internet-Draft is submitted in full conformance with the
   provisions of BCP 78 and BCP 79.

   Internet-Drafts are working documents of the Internet Engineering
   Task Force (IETF).  Note that other groups may also distribute
   working documents as Internet-Drafts.  The list of current Internet-
   Drafts is at https://datatracker.ietf.org/drafts/current/.

   Internet-Drafts are draft documents valid for a maximum of six months
   and may be updated, replaced, or obsoleted by other documents at any
   time.  It is inappropriate to use Internet-Drafts as reference
   material or to cite them other than as "work in progress."

   This Internet-Draft will expire on 12 August 2023.

Copyright Notice

   Copyright (c) 2023 IETF Trust and the persons identified as the
   document authors.  All rights reserved.

   This document is subject to BCP 78 and the IETF Trust's Legal
   Provisions Relating to IETF Documents (https://trustee.ietf.org/
   license-info) in effect on the date of publication of this document.
   Please review these documents carefully, as they describe your rights
   and restrictions with respect to this document.  Code Components
   extracted from this document must include Revised BSD License text as
   described in Section 4.e of the Trust Legal Provisions and are
   provided without warranty as described in the Revised BSD License.

Muthusamy & Ullrich      Expires 12 August 2023                 [Page 1]
RFC XXXX           The 'haptics' Top-level Media Type      February 2023

Table of Contents

   1.  Introduction  . . . . . . . . . . . . . . . . . . . . . . . .   2
     1.1.  Terminology . . . . . . . . . . . . . . . . . . . . . . .   3
   2.  Background and Justification  . . . . . . . . . . . . . . . .   3
     2.1.  MPEG ISOBMFF  . . . . . . . . . . . . . . . . . . . . . .   3
     2.2.  Haptic Sub-modalities . . . . . . . . . . . . . . . . . .   4
     2.3.  Another Human Sense . . . . . . . . . . . . . . . . . . .   4
     2.4.  Commercial Uptake . . . . . . . . . . . . . . . . . . . .   4
     2.5.  Haptic Data Formats in Use  . . . . . . . . . . . . . . .   5
     2.6.  Haptic Subtypes (envisioned standards)  . . . . . . . . .   6
     2.7.  'application' top-level type not suitable . . . . . . . .   6
   3.  Security Considerations . . . . . . . . . . . . . . . . . . .   7
   4.  IANA Considerations . . . . . . . . . . . . . . . . . . . . .   8
     4.1.  Definition and Encoding . . . . . . . . . . . . . . . . .   8
     4.2.  Registration Procedure  . . . . . . . . . . . . . . . . .   8
     4.3.  Subtype Registrations . . . . . . . . . . . . . . . . . .   8
       4.3.1.  IVS Haptics Type  . . . . . . . . . . . . . . . . . .   8
       4.3.2.  HAPT Haptics Type . . . . . . . . . . . . . . . . . .   9
   5.  Normative References  . . . . . . . . . . . . . . . . . . . .  10
   6.  Informative References  . . . . . . . . . . . . . . . . . . .  11
   Authors' Addresses  . . . . . . . . . . . . . . . . . . . . . . .  11

1.  Introduction

   The term 'haptics' refers to the generation of touch-related
   sensations in a device or interface.  Haptics is widely used in
   consumer devices in order to provide touch-based feedback to users.
   The most common use of haptics is in mobile devices, where it is used
   to provide feedback to users interacting with the touchscreen, e.g.,
   typing on a virtual keyboard.  Haptic technologies are unlike audio
   and visual enabling technologies in the sense that they require some
   form of actuation in order to create a tactile sensation.  For mobile
   phones and game controllers, these actuators are typically small
   vibrating motors.  For large touchscreens in vehicles, these
   actuators can be specialized piezoelectric materials.  Haptic
   capabilities are found in nearly every modern smartphone and game and
   virtual reality controller, making these devices an ideal target for
   enhanced media experiences.

   Internet Media Types [RFC6838] are used to label content carried over
   Internet protocols.  This document defines a new top-level type
   'haptics' according to Section 4.2.7 of [RFC6838].  This top-level
   type indicates that the content specifies haptic data.  Under this
   top-level type, different representation formats of haptics may be
   registered.

Muthusamy & Ullrich      Expires 12 August 2023                 [Page 2]
RFC XXXX           The 'haptics' Top-level Media Type      February 2023

1.1.  Terminology

   The keywords MUST, MUST NOT, REQUIRED, SHALL, SHALL NOT,
   SHOULD,SHOULD NOT, RECOMMENDED, NOT RECOMMENDED, MAY, and OPTIONAL in
   this document are to be interpreted as described in BCP 14 [RFC2119]
   [RFC8174] when, and only when, they appear in all capitals, as shown
   here.

2.  Background and Justification

   Haptic signals provide an additional layer of entertainment and
   sensory immersion for the user.  Haptic tracks, in separate files,
   can be combined with audio/video files and played back in sync to
   provide an overall immersive media experience (audio, visual,
   tactile) for the user.  More recently, haptic tracks embedded in
   standard file formats such as ISOBMFF (ISO Base Media File Format),
   enable playback of the haptic signals over one or more actuators,
   simultaneously with audio and video playback.  Haptic signals are
   also part of media streams that use RTP, such as those for streaming
   games, XR, and wearables.

2.1.  MPEG ISOBMFF

   Historically, there has not been a registration of formats for
   haptics.  However, haptics has been proposed as a first-order media
   type (at the same level as audio and video) in ISOBMFF.  This
   proposal was made to the MPEG Systems File Format sub-group in April
   2020.  The proposal was accepted, has since progressed to
   International Standard, and was published in January 2022
   [ISOBMFF-IS].  Haptics is officially part of the ISO/IEC 14496-12
   (ISOBMFF) standard, 7th Edition.  Given this development, a strong
   case can be made for haptics to be added to the list of top-level
   media types recognized by the IETF.

   We envision the following designations for haptics in mp4 files, once
   the top-level type 'haptics' is registered:

   1.  'haptics/mp4' - mp4 files with just haptic tracks and no audio or
       video in them (e.g., streaming games, haptics files for haptic
       vests, belts, gloves, etc.)

   2.  'video/mp4' - mp4 files with video, audio, and haptics (to ensure
       consistency with existing mp4 files with video content)

   3.  'audio/mp4' - mp4 files with audio and haptics (to ensure
       consistency with existing mp4 files with audio content without
       any video)

Muthusamy & Ullrich      Expires 12 August 2023                 [Page 3]
RFC XXXX           The 'haptics' Top-level Media Type      February 2023

2.2.  Haptic Sub-modalities

   There are multiple sub-modalities of haptics:

   *  Vibrotactile (touch, vibration)

   *  Kinesthetic (force feedback)

   *  Surface (surface friction)

   *  Spatial, non-contact (ultrasound)

   *  Thermal (temperature)

   Therefore, designating 'haptics' as a top-level media type would
   enable the definition of data formats pertaining to these sub-
   modalities in a more streamlined manner.  This would not be possible
   if 'haptics' were to be placed under other top-level types like
   'audio', 'video', or 'application'.

2.3.  Another Human Sense

   The top-level media type 'audio' pertains to the human sense of
   hearing, the top-level media type 'video' pertains to the human sense
   of seeing, so it only makes sense for the (equally important) human
   sense of touch to be represented by another top-level media type
   'haptics'.  Placing 'haptics' under 'audio' or 'video' is not
   reflective of the kinds of files or use cases that would need haptics
   but have nothing whatsoever to do with audio or video.

2.4.  Commercial Uptake

   Haptics is rapidly becoming a standard feature of consumer electronic
   devices.  For example:

   *  iPhone (206+ million units sold in 2020): native support for
      haptic encoded data

   *  Android (1.38+ billion units sold in 2020): API support of haptic
      buffers

   *  W3C (HTML vibration API [W3C-Vibration]): Optionally supported in
      mobile web browsers.  W3C has also defined vibration extensions
      for gamepads [W3C-Gamepad]

   *  Game consoles (39+ million units sold in 2019): MS Xbox, Sony
      PlayStation, Nintendo Switch, etc.

Muthusamy & Ullrich      Expires 12 August 2023                 [Page 4]
RFC XXXX           The 'haptics' Top-level Media Type      February 2023

   *  XR devices (9+ million units sold in 2019): OpenXR haptic API

   Haptic media is expected to be commonly exchanged between these
   devices.  Since they represent the majority of CE devices, a strong
   case can be made for 'haptics' as a top-level media type.

2.5.  Haptic Data Formats in Use

   There are multiple instances of existing haptic data formats that
   would live as sub-types under the proposed 'haptics' top-level media
   type.  While these subtypes have *not* been registered with IANA or
   standardized (yet), the prevalence of these haptic data formats in a
   large number of devices around the world, pre-dating the
   standardization of haptic tracks in ISOBMFF, provides a compelling
   argument for 'haptics' to be designated as a top-level media type:

   *  'ahap': The AHAP haptic data format [AHAP] is currently the
      standard encoding on all iOS devices + iOS connected game
      peripherals.  The format has seen usage and adoption beyond Apple
      devices as well, with decoders available for Android and other XR
      systems.

   *  'ogg': Google has introduced a proprietary extension to the OGG
      format in the latest version of Android 11.  This encoding enables
      haptic media to be stored in OGG files.

   *  'ivs': The IVS haptic data format is currently a vendor-specific
      format that is in use:

      -  In mobile phones from LG Electronics (specifically, the models
         V30, V40, and the newest V50) that are sold worldwide

      -  In gaming phones from ASUS (specifically, models ROG, ROG Phone
         II, ROG Phone 3) that are sold worldwide

   *  'hapt': The HAPT haptic data format is currently a vendor-specific
      format that is in use:

      -  In mobile haptic advertising (for W3C devices)

      -  The following Japanese game developers use the HAPT format as
         part of Immersion's TouchSense SDK:

         o  KLAB: https://www.klab.com/en/ (https://www.klab.com/en/)

         o  Craft&Meister: http://www.crafts-meister.co.jp/pc/
            company_en.html (http://www.crafts-meister.co.jp/pc/
            company_en.html)

Muthusamy & Ullrich      Expires 12 August 2023                 [Page 5]
RFC XXXX           The 'haptics' Top-level Media Type      February 2023

      -  Tencent is using the TouchSense SDK for their popular social
         media application QQ and live streaming application NOW:
         Immersion-Announces-Tencent-Licenses-TouchSense-Technology-
         Deliver
         (https://www.businesswire.com/news/home/20171026006443/en/
         Immersion-Announces-Tencent-Licenses-TouchSense%C2%AE-
         Technology-Deliver)

   Given the widespread use of these subtypes, it makes sense for
   'haptics' to be a top-level media type.

2.6.  Haptic Subtypes (envisioned standards)

   The MPEG ISOBMFF proposal included an informative annex of known
   haptic coding formats with proposed FourCC codes for them.  These
   codes are not registered yet, but the plan is indeed to standardize
   these haptic coding formats in the near future.  Once standardized,
   they will also live as subtypes under the proposed 'haptics' top-
   level media type:

   *  'hmpg': the MPEG-I haptics binary coding format described in the
      Committee Draft of ISO/IEC 23090-31: Haptics Coding
      [MPEG-Haptics-Coding]

   *  'hjif': the MPEG-I haptics JSON-based interchange format described
      in the Committee Draft of ISO/IEC 23090-31: Haptics Coding
      [MPEG-Haptics-Coding]

   *  'mihs': the MPEG-I haptics streaming format described in the
      Committee Draft of ISO/IEC 23090-31: Haptics Coding
      [MPEG-Haptics-Coding]

   *  'hiee': IEEE P1918.1.1 vibrotactile coding standard [IEEE-P191811]
      being developed under the IEEE Tactile Internet initiative as part
      of the 5G URLL profile

   *  'henm': enumerated effects haptic coding format (based on MIDI)

   *  'havc': audio-to-vibe haptic coding format (automatic audio to
      vibration conversion algorithms)

2.7.  'application' top-level type not suitable

   From the above arguments, it is clear that haptics does not really
   belong under any other media type.  To reiterate, there are three
   main reasons why the 'haptics' media type does not fit under the
   'application' top-level type:

Muthusamy & Ullrich      Expires 12 August 2023                 [Page 6]
RFC XXXX           The 'haptics' Top-level Media Type      February 2023

   *  haptics connects to a sensory system, touch/motion, directly, and
      is more specific than the abstract 'application' type, and

   *  'application' has historically been used for applications, i.e.,
      code, which means it is viewed and treated with great care for
      security.  'haptics' is not code, just as 'audio' and 'video' are
      not code either.

   *  haptics is a property of a media stream, it is not an application
      under any normal definition.  As such, it should be its own type.

3.  Security Considerations

   Haptics are interpreted data structures that represent collections of
   different media rendering instructions intended to be decoded and
   rendered on target device hardware.  Haptic data can be represented
   as collections of signal data and/or descriptive text in XML/JSON or
   similar format.  Signal data is typically not executed by endpoint
   processors and represents minimal security risk.  Descriptive text is
   typically parsed and represented in memory using standard XML data
   structures.  This data is utilized to construct one or more signals
   that are sent to the endpoint device hardware.

   Because of the media/rendering nature of the data path for haptic
   coded data the security profile of haptic data is expected to be
   largely consistent with the security profile of visual and audio
   media data.

   As with any synthesized media data (audio, video, and haptics), there
   is a security risk associated with execution of commands based on the
   descriptive encoding either through its inherent extensibility or
   through the insertion of arbitrary executable data in the descriptive
   format itself.  Indeed, media rendering systems are normally
   implemented with a mix of user and kernel space execution since these
   media must ultimately make their way to a hardware system.  In
   theory, malicious instructions present in descriptive haptic media
   have the potential to execute arbitrary code in kernel space,
   effectively bypassing system permissions structures and/or execution
   sandboxes.

   Haptics, audio, and video media have widespread use and careful
   attention should be paid by operating system and device driver
   implementors to ensure that synthesis and rendering signal paths do
   not provide attack surfaces for malicious payloads.  Ultimately, any
   coded representation of haptic media is insufficient to implicitly
   provide sufficient security and this protection should be enforced by
   the operating system implementor.

Muthusamy & Ullrich      Expires 12 August 2023                 [Page 7]
RFC XXXX           The 'haptics' Top-level Media Type      February 2023

   These security considerations apply to the subtype registrations
   described in this document as well as all future haptics
   registrations.

4.  IANA Considerations

   This specification registers a new top-level type, 'haptics', in the
   standards tree, adds it as an alternative value of "Type Name" in the
   media types registration form [Media-Type-Registration], and
   registers several subtypes for it.

4.1.  Definition and Encoding

   'haptics' as the primary media content type indicates that the
   content identified by it requires a certain haptics subsystem such as
   low-level haptics APIs, which in turn will require hardware
   capabilities such as one or more actuators to render the haptics
   media.  The 'haptics' media type does not provide any specific
   information about the underlying data format and how the haptics
   information should be interpreted -- the subtypes defined within a
   'haptics' tree name the specific haptic formats.  Unrecognized
   subtypes of 'haptics' should be treated as 'application/octet-
   stream'.  Implementations may still pass unrecognized subtypes to the
   haptics subsystem and associated rendering hardware.

4.2.  Registration Procedure

   New haptics formats should be registered using the online form
   [Media-Type-Registration].  [RFC6838] should be consulted on
   registration procedures.  In particular, the haptics specification
   should preferably be freely available.

   Note that new parameter sub-values may be defined in the future.  If
   an implementation does not recognize a sub-value in the comma-
   separated list, it should ignore the sub-value and continue
   processing the other sub-values in the list.

4.3.  Subtype Registrations

   In this section, the initial entries under the top-level 'haptics'
   media type are specified.  They also serve as examples for future
   registrations.

4.3.1.  IVS Haptics Type

   Type name: haptics

   Subtype name: ivs

Muthusamy & Ullrich      Expires 12 August 2023                 [Page 8]
RFC XXXX           The 'haptics' Top-level Media Type      February 2023

   Required parameters: N/A

   Optional parameters: N/A

   Encoding considerations: binary

   Interoperability considerations: The IVS format is a device-
   independent haptic effect coding.  It is designed to enable
   interoperability between distinct physical endpoints.  Not all
   devices may be able to render all effects present in an IVS file.

   Security considerations: See Section 3.

   Published specification: ISO/IEC JTC 1/SC 29/WG 2 N 72 "Encoder Input
   Format for MPEG Haptics" being developed by ISO/IEC JTC1/SC29 WG 2
   [MPEG-Haptics-Encoder].

   Applications that use this media type: All applications that are able
   to create, edit, or display haptic media content.

   Additional information:

   *  File extension(s): Haptic file extensions used for IVS files: .ivs
      (xml) and .ivt (binary)

   *  Macintosh file type code(s): (no code specified)

   *  Macintosh Universal Type Identifier code: N/A

   *  Fragment Identifier: N/A

   *  Deprecated Alias: N/A

   Person & email address to contact for further information: Yeshwant
   Muthusamy(yeshwant@yeshvik.com)

   Change controller: Immersion Corporation

4.3.2.  HAPT Haptics Type

   Type name: haptics

   Subtype name: hapt

   Required parameters: N/A

   Optional parameters: N/A

Muthusamy & Ullrich      Expires 12 August 2023                 [Page 9]
RFC XXXX           The 'haptics' Top-level Media Type      February 2023

   Encoding considerations: binary

   Interoperability considerations: The HAPT format is a device-
   dependent haptic effect coding based on the RIFF coding standard.  It
   is designed to enable efficient coding of a device-specific haptic
   effect.

   Security considerations: See Section 3.

   Published specification: HAPT is a logical extension of the RIFF
   standard [RIFF].

   Applications that use this media type: All applications that are able
   to create, edit, or display haptic media content.

   Additional information:

   *  File extension(s): Haptic file extensions used for HAPT files:
      .hapt

   *  Macintosh file type code(s): (no code specified)

   *  Macintosh Universal Type Identifier code: N/A

   *  Fragment Identifier: N/A

   *  Deprecated Alias: N/A

   Person & email address to contact for further information: Yeshwant
   Muthusamy(yeshwant@yeshvik.com)

   Change controller: Immersion Corporation

5.  Normative References

   [RFC2119]  Bradner, S., "Key words for use in RFCs to Indicate
              Requirement Levels", BCP 14, RFC 2119,
              DOI 10.17487/RFC2119, March 1997,
              <https://www.rfc-editor.org/info/rfc2119>.

   [RFC8174]  Leiba, B., "Ambiguity of Uppercase vs Lowercase in RFC
              2119 Key Words", BCP 14, RFC 8174, DOI 10.17487/RFC8174,
              May 2017, <https://www.rfc-editor.org/info/rfc8174>.

   [RFC6838]  Freed, N., Klensin, J., and T. Hansen, "Media Type
              Specifications and Registration Procedures", BCP 13,
              RFC 6838, DOI 10.17487/RFC6838, January 2013,
              <https://www.rfc-editor.org/info/rfc6838>.

Muthusamy & Ullrich      Expires 12 August 2023                [Page 10]
RFC XXXX           The 'haptics' Top-level Media Type      February 2023

6.  Informative References

   [ISOBMFF-IS]
              "ISO/IEC 14496-12 (7th Edition) Information technology —
              Coding of audio-visual objects — Part 12: ISO base media
              file format", <https://www.iso.org/standard/83102.html>.

   [MPEG-Haptics-Encoder]
              "Encoder Input Format for MPEG Haptics",
              <https://www.mpegstandards.org/standards/
              Explorations/40/>.

   [AHAP]     "Apple Haptic Audio Pattern",
              <https://developer.apple.com/documentation/corehaptics/
              representing_haptic_patterns_in_ahap_files>.

   [MPEG-Haptics-Coding]
              "MPEG-I: Haptics Coding",
              <https://www.mpeg.org/standards/MPEG-I/31/>.

   [W3C-Vibration]
              "W3C Vibration API (Second Edition)",
              <https://www.w3.org/TR/vibration/>.

   [W3C-Gamepad]
              "W3C Gamepad Extensions",
              <https://w3c.github.io/gamepad/extensions.html>.

   [IEEE-P191811]
              "P1918.1.1 - Haptic Codecs for the Tactile Internet",
              <https://standards.ieee.org/project/1918_1_1.html>.

   [Media-Type-Registration]
              "IANA, Application for a Media Type",
              <http://www.iana.org/form/media-types>.

   [RIFF]     "Resource Interchange File Format",
              <https://www.loc.gov/preservation/digital/formats/fdd/
              fdd000025.shtml>.

Authors' Addresses

   Yeshwant K. Muthusamy
   600 Longwood Drive
   Allen,  TX 75013
   United States of America
   Phone: +1 469-854-9836
   Email: yeshwant@yeshvik.com

Muthusamy & Ullrich      Expires 12 August 2023                [Page 11]
RFC XXXX           The 'haptics' Top-level Media Type      February 2023

   Chris Ullrich
   311 Court Ave
   Ventura,  CA 93003
   United States of America
   Phone: +1 805-320-0774
   Email: chrisullrich@gmail.com

Muthusamy & Ullrich      Expires 12 August 2023                [Page 12]