In Network Computing Enablers for Extended Reality
draft-montpetit-coin-xr-02

COIN                                                        M. Montpetit
Internet-Draft                                            Triangle Video
Intended status: Informational                             March 9, 2019
Expires: September 10, 2019

           In Network Computing Enablers for Extended Reality
                       draft-montpetit-coin-xr-02

Abstract

   Augmented Reality (AR) and Virtual Reality (VR), combined as Extended
   Reality or XR, challenge networking technologies and protocols
   because they combine the features of fast information display, image
   processing, computing and forwarding.  This document presents some of
   these challenges and how adding computing in the network could
   respond to them.

Status of This Memo

   This Internet-Draft is submitted in full conformance with the
   provisions of BCP 78 and BCP 79.

   Internet-Drafts are working documents of the Internet Engineering
   Task Force (IETF).  Note that other groups may also distribute
   working documents as Internet-Drafts.  The list of current Internet-
   Drafts is at https://datatracker.ietf.org/drafts/current/.

   Internet-Drafts are draft documents valid for a maximum of six months
   and may be updated, replaced, or obsoleted by other documents at any
   time.  It is inappropriate to use Internet-Drafts as reference
   material or to cite them other than as "work in progress."

   This Internet-Draft will expire on September 10, 2019.

Copyright Notice

   Copyright (c) 2019 IETF Trust and the persons identified as the
   document authors.  All rights reserved.

   This document is subject to BCP 78 and the IETF Trust's Legal
   Provisions Relating to IETF Documents
   (https://trustee.ietf.org/license-info) in effect on the date of
   publication of this document.  Please review these documents
   carefully, as they describe your rights and restrictions with respect
   to this document.  Code Components extracted from this document must
   include Simplified BSD License text as described in Section 4.e of

Montpetit              Expires September 10, 2019               [Page 1]
Internet-Draft                draft-COIN-XR                   March 2019

   the Trust Legal Provisions and are provided without warranty as
   described in the Simplified BSD License.

Table of Contents

   1.  Introduction  . . . . . . . . . . . . . . . . . . . . . . . .   2
     1.1.  Requirements Language . . . . . . . . . . . . . . . . . .   3
   2.  Definitions . . . . . . . . . . . . . . . . . . . . . . . . .   3
   3.  Extended Reality and In-Network Computing . . . . . . . . . .   4
     3.1.  XR Network Requirements . . . . . . . . . . . . . . . . .   4
     3.2.  In-Network Computing Advantages in XR . . . . . . . . . .   5
   4.  XR in Data Intensive Services and Applications  . . . . . . .   6
   5.  Enabling Technologies . . . . . . . . . . . . . . . . . . . .   6
     5.1.  Information Centric Networking (ICN) and Named Data
           Networking (NDN)  . . . . . . . . . . . . . . . . . . . .   7
     5.2.  Network Coding  . . . . . . . . . . . . . . . . . . . . .   7
     5.3.  Blockchains and Distributed Trust . . . . . . . . . . . .   8
   6.  Conclusion  . . . . . . . . . . . . . . . . . . . . . . . . .   9
   7.  Acknowledgements  . . . . . . . . . . . . . . . . . . . . . .   9
   8.  References  . . . . . . . . . . . . . . . . . . . . . . . . .   9
     8.1.  Normative References  . . . . . . . . . . . . . . . . . .   9
     8.2.  Informative References  . . . . . . . . . . . . . . . . .   9
   Author's Address  . . . . . . . . . . . . . . . . . . . . . . . .  10

1.  Introduction

   Virtual Reality (VR) and Augmented Reality (AR) taken together as
   Extended Reality or XR are at the center of a number of technological
   advances in many different fields, including not only gaming and
   entertainment but immersive journalism, remote diagnosis and
   maintenance, telemedicine, manufacturing and assembly and smart
   cities.

   But with the emergence of the edge and the programmability of network
   elements all the way from the data center to the users the
   possibility of creating networked, multiparty/multisource and
   interacting XR comes closer to reality.  This document wants to
   review what is necessary for the current localized and cloud enhanced
   XR to evolve to a more distributed and edge centric architecture to
   support advanced immersive application and services.  It assumes that
   network programmability will enable to tailor the network to the XR
   requirements.  This document is about requirements not solutions per
   se but will mention work that has already been done towards a more
   networked XR including Information Centric architectures, Artificial
   Intelligence and in network coding.  The networked functionality
   should enable to supplement local XR services and devices while
   keeping the very low latency and the very high data rates that are
   required by XR.

Montpetit              Expires September 10, 2019               [Page 2]
Internet-Draft                draft-COIN-XR                   March 2019

   This document is intended as informative to both the networking and
   application research community.  It does not address a specific
   network layer or protocol but provides architecture and system level
   specifications and guidelines.  For example:

      Latency: the physical distance between the XR content cloud of AR/
      VR and users are short enough to limit the propagation delay to
      the 20 ms usually cited for XR applications mixed for example with
      IoT devices and sensors delay reduction for range of interest
      (RoI) detection.

      Applications: better transcoding and use of compression
      algorithms, pre-fetching and pre-caching and movement prediction.

      Network access: push some networking functions in the kernel space
      into the user space to enable the deployment of stream specific
      algorithms for congestion control and application-based load
      balancing based on machine learning and user data patterns.

1.1.  Requirements Language

   The key words "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT",
   "SHOULD", "SHOULD NOT", "RECOMMENDED", "MAY", and "OPTIONAL" in this
   document are to be interpreted as described in RFC 2119 [RFC2119].

2.  Definitions

      - 360-degree video: 360-degree videos, also known as immersive
      videos or spherical videos, are video recordings where a view in
      every direction is recorded at the same time using an
      omnidirectional camera or a collection of cameras.  360o video is
      outside the scope of this document.

      - AR: Augmented Reality (AR) is a live direct or indirect view of
      a physical, real-world environment whose elements are augmented by
      computer-generated input such as sound, video, location or
      graphical data.  It is related to a more general concept called
      mediated reality [MEDIA], in which reality is modified (diminished
      or augmented) by computer-generated imagery.

      - VR Virtual Reality (VR): uses software-generated realistic
      imaging, sounds and other sensor inputs to replicate a real or
      imaginary setting, to simulates a user's physical presence in this
      environment and provide an immersive experience that enable the
      user to interact with objects and move within this space.

      - XR: extended reality is used to address both AR and VR together.

Montpetit              Expires September 10, 2019               [Page 3]
Internet-Draft                draft-COIN-XR                   March 2019

3.  Extended Reality and In-Network Computing

   XR is an example of the Multisource Multidestination Problem that
   combines video, haptics and tactile experiences as well, in
   interactive or networked mode multiparty and social interactions.
   Thus, XR is difficult to deliver with a client-server cloud-based
   solution as it requires a combination of stream synchronization, lows
   delay and delay variations as mentioned above as well as means to
   cover from losses and provide optimized caching in the cloud and
   rendering as close as possible to the user at the network edge.

3.1.  XR Network Requirements

   To deliver the XR experience, there is a need to achieve complete 6
   degrees of freedom meaning the 3 axes for body movement (x,y,z) plus
   pitch, yaw, rotation of the head all of which must be fulfilled in
   real time.  Low delay, low loss and low delay variation are needed to
   avoid sea sickness symptoms if the image does not follow the movement
   [CABLE].

   But this is not the only difficulty, as there is also the need to
   provide real-time interactivity for immersive sports, mobile
   immersive applications with tactile and time-sensitive data and high
   bandwidth for high resolution images.  Since XR deals with personal
   information and potentially protected content (in entertainment and
   gaming) XR must also provide a secure environment and ensure user
   privacy.  And of course, the sheer amount data needed for and
   generated by the XR applications will use recent trend analysis and
   mechanisms, including machine learning to find these trends and
   reduce the size of the data sets [INTER].

   Shared and global immersive experiences require interconnected,
   distributed and federated XR nodes.  The requirements can be
   summarized as:

      - Allow joint collaboration in VR.

      - Provide multi-view AR.

      - Add extra streams (IoT) to AR and VR experiences across data
      intensive services, manufacturing and industrial processes.

      - Provide "Social Television" experiences and global viewing and
      experience rooms.

      - Enable multistream, multidevice, multidestination applications.

Montpetit              Expires September 10, 2019               [Page 4]
Internet-Draft                draft-COIN-XR                   March 2019

      - Use new Internet Architectures at the edge for improved
      performance.

      - Integrate with holography, 3D displays and image processing
      systems [CABLE].

3.2.  In-Network Computing Advantages in XR

   One aspect of the push of XR to the edge is of course to provide
   cloud-based services with much lower latency.  While this is very
   promising the question of the localization of the networking
   resources in order to provide the service becomes an essential
   component of the overall architecture.  But it is not only finding
   the best geographical location but also providing the right level of
   reliability when one or more location is not available especially for
   mission critical services in medicine or manufacturing.  And it does
   not mean only data laid distribution but also ensuring the
   availability of the right computational capabilities.  The
   optimization of the location and type of the required resources for
   the multisouce, multidestination, mutiparty, multi-input XR
   applications can use AI and ML, and advanced load balancing and
   distributed network principles.  There is a need for more research in
   such resource allocation problems at the edge to enable autonomous
   node operation and quality of experience [SOL].These are of course
   multi-variate and heterogeneous goal optimization problems requiring
   advanced analysis with fast converging algorithms [MULTI][PACKET].
   This is essential for the federation of nodes to provide the required
   experience.

   Of course, image rendering and video processing in XR leverages
   different HW capabilities combinations of CPU and GPU.  Current
   programmable network entities need to be evaluated to see if they can
   be sufficient to provide the speed required to provide real-time
   rendering and execute complex analytics: P4 for example does not
   support the floating-point operations necessary for advanced
   graphics.

   Finally, dynamic network programmability could enable the use of
   joint learning algorithms across both data center, edge computers and
   goggles or glasses to allocate functionality and the creation of semi
   permanent datasets and analytics for usage trending.  In the end, the
   use of computing or networked XR will enable the allocation of
   control, forwarding and storage resources and related usage models
   when needed by the application.  This may mean re-evaluating the
   distribution of functionalities between datacenter and edge with less
   critical elements rendered in the cloud combined with a better
   understanding of the operational decomposition of the XR experience

Montpetit              Expires September 10, 2019               [Page 5]
Internet-Draft                draft-COIN-XR                   March 2019

   to allow the use of novel data structures, three-dimensional modeling
   and image processing algorithms.

   Other advantages of adding computing to networked XR include:

      - Multicast distribution and processing as well as peer to peer
      distribution in bandwidth and capacity constrained environments.

      - Evaluation of local caching and micro datacenters with local or
      cloud-based pre-rendering.

      - Trend or ML based congestion control to manage XR sessions
      quality of service.

      - Higher layer protocols optimization to reduce latency especially
      in data intensive applications at the edge.

      - Trust, including blockchains and smart-contracts to enable
      secure community building across domains.

      - Support for nomadicity and mobility (link to mobile edge).

      - Use of 5G slicing to create independent session-driven
      processing/rendering.

      - Performance optimization by tunneling, session virtualization
      and loss protection.

4.  XR in Data Intensive Services and Applications

   In-network computing is essential for data reduction and mutistream
   low latency services at the edge where moving the data to the cloud
   is either requiring too much bandwidth or adding unacceptable
   latency.  Examples of these services included industrial processes
   monitoring, AR-aided design and fabrication and AR/VR supported
   medical interventions.

5.  Enabling Technologies

   This section presents some salient research that will lead to in-
   network computing becoming a major enabler of networked XR.

   NOTE: more information and added sub-sections will be added in future
   versions of the draft with the collaboration of co-authors in the
   specific research areas.

Montpetit              Expires September 10, 2019               [Page 6]
Internet-Draft                draft-COIN-XR                   March 2019

5.1.  Information Centric Networking (ICN) and Named Data Networking
      (NDN)

   The Named Data Networking (NDN) architecture, one architecture of
   ICN, is particularly well suited for the multisource multi-
   destination architecture of XR because it allows to create the
   content experiences based on their components names not a location or
   pointer to a location hence provides a natural functional
   decomposition.  ICN allows content delivery to evolve from single,
   context-independent streams to context-dependent Information
   components that can adapt dynamically to the changes necessary to
   maintain the immersive nature of the experience and be delivered
   efficiently.  The combination of interest messages to signal what
   content is needed combined with the data responses help to coordinate
   the different streams and multiple users (pull mechanisms).  The ICE-
   AR [ICE] project already mentions a concept of acceleration as a
   service: the exploration of the design and the usage of computation
   at the edge including the wireless edge.

   For XR, ICN also allows to develop robust and resilient networking
   while allowing application developer to continue using known
   programming model [RICE].  This is important for the XR developers
   community that come from the entertainment, gaming or other non
   networking specific industries and could enable ICN and XR to coexist
   in user devices (the ultimate edge).  NDN concepts are already
   integrated to distributed video distribution with trust mechanisms
   (see section below) such as smart contracts on the blockchain to
   proof of origin and destination sent along with interest messages
   [HUITX].

5.2.  Network Coding

   Networked XR requires the synchronization of multiple streams but
   with its delay sensitivity the use of buffering schemes to achieve
   this synchronization is impractical.  At the same time the need to
   maintain high data integrity means that packet losses also need to be
   limited.  Network coding has proven very useful to achieve both these
   goals in commercial streaming services like Netflix, is being added
   to protocols like QUIC, multi-stream services such as Social
   Television [SOCIAL] and other data-centric low latency applications.
   This avoids being reliant on complex synchronization algorithms.  The
   many XR servcies are constrained in latency and loss budgets
   especially in mission critical applications hence even the delay due
   to encoding and decoding operations needs to be minimized.  Hence the
   idea of in-network coding and re-encoding to adapt to dynamic network
   conditions, not just end to end, can be used to ensure on time packet
   delivery with loss recovery.  In network encoding needs the type of
   programmability that COIN provides and the developement of

Montpetit              Expires September 10, 2019               [Page 7]
Internet-Draft                draft-COIN-XR                   March 2019

   programmable switches and the P4 language may allow to create very
   efficient in-network coding even considering the limitations of the
   language Note: references to be added.

5.3.  Blockchains and Distributed Trust

   If XR is to be integrated at the edge of the network to provide the
   required delay and loss guarantees, then relying on centralized
   mechanisms for trust is non-realistic.  Traditional centralized
   mechanisms to discover and admit nodes to the network, to provide
   access right and name resolution need to be updated to be used in the
   dynamic XR environment.Blockchain technology, with operations
   performed at the edge and in a decentralized way is fast becoming a
   major scalable means of providing trust and validate provenance in a
   large number of applications including those on the XR portfolio.

   Smart contracts (on the blockchain) supply a mechanism to provide the
   trust and validation for XR edge nodes.  A new XR participant node is
   admitted after it has committed to a smart contract that contains the
   rules and mechanisms to distribute content via this node in a trusted
   and secure way.  This constitutes its proof of validity.  After a
   node is admitted, it will can then provisioned with the appropriate
   software to become fully operational to provide the XR experience.
   Newly admitted nodes will be inserted in the general ledger on the
   blockchain enabling other nodes to discover them, and hence, to form
   a trusted network.  A name resolution authority can also be provided
   by the blockchain to manage and validate the origin of the content,
   the proof of origin, and to provide the ability to search such
   content.  The proof of origin can also be used to prevent some
   content from reaching one or more nodes and implement content
   filtering based on trusted authorities.  This is useful not only for
   content packets but also for packets capable of modifying the node
   operations.  Finally, when some content reaches a specific
   destination, it can be verified against the content rules of the
   reached node even and before it is sent to the application; this
   allows to provide a proof of delivery for the content and enable to
   generate statistics, performance metrics and enable the nodes to
   adapt to the XR requirements.

   All of the above assumes that the nodes can implement the functions
   needed by the blockchain hence once again infers that there is enough
   computing power in the nodes to perform these operations.  At this
   point both proof of concept and proof of every are limited due to the
   added overhead and the size of the blockchain.  As distributed
   blockchains and COIN continue to evolve this should continue to be a
   field of interest for the development of secure and private XR
   experiences.

Montpetit              Expires September 10, 2019               [Page 8]
Internet-Draft                draft-COIN-XR                   March 2019

6.  Conclusion

   More and more applications and service are being developed and
   deployed that use or will use combinations of AR and VR, XR, along
   with extra stream from sensors and IoT devices.  And many of these
   applications require to be deployed over a network because of their
   interactive or multiparty nature.  In that context, it not uniquely
   necessary to move functionality to the network but to carefully
   evaluate which elements to locate in network nodes, where these nodes
   are and what computational support they need to support the XR
   experience.  Hence, it is believed that a great enabler of networked
   XR is the capability to co-locate programmable elements in the XR
   network node to respond to the dynamics of the services in an
   efficient, resilient and secure manner.

7.  Acknowledgements

   The author would like to thank Jeffrey He, Dirk Kutscher, Cedric
   Westphal and Weiguang Wang for their contributions to the
   presentation that lead to this draft.

8.  References

8.1.  Normative References

   [RFC2119]  Bradner, S., "Key words for use in RFCs to Indicate
              Requirement Levels", BCP 14, RFC 2119,
              DOI 10.17487/RFC2119, March 1997,
              <https://www.rfc-editor.org/info/rfc2119>.

8.2.  Informative References

   [CABLE]    Hinds, A., "The Near Future of Immersive Experiences:
              Where We Are on the Journey, What Lies Ahead, and What It
              Takes to Get There.", SIGCOMM 2018 Workshop on AR/VR.
              http://conferences.sigcomm.org/sigcomm/2018/workshop-
              arvr.html, August 2018.

   [HUITX]    "8X: ICN Based Video Distribution.", 8XLab Web
              Site. https://www.8xlabs.com, 2018.

   [ICE]      Burke, J., "ICN-Enabled Secure Edge Networking with
              Augmented Reality: ICE-AR.", ICE-AR Presentation at
              NDNCOM. https://www.nist.gov/news-
              events/events/2018/09/named-data-networking-community-
              meeting-2018, September 2018.

Montpetit              Expires September 10, 2019               [Page 9]
Internet-Draft                draft-COIN-XR                   March 2019

   [INTER]    Bastug, E., "Towards Interconnected Virtual
              Reality:Opportunities, Challenges and Enablers.", IEEE
              Communications Magazine, Volume 55 , Issue:
              6. https://arxiv.org/pdf/1611.05356.pdf, June 2017.

   [MEDIA]    Wikipedia.org, "Mediated Reality.", 2018,
              <https://en.wikipedia.org/wiki/Computer-mediated_reality>.

   [MULTI]    Batalla, J., "Evolutionary Multi-objective optimization
              algorithm for multimedia delivery in critical applications
              through Content-Aware Networks.", The Journal of
              Supercomputing. https://link.springer.com/article/10.1007/
              s11227-016-1731-x, March 2017.

   [PACKET]   Jeyakumar, V., "Millions of Little Minions: Using Packets
              for Low Latency Network Programming and Visibility.",
              Proceedings of SIGCOMM 2014.
              http://conferences.sigcomm.org/sigcomm/2014/program.php,
              August 2014.

   [RICE]     Krol, M., "RICE: Remote Method Invocation in ICN.",
              Proceedings of the ACM Conference on Information-Centric
              Networking 2018. http://conferences.sigcomm.org/acm-
              icn/2018/proceedings/icn18-final9.pdf, September 2018.

   [SOCIAL]   Montpetit, M. and M. Medard, "Social Television: Enabling
              Technologies and Architectures.", Proceedings of the IEEE,
              Volume 100, pp.
              1395-1399. http://proceedingsoftheieee.ieee.org, May 2012.

   [SOL]      Heorhiadi, V., "Simplifying Software-Defined Network
              Optimization Using SOL.", 13th USENIX Symposium on
              Networked Systems Design and Implementation.
              https://www.usenix.org/system/files/conference/nsdi16/
              nsdi16-paper-heorhiadi.pdf, March 2016.

   [VRSICK]   LaViola, J., "SA Discussion of Cybersickness in Virtual
              Environments.", ACM SIGCHI Bulletin 32(1):47-56.
              http://www.eecs.ucf.edu/~jjl/pubs/cybersick.pdf, January
              2000.

Author's Address

   Marie-Jose
   Triangle Video

   Email: marie@mjmontpetit.com

Montpetit              Expires September 10, 2019              [Page 10]