DetNet                                                         H. Wang
Internet Draft                                                 P. Wang
Intended status: Standards Track                               H. Yang
Expires: November 15, 2018                     Chongqing University of
                                          Posts and Telecommunications
                                                          May 14, 2018


      Joint Real-Time Scheduling Methods for Deterministic Industrial
                          Field/Backhaul Networks
                   draft-wang-detnet-joint-scheduling-03


Abstract

   In industrial field/backhaul networks, joint real-time scheduling
   method is important to make end-to-end flows meet their deadline.
   This document proposes four joint scheduling methods, and they
   involve four scenarios: time-slotted industrial backhaul network,
   regarding industrial backhaul network as a black box system,
   ignoring delay of industrial backhaul and establishing latency model
   of industrial backhaul network.

Status of this Memo

   This Internet-Draft is submitted in full conformance with the
   provisions of BCP 78 and BCP 79.

   Internet-Drafts are working documents of the Internet Engineering
   Task Force (IETF), its areas, and its working groups. Note that
   other groups may also distribute working documents as Internet-
   Drafts.

   Internet-Drafts are draft documents valid for a maximum of six
   months and may be updated, replaced, or obsoleted by other documents
   at any time. It is inappropriate to use Internet-Drafts as reference
   material or to cite them other than as "work in progress."

   The list of current Internet-Drafts can be accessed at
   http://www.ietf.org/ietf/1id-abstracts.txt

   The list of Internet-Draft Shadow Directories can be accessed at
   http://www.ietf.org/shadow.html

   This Internet-Draft will expire on November 15, 2018.






Wang, et al.          Expires November 15, 2018               [Page 1]


 Internet-Draft           DetNet Joint Scheduling              May 2018


Copyright Notice

   Copyright (c) 2018 IETF Trust and the persons identified as the
   document authors. All rights reserved.

   This document is subject to BCP 78 and the IETF Trust's Legal
   Provisions Relating to IETF Documents
   (http://trustee.ietf.org/license-info) in effect on the date of
   publication of this document. Please review these documents
   carefully, as they describe your rights and restrictions with
   respect to this document. Code Components extracted from this
   document must include Simplified BSD License text as described in
   Section 4.e of the Trust Legal Provisions and are provided without
   warranty as described in the Simplified BSD License.

Table of Contents


   1. Introduction ................................................. 2
   2. Deterministic Industrial Field/Backhaul Network Requirement .. 4
   3. Deterministic Industrial Field/Backhaul Network Joint Scheduling
      Key Technology ............................................... 5
      3.1. End-to-end Network Data Stream .......................... 5
      3.2. Network Communication Resource .......................... 5
      3.3. Network Time Slot Scheduling ............................ 6
   4. Joint Real-Time Scheduling Methods for Deterministic Industrial
      Field-Backhaul Network ....................................... 6
      4.1. Time-Slotted Industrial Backhaul Networks ............... 6
      4.2. Consider Industrial Backhaul Network as a Black Box .... 10
      4.3. Ignore the Delay of Industrial Backhaul Network ........ 11
      4.4. Build Delay Model of Industrial Backhaul Network ....... 11
   5. Security Considerations ..................................... 11
   6. IANA Considerations ......................................... 11
   7. References .................................................. 12
      7.1. Normative References ................................... 12
      7.2. Informative References ................................. 12
   Authors' Addresses ............................................. 13

1. Introduction

   Industrial field networks are often deployed to process control
   industry to monitor industrial field equipment. Industrial field
   network can improve production efficiency, reduce human intervention
   and decrease cost, which are significant for industrial
   modernization.




Wang, et al.          Expires November 15, 2018               [Page 2]


 Internet-Draft           DetNet Joint Scheduling              May 2018


   Industrial field bus and industrial Ethernet are two kinds of common
   networks deployed in industrial automation, while they are wired
   networks. With the development of industrial wireless technology,
   Wireless Sensor Networks (WSN), a typical industrial wireless
   network, has been applied to industrial network. WSN can free
   traditional field devices from the limits of abundant cables, and it
   is flexible to deploy in industrial environment. WSN can be applied
   to building automation, process automation, and industrial
   automation. Currently, There are three major industrial wireless
   networks international standards: ISA100.11a[IEC62734],
   WirelessHART[IEC62591], WIA-PA[IEC62601].

   Industrial backhaul network is used as transition network, which
   combines industrial field network with high-level network to achieve
   the goal of interconnection. It mainly solves the problem that makes
   the sensor or control data from industrial field network transmit to
   high-level network. Generally, industrial field network is deployed
   to a specific region. Through industrial backhaul network, data of
   industrial field network can be transferred to internet or other
   industrial field networks. Industrial backhaul network is a medium-
   sized network, which can cover from a few kilometers to tens of
   kilometers. The major technology of industrial wireless backhaul
   network consists of Wi-Fi, WiMAX and LTE.

   To apply well in the burgeoning industry 4.0, which aims to elevate
   the level of manufacturing, industrial field network should not be
   confined to a plant network only. Therefore, it is necessary to
   introduce the technology of industrial backhaul network to break the
   restrictions of interconnection between different networks, and
   construct a hybrid industrial network. Figure 1 indicates a typical
   network architecture of the hybrid industrial network. It is a type
   of architecture of industrial deterministic network that was
   illustrated with use cases in the drafts proposed by DetNet
   Workgroup of IETF of [I-D.bas-usecase-detnet] and [I-D.finn-detnet-
   architecture].













Wang, et al.          Expires November 15, 2018               [Page 3]


 Internet-Draft           DetNet Joint Scheduling              May 2018



                  +-----------------------------------+
                  |                                   |
                  |                                   |
                  |           Backhaul network        |
                  |                                   |
                  |                                   |
                  +-----------------------------------+
                           /                  \
                          /                    \
   +-------------------------------+  +-------------------------------+
   |                               |  |                               |
   |                               |  |                               |
   |         Field network         |  |         Field network         |
   |                               |  |                               |
   |                               |  |                               |
   +-------------------------------+  +-------------------------------+

            Figure 1. Typical industrial field/backhaul network

   In the hybrid network architecture, field network may be an
   ISA100.11a. In Figure 1, a node deployed in a plant can communicate
   with a node in another plant through backhaul network.

2. Deterministic Industrial Field/Backhaul Network Requirement

   The draft of [I-D.finn-detnet-problem-statement], proposed by DetNet
   Workgroup of IETF, has described the requirements of deterministic
   network and deterministic scheduling partially. Due to industrial
   field network directly monitor the industrial process, a difference
   between industrial field network and general network exists.
   Industrial field network has high demands on the deterministic delay
   bounds. In a field network, the delay of data flows will affect
   productivity, and even cause industrial accidents when happening
   high packet loss ratio and transmission latency. For example, real-
   time measure and control of liquid level is required to avoid
   overfilling of oil tanks, because overflow may lead to serious
   economic loss and environmental threats. Therefore, it requires a
   deterministic joint scheduling method to guarantee the deterministic
   transmission of data stream in the new network architecture.









Wang, et al.          Expires November 15, 2018               [Page 4]


 Internet-Draft           DetNet Joint Scheduling              May 2018


3. Deterministic Industrial Field/Backhaul Network Joint Scheduling Key
   Technology

3.1. End-to-end Network Data Stream

   In industrial field/backhaul network, end-to-end data stream
   indicates a complete transmission path that a source node of field
   network transfers to destination node located in another field
   network through an industrial backhaul network.

   Industrial field/backhaul network data stream has following features:

   o Period. Every data stream generates data with periodicity.

   o Deterministic. Every data stream has a deadline, and scheduling
      methods should ensure each data stream arrives at destination
      node before its deadline.

   o Sequential. A path of an end-to-end data stream contains some
      transmission links. In the process of scheduling, it must be
      scheduled in the order of sequence of links on the path.

   o Priority. End-to-end data stream has a  priority. When data
      streams with different priorities occur collisions, the data
      streams with lower priority should be delayed by higher priority
      data streams.

3.2. Network Communication Resource

   In deterministic industrial field/backhaul network architecture,
   network communication resources include time slot, channel and link.
   If backhaul network adopts Software Defined Network (SDN)
   architecture, then the SDN controller can schedule the bandwidth and
   cache of switch. Therefore, bandwidth and cache resources can be
   included in schedulable communication resources.

   o Time slot. Time slot is the basic transmission unit in the
      network communications based on Time Division Multiple Access
      (TDMA). In the entire network, the length of time slots is fixed
      and stays the same. Only one sending packet and its corresponding
      ACK can be accommodated in one time slot.

   o Channel. In order to increase network throughput, industrial
      field network provides a number of channels with different
      frequencies.




Wang, et al.          Expires November 15, 2018               [Page 5]


 Internet-Draft           DetNet Joint Scheduling              May 2018


   o Link. Link refers to a direct packet transmission between two
      nodes that located in a communication radius of each other. A
      data stream comprises many links.

3.3. Network Time Slot Scheduling

   In TDMA-based industrial field network, time is divided into time
   slots with the same length. In the time-slot scheduling process, it
   will cause link collisions when a node transmits and receives
   simultaneously, and it will cause channel collisions when the same
   channel is used within a certain range. As shown in Figure 2, the
   time-slot scheduling process should avoid such collisions.

        +---+   +---+   +---+           +---+   +---+   +---+   +---+
        | A |-->| B |-->| C |           | A |-->| B |   | C |-->| D |
        +---+   +---+   +---+           +---+   +---+   +---+   +---+

         +---------+------------+         +---------+------------+
         |Time slot| Time slot 0|         |Time slot| Time slot 0|
         +---------+------------+         +---------+------------+
         |Channel 0|   A->B     |         |Channel 0|   A->B     |
         +---------+------------+         |         |   C->D     |
         |Channel 1|   B->C     |         +---------+------------+
         +---------+------------+
              Figure 2. Link Collision & Channel Collision

4. Joint Real-Time Scheduling Methods for Deterministic Industrial
   Field-Backhaul Network

   Joint real-time scheduling methods of industrial field/backhaul
   networks intend to solve the deterministic problem of industrial
   field/backhaul networks. Due to the investigative architecture
   includes backhaul network, the deterministic scheduling algorithm
   needs to collaborate with backhaul network to conduct joint
   scheduling to ensure data deterministic transmission. The proposed
   joint scheduling methods are described as follows.

4.1. Time-Slotted Industrial Backhaul Networks

   In order to ensure determinism, industrial field networks adopts
   TDMA to make the network time-slotted. If the industrial backhaul
   network can also be time-slotted, then the deterministic scheduling
   algorithm can jointly schedule with small modification. Industrial
   backhaul network contains various of network standards such as WIFI,
   WiMAX, and LTE. WiMAX and LTE are high cost and poor feasibility,
   thus we assume the IEEE 802.11 as backhaul network. Wi-Fi network
   has various operating modes, such as peer-to-peer mode, point to


Wang, et al.          Expires November 15, 2018               [Page 6]


 Internet-Draft           DetNet Joint Scheduling              May 2018


   multi-point networking mode and the relay network mode. Here we
   consider the hierarchical network architecture in a way of point to
   multi-point networking mode, as shown in Figure 3.

               +----------------------------------------+
               |                                        |
               |               +--------+               |
               |       +-------| Head AP|-------+       |
               |       |       +--------+       |       |
               |       |                        |       |
               |   +--------+              +--------+   |
           +---+---|   AP1  |              |   AP2  |---+---+
           |   |   +--------+              +--------+   |   |
           |   +----------------------------------------+   |
           |                                                |
 +---------------------------------++----------------------------------+
 |ISA100.11a field wireless network||ISA100.11a field wireless network |
 +---------------------------------++----------------------------------+

          Figure 3. Industrial Backhaul Network consisting of WIFI

   Although IEEE 802.11 supports multiple channels, but AP is not able
   to perform channel hopping between transmission timeslots, which
   means that the AP cannot use a channel in the current time slot and
   use another channel the next time slot. We assume that AP1 and AP2
   in Figure 3 can transmit packets simultaneously as long as their
   transmission tasks do not contain the same AP, i.e. head AP. For
   example, when a data stream of field network is transmitting packets
   to AP1 in a time slot, AP2 is able to receive packets from head AP,
   or send packets to field network in the same time slot. Therefore,
   the backhaul network framework with wireless APs can be considered
   as a single-channel linear network, which is shown in Figure 4.


+---------+    +--------+     +--------+    +--------+    +---------+
| Gateway |--> |   AP   |-->  |   AP   |--> |   AP   |--> | Gateway |
+---------+    +--------+     +--------+    +--------+    +---------+

                 Figure 4. A single-channel linear network

   Therefore, the data stream in industrial field/backhaul network can
   be seen as equivalent to the data stream in field network, except
   that each data stream needs to flow through the WIFI. The scheduling
   process is illustrated as follows:


Wang, et al.          Expires November 15, 2018               [Page 7]


 Internet-Draft           DetNet Joint Scheduling              May 2018


   1. Abstract end-to-end data stream in the entire network, and
      allocate a priority for each stream.

   2. Establish the delay model of network data stream. If collisions
      happened between different priority data stream, the low priority
      data stream will be delayed by high priority, so a model can be
      built under the worst circumstances that the low-priority data
      streams impacted by all higher priority data streams.

   3. Estimate the network schedulability. A data stream is schedulable
      when the minimum time for the data stream to complete its once
      transmission task plus the worst delay time caused by higher
      priority data streams is less than or equal to its deadline, In
      the current priority allocation scheme, if each data stream is
      schedulable, the network can be considered as schedulable. If the
      data stream cannot be scheduled, then try to change the priority
      allocation scheme and estimate again until a corresponding scheme
      is found or return no schedulable results.

   4. Allocate time slot and channel for each data stream. Traverse
      data streams according to their priorities, and each data stream
      should allocate link that is about to be released in a time slot.
      According to the rule that low priority data streams should give
      way to high priority data streams, the channels can be utilized
      if it is not unoccupied. However, if collisions happened between
      data streams of different priority, then the lower priority data
      stream should be placed in the next time slot until there are no
      unallocated higher priority data streams. Repeat these steps
      until the whole network scheduling is completed.

   The scheduling process is described in Figure 5:

















Wang, et al.          Expires November 15, 2018               [Page 8]


 Internet-Draft           DetNet Joint Scheduling              May 2018


          +----------+
          |   Begin  |
          +----------+
               |
               |
   +---------------------------+
   |  Initial the priority of  |
   |      each data stream     |
   +---------------------------+
               |<--------------------------------------+
               |                                       |
     +--------------------+            +------------------------------+
    /  Traverse every data \     no    | If the data stream cannot be |
   / stream and estimate the\--------->| scheduled, then change the   |
   \ schedulablity according/          | priority allocation scheme   |
    \   to delay model     /           | and estimate again           |
     +--------------------+            +------------------------------+
               |
               |yes
   +-----------------------------------+
   | Traverse data streams according to|
   | their priority, each data stream  |
   | should allocate the next link that|
   | is about to be released in each   |
   | time slot to the greatest extent  |
   +-----------------------------------+
               |
               |
   +-----------------------------------+
   | The spare channels can be utilized|
   | if there is no collision. If      |
   | collisions happened, then the     |
   | lower priority data stream should |
   | be placed in the next time slot   |
   +-----------------------------------+
               |
               |
           +-------+
           |  End  |
           +-------+
    Figure 5. Scheduling of times-slotted industrial backhaul network


Wang, et al.          Expires November 15, 2018               [Page 9]


 Internet-Draft           DetNet Joint Scheduling              May 2018


   Further, if the backhaul network can support TDMA mechanism like the
   industrial field network completely, the deterministic scheduling
   methods in field network can be applied in industrial field/backhaul
   networks.

   For backhaul network using wired technology, time-sensitive network
   based on Ethernet is preferred for industrial scenarios. Time-
   sensitive network can provide dedicated slots for scheduled traffic,
   so above scheduling method can be used in this kind of backhaul
   network to guarantee the deterministic performance for data flows
   across field and backhaul networks.

4.2. Consider Industrial Backhaul Network as a Black Box

   In order to solve the deterministic problem of industrial network,
   backhaul network can be regarded as a black box so that we can only
   consider its delay impacts and ignore its internal details.

   When the packet passes through the industrial backhaul network, we
   can give it a timestamp at the application layer and read it after
   the transmission is ready to leave the backhaul network. Delay
   caused in backhaul network can be calculated, and a fitting curve of
   delay can be obtained by collecting large amount of data. It has
   been verified experimentally that the delay is concentrated in a
   numerical range despite its randomness. Therefore, we can estimate
   the approximate delay time caused by industrial backhaul network.

   A main scheduling path can be configured according to the average
   delay of the backhaul network. Some redundant paths should be pre-
   configured in case the delay of the main path is too high. The
   scheduling process of industrial field/backhaul network can be
   divided into three sections, as shown in Figure 6:

+--------------------+  +-----------------+  +------------------------+
|   Scheduling of    |  |    Delay of     |  |      Scheduling of     |
|source field subnet |->| backhaul network|->|destination field subnet|
|   (deterministic)  |  |(indeterministic)|  |( deterministic dynamic)|
+--------------------+  +-----------------+  +------------------------+
        Period 1              Period 2                Period 3

                   Figure 6. Three periods of scheduling

   In source field subnet we can apply the deterministic scheduling
   algorithm of field network to get the time spent by each data stream
   before entering the source subnet. Then the data stream enters the
   backhaul network, which will cause indeterministic delay in a
   numerical range. When the data stream leaves the backhaul network,


Wang, et al.          Expires November 15, 2018              [Page 10]


 Internet-Draft           DetNet Joint Scheduling              May 2018


   the timestamp should be parsed. If the deadline is missed, it
   indicates that the packet has gone through poor network and needs to
   be retransmitted. If there is free time after leaving the backhaul
   network, scheduling path can be dynamically selected at downward
   gateway to get the schedulability of the end-to-end data stream.

4.3. Ignore the Delay of Industrial Backhaul Network

   Since the field network is slow-speed (250 KB/s), while industrial
   backhaul network is high-speed, if the industrial backhaul networks
   adopt IEEE 802.11 protocol, gigabit wireless routers supporting IEEE
   802.11 ac can make the delay of industrial backhaul network quite
   low. As a result, the joint deterministic scheduling of the entire
   network only needs to consider the field networks.

4.4. Build Delay Model of Industrial Backhaul Network

   If industrial backhaul network is constructed with IEEE 802.11, the
   network access delay test model in IEEE 802.11 Distributed
   Coordination Function (DCF) mode can be established by using Markov
   chain or queue theory. While the model in IEEE 802.11 Point
   Coordination Function (PCF) mode can be established based on queue
   theory.

   Therefore, the field network needs to build a delay model, while
   backhaul network follows another delay model, then the total
   transmission scheduling delay will have certain regularity. The
   total transmission delay will meet delay requirements with specified
   probability by scheduling, in other words, the unsuccessful
   scheduling is acceptable.

5. Security Considerations

6. IANA Considerations

   This memo includes no request to IANA.












Wang, et al.          Expires November 15, 2018              [Page 11]


 Internet-Draft           DetNet Joint Scheduling              May 2018


7. References

7.1. Normative References

7.2. Informative References

[IEC62734]
           ISA/IEC, "ISA100.11a, Wireless Systems for Automation,
           also IEC 62734", 2011, <http://www.isa100wci.org/enUS/
           Documents/PDF/3405-ISA100-WirelessSystems-Future-brochWEB-
           ETSI.aspx>.

[IEC62591]
           IEC, "Industrial Communication Networks -
           Wireless Communication Network and Communication Profiles
           - WirelessHART - IEC 62591", 2010,
           <https://webstore.iec.ch/p-
           preview/info_iec62591%7Bed1.0%7Den.pdf>

[IEC62601]
           IEC, "Industrial networks - Wireless communication network
           and communication profiles - WIA-PA - IEC 62601", 2015, <
           https://webstore.iec.ch/preview/info_iec62601%7Bed2.0%7Db
           .pdf>

[I-D.finn-detnet-problem-statement]
           Finn, N. and P. Thubert, "Deterministic Networking Problem
           Statement", draft-finn-detnet-problem-statement-05 (work in
           progress), March 2016.

[I-D.finn-detnet-architecture]
           Finn, N., Thubert, P., and M. Teener, "Deterministic
           Networking Architecture", draft-finn-detnet-architecture-08
           (work in progress), August 2016.

[I-D.bas-usecase-detnet]
           Kaneko, Y., Toshiba and Das, S, "Building Automation Use
           Cases and Requirements for Deterministic Networking", draft-
           bas-usecase-detnet-00 (work in progress), October 2015.










Wang, et al.          Expires November 15, 2018              [Page 12]


 Internet-Draft           DetNet Joint Scheduling              May 2018


Authors' Addresses

   Heng Wang
   Chongqing University of Posts and Telecommunications
   2 Chongwen Road
   Chongqing, 400065
   China

   Phone: (86)-23-6248-7845
   Email: wangheng@cqupt.edu.cn


   Ping Wang
   Chongqing University of Posts and Telecommunications
   2 Chongwen Road
   Chongqing, 400065
   China

   Phone: (86)-23-6246-1061
   Email: wangping@cqupt.edu.cn


   Hang Yang
   Chongqing University of Posts and Telecommunications
   2 Chongwen Road
   Chongqing, 400065
   China

   Phone: (86)-23-6246-1061
   Email: 18716322620@163.com


















Wang, et al.          Expires November 15, 2018              [Page 13]