Framework of Fast Fault Detection for IP-based Network

Abstract

The IP-based distributed system and software application layer often use heartbeat to maintain the network topology status. However, the heartbeat setting is long, which prolongs the system fault detection time. IP-based storage network is the typical usage of that scenario. When the IP-based storage network fault occurs, NVMe connections need to be switched over. Currently, no effective method is available for quick detection, switchover is performed only based on keepalive timeout, resulting in low performance.¶

This document defines the basic framework of how network assisted host devices can quickly detect application connection failures caused by network faults.¶

1. Introduction

Today, distributed systems based on network communication are widely used. In order to ensure that both ends of the distributed system can perceive faults, heartbeat is a common technology. However, relying on the heartbeat to detect whether the peer is faulty also faces challenges: if the heartbeat is set too short, it may be misjudged by network disturbances; if the heartbeat is set too long, when a fault occurs, it will not be found for a long time.¶

Application scenarios such as IP-based NVMe, distributed storage, and cluster computing are typical scenarios for such technologies.¶

The [I-D.guo-ffd-requirement] describes the problems of the current IP-based NVMe solution. On an IP-based storage area network, if the access link of a storage device is faulty, hosts cannot access the storage device. Because the host cannot directly detect the fault, the host has to wait for the KA timeout. To speed up fault detection, hosts and storage devices can implement fast KA or BFD. However, this solution introdueced additional cost on hosts and storage devices and is hard to use in large-scale IP-based storage area network. In fact, the IP network can directly detect these faults, so we can use the IP network to assist these access endpoints to quickly perceive the fault, so as to perform quickly service recovery.¶

2. Terminology

NoF : NVMe of Fabrics¶

FC : Fiber Channel¶

NVMe : Non-Volatile Memory Express¶

SAN: Storage Area Network¶

3. Reference Models

The frame solution here is applicable to the system where the terminals are directly connected to the IP network.¶

 +--------+    +-----------+     +-----------+    +--------+
 |Terminal|----| IP Network|-----| IP Network|----|Terminal|
 | Device |    |   Device  |     |   Device  |    | Device |
 +--------+    +-----------+     +-----------+    +--------+
             Figure 1 : Basic framework

Terminals are connected to the IP network, and they establish IP connections through the reachability provided by the IP network. When the connection path fails, they cannot be detected quickly. They can only detect it after the keep-alive timeout, and then can carry out service protection processing. This time may be relatively long. Therefore, it is necessary to notify the terminal device of some failures in the network, such as access port failures and internal network failures that will cause IP connection failures between terminals, so that the terminal device can respond quickly and perform corresponding service processing.¶

As introduced in Introduction, there are scenarios such as IP-based NVMe, distributed storage, and cluster computing. Here we take IP-based NVME as a typical scenario for introduction, and the processing behavior of other scenarios is similar.¶

An IP-based storage area network mainly includes three types of roles:¶

o Initiator, the terminal device, is also called the host.¶

o Switch, which is a network device used to access terminal devices.¶

o Target is also a terminal device, also known as a storage device.¶

The host and storage devices use the Ethernet-based NVMe protocol to transmit data through the IP network to provide high-performance storage services.¶

               +--+       +--+
    Host       |H1|       |H2|
 (Initiator)   +-,+       +_.+
                | `',   _-` |
                |    _-`    |
                | _-`   `', |
      IP     +----+       +----+
   Network   | SW1|       | SW2|
             +---,+       +_.--+
                | `',   _-` |
                |    `',    |
                | _-`   `', |
   Storage     +-`+       +`'+
   (Target)    |S1|       |S2|
               +--+       +--+
     Figure 1 : Small-scale SAN

               +--+      +--+      +--+      +--+
   Host        |H1|      |H2|      |H3|      |H4|
(Initiator)    +/-+      +-,+      +.-+      +/-+
                |         | '.   ,-`|         |
                |         |   `',   |         |
                |         | ,-`  '. |         |
              +-\--+    +--`-+    +`'--+    +-\--+
              | SW1|    | SW2|    | SW3|    | SW4|
              +--,-+    +---,,    +,.--+    +-.--+
                  `.          `'.,`         .`
                    `.   _,-'`    ``'.,   .`
    IP              +--'`+            +`-`-+
  Network           | SW5|            | SW6|
                    +--,,+            +,.,-+
                    .`   `'.,     ,.-``   ',
                  .`         _,-'`          `.
              +--`-+    +--'`+    `'---+    +-`'-+
              | SW7|    | SW8|    | SW9|    |SW10|
              +-.,-+    +-..-+    +-.,-+    +-_.-+
                | '.   ,-` |        | `.,   .' |
                |   `',    |        |    '.`   |
                | ,-`  '.  |        | ,-`  `', |
  Storage      +-`+      `'\+      +-`+      +`'+
  (Target)     |S1|      |S2|      |S3|      |S4|
               +--+      +--+      +--+      +--+
               Figure 2 : Large-scale SAN

4. Functional Components

The NVMe IP-based SANs consists of storage devices, hosts and switches. Hosts and storage devices need to obtain required fault information from the IP network. Switches need to synchronize locally detected fault information on the IP network so that other switches can obtain the faults and notify hosts or storage devices that require the fault infomation.¶

  +-------+                  +------+
  |Storage|                  |Switch|
  +-------+                  +------+
      |      Subscribe Msg      |
      | ----------------------->|
      |                         |
      |     Notification Msg    |
      | <-----------------------|
      |                         |
      |                         |
      Figure 3 : Storage Device

+-------+                  +------+
|  HOST |                  |Switch|
+-------+                  +------+
    |      Subscribe Msg      |
    | ----------------------->|
    |                         |
    |     Notification Msg    |
    | <-----------------------|
    |                         |
    |                         |
     Figure 4 : Host Device

+------+                  +------+
|Switch|                  |Switch|
+------+                  +------+
   |    Information Sync     |
   | ----------------------->|
   |                         |
   |                         |
   |                         |
    Figure 5 : Network Device

5. Procedures

+-------+           +----+      +------+      +----+         +-------+
|  HOST |-----------|TOR1|------|Spine1|------|TOR3|---------|Storage|
+---/---+           +-/--+      +--/---+      +-/--+         +---/---+
    |---------------->|  Info Sync |  Info Sync |<---------------|
    |  SubscribeMsg   |----------->|<-----------|  Subscribe Msg |
    |                 |<-----------|----------->|                |
    |<----------------|  Info Sync |  Info Sync |                |
    |Notification Msg |            |            |                |
    |                 |            |            |                |
            Figure 7 : Information Advertisement

8. References