T2TRG                                               Hong, Choong Seon
Internet-Draft                                   Kyung Hee University
Intended status: Standards Track                    Munir, Md. Shirajum
Expires: August 15, 2022                         Kyung Hee University
                                                      Kitae Kim
                                                      Kyung Hee University
                                       Seok Won Kang
                                         Kyung Hee University




Proactive energy management for smart city with edge
computing using meta-reinforcement learning scheme
                        draft-hongcs-t2trg-pem-00

Abstract

Renewable energy enabled sustainable energy management ensures
a high degree of reliability in order to fulfill the energy
demand of a smart city. In such case, renewable energy
generation is random over time and also energy consumption of
smart city users’ is nondeterministic in nature. Therefore, to
ensure sustainable energy management for smart city,
a proactive energy management scheme should be integrated into
smart city network. In which, edge node should be considered as
local computational unit for each energy user and microgrid
controller should be played the role of energy management decision
aggregator. As a result, proactive energy management scheme
not only overcomes the challenges of renewable energy-aware
demand scheduling but also establishes a strong relationship
for both energy generation and consumption over time.
Therefore, a distributed mechanism is considered, where the edge
node for executing local agent to determine an individual
users’ policy with respect to energy consumption and renewable
energy generation (users’ own sources). On the other hand,
microgrid controller determines meta-policy through a meta-agent
with Recurrent Neural Network (RNN). Since a meta-agent accepts
local policy as an input with historical observations, which
ensures fast and efficient execution of proactive energy management
for the smart city.

Status of this Memo

This Internet-Draft is submitted in full conformance with the
provisions of BCP 78 and BCP 79.

Internet-Drafts are working documents of the Internet Engineering
Task Force (IETF).  Note that other groups may also distribute
working documents as Internet-Drafts. The list of current Internet-
Drafts is at http://datatracker.ietf.org/drafts/current/.

Internet-Drafts are draft documents valid for a maximum of six
months and may be updated, replaced, or obsoleted by other
documents at any time.  It is inappropriate to use Internet-Drafts
as reference material or to cite them other than as
"work in progress."




Hong, et al.          Expires  August 15, 2022                  [Page 1]


Internet-Draft  Proactive energy management for smart city  October 2020

This Internet-Draft will expire on August xx, 20xx.

Copyright Notice

Copyright (c) 2020 IETF Trust and the persons identified as the
document authors.  All rights reserved.

 This document is subject to BCP 78 and the IETF Trust's Legal
 Provisions Relating to IETF Documents
 (http://trustee.ietf.org/license-info) in effect on the date of
 publication of this document.  Please review these documents
 carefully, as they describe your rights and restrictions with respect
 to this document.  Code Components extracted from this document must
 include Simplified BSD License text as described in Section 4.e of
 the Trust Legal Provisions and are provided without warranty as
 described in the Simplified BSD License.

Table of Contents

 1.  Introduction . . . . . . . . . . . . . . . . . . . .. . . . . .  2
      1.1.  Terminology and Requirements Language  . . . . . . . . .  2
 2.  Energy Data Flow Management . . . . . . . . . . . . . . . . . .  3
      2.1.  Energy Data Flow . . . . . . . . . . . . . .. . . .  . .  4
      2.2.  Energy Data Format . . . . . . . . . . . . . . . . . . .  5
 3.  Proactive Energy Management for Smart City . . . . . . . .  . .  6
      3.1   Meta-Reinforcement Learning with Edge Computing . .  . .  7
      3.2.  Process flow of proactive energy management. . . . . . . .8
 4.  IANA Considerations  . . . . . . .. . . . .  . . . . . . . . . . 8
 5.  Security Considerations  . . . . . . . . . . . .  . . .  . . . . 9
 6.  References . . . . . . . . . . . . . . . . . . . . . . . . . . . 9
 6.1.  Normative References . . . . . . . . . . . . . . . . . . . . . 9
 6.2.  Informative References . . . . . .. . . . .  . . . . . . . . . 9
 Authors' Addresses . . . . . . . . . . . . . . . . . . . . .. . . . 10


Hong, et al.          Expires  August 15, 2022                  [Page 2]


Internet-Draft  Proactive energy management for smart city  October 2020

1.  Introduction

In the modern development arena, smart city, and renewable energy
are indispensable toward the ecological growth of urban technology
to enable sustainable smart services [a,b]. A microgrid is
capable to fulfill that huge amount of energy demand by enabling
the efficient demand scheduling of smart city energy consumptions.
However, the challenges come with the unpredictable nature of
both energy consumption and renewable generation, which also have
a strong relationship over the history of energy consumption and
generation [c,d].

Therefore, to overcome those challenges, a proactive energy
management is essential such that both energy consumption
and renewable energy generation can be considered. In order to
do that, meta-reinforcement learning (Meta-RL)-based [e]
energy scheduling model, in which this method is capable of
handling both energy consumption and generation with the historical
and current observations using Recurrent Neural Network (RNN).

Proactive energy management for the smart city should be solved by
distributed manner.
        . First, a local agent with an edge computing facility determines
          a local policy with respect to energy consumption and generation
          (user’s own renewable sources) for its nearby energy users’.

        . Second, by reusing the historical observations and local policy,
           the microgrid controller estimates the meta-policy for energy
           scheduling. Further, it takes necessary action based on meta-policy
           to enable sustainable energy management for the smart city.


1.1.  Terminology and Requirements Language

   The key words "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT",
   "SHOULD", "SHOULD NOT", "RECOMMENDED", "MAY", and "OPTIONAL" in this
   document are to be interpreted as described in RFC 2119 [RFC2119].

Hong, et al.          Expires  August 15, 2022                  [Page 3]


Internet-Draft  Proactive energy management for smart city  October 2020

2.   Energy Data Flow Management

The Microgrid-powered smart city includes both renewable and
non-renewable energy sources, where each individual building
has its own renewable (solar) energy sources. The city
connected with edge computing enabled wireless networks
to fulfill smart city services. Each energy user (i.e.,
home, building, school, commercial building, and so on)
is associated with its nearby edge computing server. Energy
demand, renewable generation, and required amount should send
to its associated edge server. Based on each user data, edge
server determine its own local energy management policy by
applying Deep Q-learning. The output (reward(t), action(t)
reward(t-1), action(t-1)) should send to microgrid controller.
Microgrid controller decides the overall energy management
policy for the smart city and send feed back to each user
via edge server. The communication should be through the wireless
communications protocol (LTE, 5G, LTU, Wifi, etc.) for exchanging
the energy management data.

Hong, et al.          Expires  August 15, 2022                  [Page 4]


Internet-Draft  Proactive energy management for smart city  October 2020

2.1. Energy Data Flow

Energy data flow of proactive energy management for smart
city as shown in Figure 1.
        . Energy user with renewable energy sources can send energy
        demand and generation raw data to MEC Server # in smart
        city network

        . Each energy user's observational data (reward and action)
        by executing reinforcement learning (Deep Q-learning) from
        MEC should send to microgrid controller

        . Other energy sources (except the energy sources that are
        associated with smart city user) should send to microgrid
        controller

        . Microgrid controller send the energy management executing
        command to each users’




 Figure 1: Energy data flow of proactive energy management for
           smart city


Hong, et al.          Expires  August 15, 2022                  [Page 5]


Internet-Draft  Proactive energy management for smart city  October 2020

2.2  Energy Data Format
The data format complies with tuple.
Figure 2 represent the data format of proactive energy management
scheme.



Figure 2: Energy data format of proactive energy management
                  for smart city





Hong, et al.          Expires  August 15, 2022                  [Page 6]


Internet-Draft  Proactive energy management for smart city  October 2020


3.  Proactive Energy Management for Smart City

Each edge server estimates the local policy for the associated energy
users while Microgrid controller determines the meta policy using
a little amount of information from local policy. Establishing a
strong correlation between energy generation and consumption using
Markovian properties for each energy user.


Hong, et al.          Expires  August 15, 2022                  [Page 7]


Internet-Draft  Proactive energy management for smart city  October 2020


3.1 Meta-Reinforcement Learning with Edge Computing

The proactive energy demand scheduling for smart city problem
is solved distributively, where first, obtain the local policy
by learning the local agent with respect to energy consumption
and renewable energy generation through the nearby edge server.
Second, in order to generate meta-policy, we send local policy
information to the microgrid controller alone with previous
policy observation, so that meta-agent can learn very fast
with the optimal decision. This procedure is the same for
every energy user and meta-RL model procedure is shown in
Figure 3.




Figure 3: Proactive energy management for smart city using
                  Meta-RL

Hong, et al.          Expires  August 15, 2022                  [Page 8]


Internet-Draft  Proactive energy management for smart city  October 2020


3.2.  Process flow of proactive energy management

Process flow of proactive energy management is illustrated in
Figure 4.
        . Energy generation and demand data from all users at associated
          edge server

        . Local policy estimation process for each user’s at the edge server
          using DQN and observation data by local agent send to microgrid
          controller

        . Meta energy management policy estimation using local policy at
          microgrid controller and action command send to user through
          edge server

        . Apply energy management policy to smart city users by the edge
          server service





Figure 4: Process flow of proactive energy management for smart city



Hong, et al.          Expires  August 15, 2022                  [Page 9]


Internet-Draft  Proactive energy management for smart city  October 2020

4.  IANA Considerations

There are no IANA considerations related to this document.

5.  Security Considerations

This note touches communication security as in wireless communications
protocol (LTE, 5G, LTU, Wifi, etc.).

6.  References

6.1.  Normative References

   [RFC2119]  Bradner, S., "Key words for use in RFCs to Indicate
              Requirement Levels", BCP 14, RFC 2119, March 1997.

   [a]  M. S. Munir, S. F. Abedin, M. G. R. Alam, N. H. Tran
                and C. S. Hong, “Intelligent service fulfillment
                for software defined networks in smart city,” 2018
                International Conference on Information Networking
                (ICOIN), pp. 516-521, 2018. (in Chiang Mai, Thailand).

   [b]  M. S. Munir, S. F. Abedin, M. G. R. Alam, D. H. Kim
           and C. S. Hong, " Smart Agent based Dynamic Data
           Aggregation for Delay Sensitive Smart City Services,"
           Journal of KIISE, vol. 45, no. 4, pp. 395-402, April
           2018.

   [c] Y. Zhang, M. H. Hajiesmaili, S. Cai, M. Chen and Q. Zhu,
           "Peak-Aware Online Economic Dispatching for Microgrids,"
           in IEEE Transactions on Smart Grid, vol. 9, no. 1, pp.
           323-335, Jan. 2018.

   [D] M. S. Munir, S. F. Abedin, M. G. R. Alam, D. H. Kim,
           and C. S. Hong, “RNN based Energy Demand Prediction for
           Smart-Home in Smart-Grid Framework,” Korea Software
           Congress 2017, pp. 437-439, 2017 (in Korea).

   [E] J. X. Wang et al., “Learning to reinforcement learn,”
           CogSci , 2017. (In London, UK).

6.2.  Informative References


Hong, et al.          Expires  August 15, 2022                  [Page 10]

Internet-Draft  Proactive energy management for smart city  October 2020


Authors' Addresses


Choong Seon Hong
Computer Science and Engineering Department, Kyung Hee University
Yongin, South Korea
Phone: +82 (0)31 201 2532
Email: cshong@khu.ac.kr
Md. Shirajum Munir
Computer Science and Engineering Department, Kyung Hee University
Yongin, South Korea
Phone: +82 (0)31 201 2987
Email: munir@khu.ac.kr
Ki Tae Kim
Computer Science and Engineering Department, Kyung Hee University
Yongin, South Korea
Phone: +82 (0)31 201 2532
Email: glideslope@khu.ac.kr
Seok Won Kang
Computer Science and Engineering Department, Kyung Hee University
Yongin, South Korea
Phone: +82 (0)31 201 2532
Email: dudtntdud@khu.ac.kr