Digital Twin Assisted Risk-Aware Sleep Mode Management Using Deep Q-Networks

Abstract

Base stations (BSs) are the most energy-consuming segment of mobile networks. To reduce BS energy consumption, different components of BSs can sleep when BS is not active. According to the activation/deactivation time of the BS components, multiple sleep modes (SMs) are defined in the literature. In this study, we model the problem of BS energy saving utilizing multiple sleep modes as a sequential MDP and propose an online traffic-aware deep reinforcement learning approach to maximize the long-term energy saving. However, there is a risk that BS is not sleeping at the right time and incurs large delays to the users. To tackle this issue, we propose to use a digital twin model to encapsulate the dynamics underlying the investigated system and estimate the risk of decision-making (RDM) in advance. We define a novel metric to quantify RDM and predict the performance degradation. The RDM calculated by DT is compared with a tolerable threshold set by the mobile operator. Based on this comparison, BS can decide to deactivate the SMs, re-train when needed to avoid taking high risks, or activate the SMs to benefit from energy savings. For deep reinforcement learning, we use long-short term memory (LSTM), to take into account the long and short-term dependencies in input traffic, and approximate the Q-function. We train the LSTM network using the experience replay method over a real traffic data set obtained from an operator BS in Stockholm. The data set contains data rate information with very coarse-grained time granularity. Thus, we propose a scheme to generate a new data set using the real network data set which 1) has finer-grained time granularity and 2) considers the bursty behavior of traffic data. Simulation results show that using proposed methods, considerable energy saving is obtained, compared to the baselines at cost of negligible number of delayed users.