arXiv subjects

Yuanyuan Shi

Publications and source records attributed to Yuanyuan Shi.

87 records · Page 5Linked to original sources

arXiv2021

Stable Online Control of Linear Time-Varying Systems

Linear time-varying (LTV) systems are widely used for modeling real-world dynamical systems due to their generality and simplicity. Providing stability guarantees for LTV systems is one of the central problems in control theory. However, existing approaches that guarantee stability typically lead to significantly sub-optimal cumulative control cost in online settings where only current or short-term system information is available. In this work, we propose an efficient online control algorithm, COvariance Constrained Online Linear Quadratic (COCO-LQ) control, that guarantees input-to-state stability for a large class of LTV systems while also minimizing the control cost. The proposed method incorporates a state covariance constraint into the semi-definite programming (SDP) formulation of the LQ optimal controller. We empirically demonstrate the performance of COCO-LQ in both synthetic experiments and a power system frequency control example.

math.OC↗

arXiv2020

Multi-Agent Reinforcement Learning in Cournot Games

In this work, we study the interaction of strategic agents in continuous action Cournot games with limited information feedback. Cournot game is the essential market model for many socio-economic systems where agents learn and compete without the full knowledge of the system or each other. We consider the dynamics of the policy gradient algorithm, which is a widely adopted continuous control reinforcement learning algorithm, in concave Cournot games. We prove the convergence of policy gradient dynamics to the Nash equilibrium when the price function is linear or the number of agents is two. This is the first result (to the best of our knowledge) on the convergence property of learning algorithms with continuous action spaces that do not fall in the no-regret class.

math.OC↗

arXiv2020

Safe Reinforcement Learning of Control-Affine Systems with Vertex Networks

This paper focuses on finding reinforcement learning policies for control systems with hard state and action constraints. Despite its success in many domains, reinforcement learning is challenging to apply to problems with hard constraints, especially if both the state variables and actions are constrained. Previous works seeking to ensure constraint satisfaction, or safety, have focused on adding a projection step to a learned policy. Yet, this approach requires solving an optimization problem at every policy execution step, which can lead to significant computational costs. To tackle this problem, this paper proposes a new approach, termed Vertex Networks (VNs), with guarantees on safety during exploration and on learned control policies by incorporating the safety constraints into the policy network architecture. Leveraging the geometric property that all points within a convex set can be represented as the convex combination of its vertices, the proposed algorithm first learns the convex combination weights and then uses these weights along with the pre-calculated vertices to output an action. The output action is guaranteed to be safe by construction. Numerical examples illustrate that the proposed VN algorithm outperforms vanilla reinforcement learning in a variety of benchmark control tasks.

cs.LG↗

arXiv2020

Input Convex Neural Networks for Optimal Voltage Regulation

The increasing penetration of renewables in distribution networks calls for faster and more advanced voltage regulation strategies. A promising approach is to formulate the problem as an optimization problem, where the optimal reactive power injection from inverters are calculated to maintain the voltages while satisfying power network constraints. However, existing optimization algorithms require the exact topology and line parameters of underlying distribution system, which are not known for most cases and are difficult to infer. In this paper, we propose to use specifically designed neural network to tackle the learning and optimization problem together. In the training stage, the proposed input convex neural network learns the mapping between the power injections and the voltages. In the voltage regulation stage, such trained network can find the optimal reactive power injections by design. We also provide a practical distributed algorithm by using the trained neural network. Theoretical bounds on the representation performance and learning efficiency of proposed model are also discussed. Numerical simulations on multiple test systems are conducted to illustrate the operation of the algorithm.

math.OC↗

arXiv2020

No-regret Learning in Cournot Games

This paper examines the convergence of no-regret learning in Cournot games with continuous actions. Cournot games are the essential model for many socio-economic systems, where players compete by strategically setting their output quantity. We assume that players do not have full information of the game and thus cannot pre-compute a Nash equilibrium. Two types of feedback are considered: one is bandit feedback and the other is gradient feedback. To study the convergence of the induced sequence of play, we introduce the notion of convergence in measure, and show that the players' actual sequence of action converges to the unique Nash equilibrium. In addition, our results naturally extend the no-regret learning algorithms' time-average regret bounds to obtain the final-iteration convergence rates. Together, our work presents significantly sharper convergence results for learning in games without strong assumptions on game property (e.g., monotonicity) and shows how exploiting the game information feedback can influence the convergence rates.

cs.GT↗

arXiv2020

Robust Reinforcement Learning for Continuous Control with Model Misspecification

We provide a framework for incorporating robustness -- to perturbations in the transition dynamics which we refer to as model misspecification -- into continuous control Reinforcement Learning (RL) algorithms. We specifically focus on incorporating robustness into a state-of-the-art continuous control RL algorithm called Maximum a-posteriori Policy Optimization (MPO). We achieve this by learning a policy that optimizes for a worst case expected return objective and derive a corresponding robust entropy-regularized Bellman contraction operator. In addition, we introduce a less conservative, soft-robust, entropy-regularized objective with a corresponding Bellman operator. We show that both, robust and soft-robust policies, outperform their non-robust counterparts in nine Mujoco domains with environment perturbations. In addition, we show improved robust performance on a high-dimensional, simulated, dexterous robotic hand. Finally, we present multiple investigative experiments that provide a deeper insight into the robustness framework. This includes an adaptation to another continuous control RL algorithm as well as learning the uncertainty set from offline data. Performance videos can be found online at https://sites.google.com/view/robust-rl.

cs.LG↗

arXiv2019

Visualization of Ion Channels in Membranes using Electrochemical Strain Microscopy

Polymer composite electrolytes of Nafion and phosphotungstic acid (PWA) are fabricated and analyzed using electrochemical strain microscopy (ESM) and conductive atomic force microscopy (C-AFM) to visualize hydrophilic ion channels near the surface, which are composed of water and sulfonic acid groups. The results indicate that the fibrillar objects in ESM image, without significant changes in topography, are hydrophilic ion channels and additional ion channels formed by interaction between PWA and sulfonic groups in Nafion. In this study, the buried ion channels lying under the surface are probed as well as the inlet and outlet of the channels on the surface through combined use of ESM and C-AFM. The results further enhance the understanding of ionic conduction in composite polymer electrolytes in various fields.

physics.app-ph↗

arXiv2019

Optimal Control Via Neural Networks: A Convex Approach

Control of complex systems involves both system identification and controller design. Deep neural networks have proven to be successful in many identification tasks, however, from model-based control perspective, these networks are difficult to work with because they are typically nonlinear and nonconvex. Therefore many systems are still identified and controlled based on simple linear models despite their poor representation capability. In this paper we bridge the gap between model accuracy and control tractability faced by neural networks, by explicitly constructing networks that are convex with respect to their inputs. We show that these input convex networks can be trained to obtain accurate models of complex physical systems. In particular, we design input convex recurrent neural networks to capture temporal behavior of dynamical systems. Then optimal controllers can be achieved via solving a convex model predictive control problem. Experiment results demonstrate the good potential of the proposed input convex neural network based approach in a variety of control applications. In particular we show that in the MuJoCo locomotion tasks, we could achieve over 10% higher performance using 5* less time compared with state-of-the-art model-based reinforcement learning method; and in the building HVAC control example, our method achieved up to 20% energy reduction compared with classic linear models.

math.OC↗

arXiv2018

Optimal Battery Control Under Cycle Aging Mechanisms in Pay for Performance Settings

We study the optimal control of battery energy storage under a general "pay-for-performance" setup such as providing frequency regulation and renewable integration. In these settings, batteries need to carefully balance the trade-off between following the instruction signals and their degradation costs in real-time. Existing battery control strategies either do not consider the uncertainty of future signals, or cannot accurately account for battery cycle aging mechanism during operation. In this work, we take a different approach to the optimal battery control problem. Instead of attacking the complexity of battery degradation function or the lack of future information one at a time, we address these two challenges together in a joint fashion. In particular, we present an electrochemically accurate and trackable battery degradation model called the rainflow cycle-based model. We prove the degradation cost is convex. Then we propose an online control policy with a simple threshold structure and show it achieve near-optimal performance with respect to an offline controller that has complete future information. We explicitly characterize the optimality gap and show it is independent to the duration of operation. Simulation results with both synthetic and real regulation traces are conducted to illustrate the theoretical results.

math.OC↗

arXiv2018

Optimal Battery Participation in Frequency Regulation Markets

Battery participants in performance-based frequency regulation markets must consider the cost of battery aging in their operating strategies to maximize market profits. In this paper we solve this problem by proposing an optimal control policy and an optimal bidding policy based on realistic market settings and an accurate battery aging model. The proposed control policy has a threshold structure and achieves near-optimal performance with respect to an offline controller that has complete future information. The proposed bidding policy considers the optimal control policy to maximize market profits while satisfying the market performance requirement through a chance-constraint. It factors the value of performance and supports a trade-off between higher profits and a lower risk of violating performance requirements. We demonstrate the optimality of both policies using simulations. A case study based on the PJM regulation market shows that our approach is effective at maximizing operating profits.

math.OC↗

arXiv2017

Modeling and Optimization of Complex Building Energy Systems with Deep Neural Networks

Modern buildings encompass complex dynamics of multiple electrical, mechanical, and control systems. One of the biggest hurdles in applying conventional model-based optimization and control methods to building energy management is the huge cost and effort of capturing diverse and temporally correlated dynamics. Here we propose an alternative approach which is model-free and data-driven. By utilizing high volume of data coming from advanced sensors, we train a deep Recurrent Neural Networks (RNN) which could accurately represent the operation's temporal dynamics of building complexes. The trained network is then directly fitted into a constrained optimization problem with finite horizons. By reformulating the constrained optimization as an unconstrained optimization problem, we use iterative gradient descents method with momentum to find optimal control inputs. Simulation results demonstrate proposed method's improved performances over model-based approach on both building system modeling and control.

math.OC↗

arXiv2017

Using Battery Storage for Peak Shaving and Frequency Regulation: Joint Optimization for Superlinear Gains

We consider using a battery storage system simultaneously for peak shaving and frequency regulation through a joint optimization framework which captures battery degradation, operational constraints and uncertainties in customer load and regulation signals. Under this framework, using real data we show the electricity bill of users can be reduced by up to 15\%. Furthermore, we demonstrate that the saving from joint optimization is often larger than the sum of the optimal savings when the battery is used for the two individual applications. A simple threshold real-time algorithm is proposed and achieves this super-linear gain. Compared to prior works that focused on using battery storage systems for single applications, our results suggest that batteries can achieve much larger economic benefits than previously thought if they jointly provide multiple services.

eess.SY↗

arXiv2017

A Convex Cycle-based Degradation Model for Battery Energy Storage Planning and Operation

A vital aspect in energy storage planning and operation is to accurately model its operational cost, which mainly comes from the battery cell degradation. Battery degradation can be viewed as a complex material fatigue process that based on stress cycles. Rainflow algorithm is a popular way for cycle identification in material fatigue process, and has been extensively used in battery degradation assessment. However, the rainflow algorithm does not have a closed form, which makes the major difficulty to include it in optimization. In this paper, we prove the rainflow cycle-based cost is convex. Convexity enables the proposed degradation model to be incorporated in different battery optimization problems and guarantees the solution quality. We provide a subgradient algorithm to solve the problem. A case study on PJM regulation market demonstrates the effectiveness of the proposed degradation model in maximizing the battery operating profits as well as extending its lifetime.

math.OC↗

arXiv2017

Optimal Regulation Response of Batteries Under Cycle Aging Mechanisms

When providing frequency regulation in a pay-for-performance market, batteries need to carefully balance the trade-off between following regulation signals and their degradation costs in real-time. Existing battery control strategies either do not consider mismatch penalties in pay-for-performance markets, or cannot accurately account for battery cycle aging mechanism during operation. This paper derives an online control policy that minimizes a battery owner's operating cost for providing frequency regulation in a pay-for-performance market. The proposed policy considers an accurate electrochemical battery cycle aging model, and is applicable to most types of battery cells. It has a threshold structure, and achieves near-optimal performance with respect to an offline controller that has complete future information. We explicitly characterize this gap and show it is independent of the duration of operation. Simulation results with both synthetic and real regulation traces are conducted to illustrate the theoretical results.

math.OC↗

arXiv2016

Leveraging energy storage to optimize data center electricity cost in emerging power markets

Energy storage in data centers has mainly been used as devices to backup generators during power outages. Recently, there has been a growing interest in using energy storage devices to actively shape power consumption in data centers to reduce their skyrocketing electricity bills. In this paper, we consider using energy storage in data centers for two applications in a joint fashion: reducing peak demand charges and enabling data centers to participate in regulation markets. We develop an optimization framework that captures the cost of electricity, degradation of energy storage devices, as well as the benefit from regulation markets. Under this frame- work, using real data Microsoft data center traces and PJM regulation signals, we show the electricity bill of a data center can be reduced by up to 20%. Furthermore, we demonstrate that the saving from joint optimization can be even larger than the sum of individually optimizing each component. We quantify the particular aspects of data center load profiles that lead to this superlinear gain. Compared to prior works that consider using energy storage devices for each single application alone, our results suggest that energy storage in data centers can have much larger impacts than previously thought possible.

cs.DC↗