arXiv subjects

Insoon Yang

Publications and source records attributed to Insoon Yang.

52 records · Page 3Linked to original sources

arXiv2020

Hamilton-Jacobi-Bellman Equations for Q-Learning in Continuous Time

In this paper, we introduce Hamilton-Jacobi-Bellman (HJB) equations for Q-functions in continuous time optimal control problems with Lipschitz continuous controls. The standard Q-function used in reinforcement learning is shown to be the unique viscosity solution of the HJB equation. A necessary and sufficient condition for optimality is provided using the viscosity solution framework. By using the HJB equation, we develop a Q-learning method for continuous-time dynamical systems. A DQN-like algorithm is also proposed for high-dimensional state and control spaces. The performance of the proposed Q-learning algorithm is demonstrated using 1-, 10- and 20-dimensional dynamical systems.

math.OC↗

arXiv2020

Minimax control of ambiguous linear stochastic systems using the Wasserstein metric

In this paper, we propose a minimax linear-quadratic control method to address the issue of inaccurate distribution information in practical stochastic systems. To construct a control policy that is robust against errors in an empirical distribution of uncertainty, our method is to adopt an adversary, which selects the worst-case distribution. To systematically adjust the conservativeness of our method, the opponent receives a penalty proportional to the amount, measured with the Wasserstein metric, of deviation from the empirical distribution. In the finite-horizon case, using a Riccati equation, we derive a closed-form expression of the unique optimal policy and the opponent's policy that generates the worst-case distribution. This result is then extended to the infinite-horizon setting by identifying conditions under which the Riccati recursion converges to the unique positive semi-definite solution to an associated algebraic Riccati equation (ARE). The resulting optimal policy is shown to stabilize the expected value of the system state under the worst-case distribution. We also discuss that our method can be interpreted as a distributional generalization of the $H_\infty$-method.

eess.SY↗

arXiv2020

Learning-based distributionally robust motion control with Gaussian processes

Safety is a critical issue in learning-based robotic and autonomous systems as learned information about their environments is often unreliable and inaccurate. In this paper, we propose a risk-aware motion control tool that is robust against errors in learned distributional information about obstacles moving with unknown dynamics. The salient feature of our model predictive control (MPC) method is its capability of limiting the risk of unsafety even when the true distribution deviates from the distribution estimated by Gaussian process (GP) regression, within an ambiguity set. Unfortunately, the distributionally robust MPC problem with GP is intractable because the worst-case risk constraint involves an infinite-dimensional optimization problem over the ambiguity set. To remove the infinite-dimensionality issue, we develop a systematic reformulation approach exploiting modern distributionally robust optimization techniques. The performance and utility of our method are demonstrated through simulations using a nonlinear car-like vehicle model for autonomous driving.

cs.RO↗

arXiv2020

STAR: Spatio-Temporal Prediction of Air Quality Using A Multimodal Approach

With the increase of global economic activities and high energy demand, many countries have raised concerns about air pollution. However, air quality prediction is a challenging issue due to the complex interaction of many factors. In this paper, we propose a multimodal approach for spatio-temporal air quality prediction. Our model learns the multimodal fusion of critical factors to predict future air quality levels. Based on the analyses of data, we also assessed the impacts of critical factors on air quality prediction. We conducted experiments on two real-world air pollution datasets. For Seoul dataset, our method achieved 11% and 8.2% improvement of the mean absolute error in long-term predictions of PM2.5 and PM10, respectively, compared to baselines. Our method also reduced the mean absolute error of PM2.5 predictions by 20% compared to the previous state-of-the-art results on China 1-year dataset.

eess.SP↗

arXiv2020

Safe reinforcement learning for probabilistic reachability and safety specifications: A Lyapunov-based approach

Emerging applications in robotics and autonomous systems, such as autonomous driving and robotic surgery, often involve critical safety constraints that must be satisfied even when information about system models is limited. In this regard, we propose a model-free safety specification method that learns the maximal probability of safe operation by carefully combining probabilistic reachability analysis and safe reinforcement learning (RL). Our approach constructs a Lyapunov function with respect to a safe policy to restrain each policy improvement stage. As a result, it yields a sequence of safe policies that determine the range of safe operation, called the safe set, which monotonically expands and gradually converges. We also develop an efficient safe exploration scheme that accelerates the process of identifying the safety of unexamined states. Exploiting the Lyapunov shielding, our method regulates the exploratory policy to avoid dangerous states with high confidence. To handle high-dimensional systems, we further extend our approach to deep RL by introducing a Lagrangian relaxation technique to establish a tractable actor-critic algorithm. The empirical performance of our method is demonstrated through continuous control benchmark problems, such as a reaching task on a planar robot arm.

cs.RO↗

arXiv2020

Wasserstein Distributionally Robust Motion Control for Collision Avoidance Using Conditional Value-at-Risk

In this paper, a risk-aware motion control scheme is considered for mobile robots to avoid randomly moving obstacles when the true probability distribution of uncertainty is unknown. We propose a novel model predictive control (MPC) method for limiting the risk of unsafety even when the true distribution of the obstacles' movements deviates, within an ambiguity set, from the empirical distribution obtained using a limited amount of sample data. By choosing the ambiguity set as a statistical ball with its radius measured by the Wasserstein metric, we achieve a probabilistic guarantee of the out-of-sample risk, evaluated using new sample data generated independently of the training data. To resolve the infinite-dimensionality issue inherent in the distributionally robust MPC problem, we reformulate it as a finite-dimensional nonlinear program using modern distributionally robust optimization techniques based on the Kantorovich duality principle. To find a globally optimal solution in the case of affine dynamics and output equations, a spatial branch-and-bound algorithm is designed using McCormick relaxation. The performance of the proposed method is demonstrated and analyzed through simulation studies using a nonlinear car-like vehicle model and a linearized quadrotor model.

cs.RO↗

arXiv2018

Safety-Aware Optimal Control of Stochastic Systems Using Conditional Value-at-Risk

In this paper, we consider a multi-objective control problem for stochastic systems that seeks to minimize a cost of interest while ensuring safety. We introduce a novel measure of safety risk using the conditional value-at-risk and a set distance to formulate a safety risk-constrained optimal control problem. Our reformulation method using an extremal representation of the safety risk measure provides a computationally tractable dynamic programming solution. A useful byproduct of the proposed solution is the notion of a risk-constrained safe set, which is a new stochastic safety verification tool. We also establish useful connections between the risk-constrained safe sets and the popular probabilistic safe sets. The tradeoff between the risk tolerance and the mean performance of our controller is examined through an inventory control problem.

math.OC↗

arXiv2017

Online Combinatorial Optimization for Interconnected Refrigeration Systems: Linear Approximation and Submodularity

Commercial refrigeration systems consume 7% of the total commercial energy consumption in the United States. Improving their energy efficiency contributes to the sustainability of global energy systems and the supermarket business sector. This paper proposes a new control method that can save the energy consumption of multi-case supermarket refrigerators by explicitly taking into account their interconnected and switched system dynamics. Its novelty is a bilevel combinatorial optimization formulation to generate ON/OFF control actions for expansion valves and compressors. The inner optimization module keeps display case temperatures in a desirable range and the outer optimization module minimizes energy consumption. In addition to its energy-saving capability, the proposed controller significantly reduces the frequency of compressor switchings by employing a conservative compressor control strategy. However, solving this bilevel optimization problem associated with interconnected and switched systems is a computationally challenging task. To solve the problem in near real time, we propose two approximation algorithms that can solve both the inner and outer optimization problems at once. The first algorithm uses a linear approximation, and the second is based on the submodular structure of the optimization problem. Both are (polynomial-time) scalable algorithms and generate near-optimal solutions with performance guarantees. Our work complements existing optimization-based control methods (e.g., MPC) for supermarket refrigerators, as our algorithms can be adopted as a tool for solving combinatorial optimization problems arising in these methods.

eess.SY↗

arXiv2017

A dynamic game approach to distributionally robust safety specifications for stochastic systems

This paper presents a new safety specification method that is robust against errors in the probability distribution of disturbances. Our proposed distributionally robust safe policy maximizes the probability of a system remaining in a desired set for all times, subject to the worst possible disturbance distribution in an ambiguity set. We propose a dynamic game formulation of constructing such policies and identify conditions under which a non-randomized Markov policy is optimal. Based on this existence result, we develop a practical design approach to safety-oriented stochastic controllers with limited information about disturbance distributions. This control method can be used to minimize another cost function while ensuring safety in a probabilistic way. However, an associated Bellman equation involves infinite-dimensional minimax optimization problems since the disturbance distribution may have a continuous density. To resolve computational issues, we propose a duality-based reformulation method that converts the infinite-dimensional minimax problem into a semi-infinite program that can be solved using existing convergent algorithms. We prove that there is no duality gap, and that this approach thus preserves optimality. The results of numerical tests confirm that the proposed method is robust against distributional errors in disturbances, while a standard stochastic safety specification tool is not.

math.OC↗

arXiv2017

Optimal Control of Conditional Value-at-Risk in Continuous Time

We consider continuous-time stochastic optimal control problems featuring Conditional Value-at-Risk (CVaR) in the objective. The major difficulty in these problems arises from time-inconsistency, which prevents us from directly using dynamic programming. To resolve this challenge, we convert to an equivalent bilevel optimization problem in which the inner optimization problem is standard stochastic control. Furthermore, we provide conditions under which the outer objective function is convex and differentiable. We compute the outer objective's value via a Hamilton-Jacobi-Bellman equation and its gradient via the viscosity solution of a linear parabolic equation, which allows us to perform gradient descent. The significance of this result is that we provide an efficient dynamic programming-based algorithm for optimal control of CVaR without lifting the state-space. To broaden the applicability of the proposed algorithm, we propose convergent approximation schemes in cases where our key assumptions do not hold and characterize relevant suboptimality bounds. In addition, we extend our method to a more general class of risk metrics, which includes mean-variance and median-deviation. We also demonstrate a concrete application to portfolio optimization under CVaR constraints. Our results contribute an efficient framework for solving time-inconsistent CVaR-based sequential optimization.

math.OC↗

arXiv2015

Optimal Dynamic Contracts for a Large-Scale Principal-Agent Hierarchy: A Concavity-Preserving Approach

We present a continuous-time contract whereby a top-level player can incentivize a hierarchy of players below him to act in his best interest despite only observing the output of his direct subordinate. This paper extends Sannikov's approach from a situation of asymmetric information between a principal and an agent to one of hierarchical information between several players. We develop an iterative algorithm for constructing an incentive compatible contract and define the correct notion of concavity which must be preserved during iteration. We identify conditions under which a dynamic programming construction of an optimal dynamic contract can be reduced to only a one-dimensional state space and one-dimensional control set, independent of the size of the hierarchy. In this sense, our results contribute to the applicability of dynamic programming on dynamic contracts for a large-scale principal-agent hierarchy.

math.OC↗

arXiv2015

Approximation Algorithms for Optimization of Combinatorial Dynamical Systems

This paper considers an optimization problem for a dynamical system whose evolution depends on a collection of binary decision variables. We develop scalable approximation algorithms with provable suboptimality bounds to provide computationally tractable solution methods even when the dimension of the system and the number of the binary variables are large. The proposed method employs a linear approximation of the objective function such that the approximate problem is defined over the feasible space of the binary decision variables, which is a discrete set. To define such a linear approximation, we propose two different variation methods: one uses continuous relaxation of the discrete space and the other uses convex combinations of the vector field and running payoff. The approximate problem is a 0-1 linear program, which can be solved by existing polynomial-time exact or approximation algorithms, and does not require the solution of the dynamical system. Furthermore, we characterize a sufficient condition ensuring the approximate solution has a provable suboptimality bound. We show that this condition can be interpreted as the concavity of the objective function. The performance and utility of the proposed algorithms are demonstrated with the ON/OFF control problems of interdependent refrigeration systems.

math.OC↗

arXiv2014

Risk-Limiting Dynamic Contracts for Direct Load Control

This paper proposes a novel continuous-time dynamic contract framework that has a risk-limiting capability. If a principal and an agent enter into such a contract, the principal can optimally manage its performance and risk with a guarantee that the agent's risk is less than or equal to a pre-specified level and that the agent's expected payoff is greater than or equal to another pre-specified threshold. We achieve such risk-management capabilities by formulating the contract design problem as mean-variance constrained risk-sensitive control. A dynamic programming-based method is developed to solve the problem. The key idea of our proposed solution method is to reformulate the inequality constraints on the mean and the variance of the agent's payoff as dynamical system constraints by introducing new state and control variables. The reformulations use the martingale representation theorem. The proposed contract method enables us to develop a new direct load control method that provides the load-serving entity with financial risk management solutions in real-time electricity markets. We also propose an approximate decomposition of the optimal contract design problem for multiple customers into multiple low-dimensional contract problems for one customer. This allows the direct load control program to work with a large number of customers without any scalability issues. Furthermore, the contract design procedure can be completely parallelized. The performance and usefulness of the proposed contract method and its application to direct load control are demonstrated using data on the electric energy consumption of customers in Austin, Texas as well as the Electricity Reliability Council of Texas' locational marginal price data.

math.OC↗

arXiv2014

Path integral formulation of stochastic optimal control with generalized costs

Path integral control solves a class of stochastic optimal control problems with a Monte Carlo (MC) method for an associated Hamilton-Jacobi-Bellman (HJB) equation. The MC approach avoids the need for a global grid of the domain of the HJB equation and, therefore, path integral control is in principle applicable to control problems of moderate to large dimension. The class of problems path integral control can solve, however, is defined by requirements on the cost function, the noise covariance matrix and the control input matrix. We relax the requirements on the cost function by introducing a new state that represents an augmented running cost. In our new formulation the cost function can contain stochastic integral terms and linear control costs, which are important in applications in engineering, economics and finance. We find an efficient numerical implementation of our grid-free MC approach and demonstrate its performance and usefulness in examples from hierarchical electric load management. The dimension of one of our examples is large enough to make classical grid-based HJB solvers impractical.

math.OC↗

arXiv2014

Dynamic Contracts with Partial Observations: Application to Indirect Load Control

This paper proposes a method to design an optimal dynamic contract between a principal and an agent, who has the authority to control both the principal's revenue and an engineered system. The key characteristic of our problem setting is that the principal has very limited information: the principal has no capability to monitor the agent's control or the state of the engineered system. The agent has perfect observations. With this asymmetry of information, we show that the principal can induce the agent to control both the revenue and the system processes in a way that maximizes the principal's utility, if the principal offers appropriate real-time and end-time compensation. We reformulate the dynamic contract design problem as a stochastic optimal control of both the engineered system and the agent's future expected payoff, which can be numerically solved using an associated Hamilton-Jacobi-Bellman equation. The performance and usefulness of the proposed contract are demonstrated with an indirect load control problem.

math.OC↗

arXiv2013

Identification of Parameters and Initial Values for Reaction-Diffusion Systems in Protein Networks (Extended Version)

Spatio-temporal biochemical signaling in a large class of protein-protein interaction networks is well modeled by a reaction-diffusion system. The global existence of the solution to the reaction-diffusion system is determined by the reaction kinetics model and the protein network topology. We propose a novel reaction kinetics model that guarantees that the reaction-diffusion system with this model has a nonnegative invariant global classical solution for any network topology. We then present a computational method to identify the unknown parameters and initial values for a reaction-diffusion system with this reaction kinetics model. The identification approach solves an optimization problem that minimizes the cost function defined as the $L^2$-norm of the difference between the data and the solution of the reaction-diffusion system. We utilize an adjoint-based optimal control method to obtain the gradients of the cost function with respect to the parameters and initial values. The regularity of the global classical solutions of the reaction-diffusion system and its corresponding adjoint system avoids situations in which the gradients blow up, and therefore guarantees the success of the identification method for any network structure. Utilizing this gradient information, an efficient algorithm to solve the optimization problem is proposed and applied to estimate the mass diffusivities, rate constants and initial values of a reaction-diffusion system that models protein-protein interactions in a signaling network that regulates the actin cytoskeleton in a malignant breast cell.

math.OC↗