arXiv subjects

Insoon Yang

Publications and source records attributed to Insoon Yang.

At least 37 records · Page 2Linked to original sources

arXiv2022

Using affine policies to reformulate two-stage Wasserstein distributionally robust linear programs to be independent of sample size

Intensively studied in theory as a promising data-driven tool for decision-making under ambiguity, two-stage distributionally robust optimization (DRO) problems over Wasserstein balls are not necessarily easy to solve in practice. This is partly due to large sample size. In this article, we study a generic two-stage distributionally robust linear program (2-DRLP) over a 1-Wasserstein ball using an affine policy. The 2-DRLP has right-hand-side uncertainty with a rectangular support. Our main contribution is to show that the 2-DRLP problem has a tractable reformulation with a scale independent of sample size. The reformulated problem can be solved within a pre-defined optimality tolerance using robust optimization techniques. To reduce the inevitable conservativeness of the affine policy while preserving independence of sample size, we further develop a method for constructing an uncertainty set with a probabilistic guarantee over which the Wasserstein ball is re-defined. As an application, we present a novel unit commitment model for power systems under uncertainty of renewable energy generation to examine the effectiveness of the proposed 2-DRLP technique. Extensive numerical experiments demonstrate that our model leads to better out-of-sample performance on average than other state-of-the-art distributionally robust unit commitment models while staying computationally competent.

math.OC↗

arXiv2022

Wasserstein Distributionally Robust Control of Partially Observable Linear Stochastic Systems

Distributionally robust control (DRC) aims to effectively manage distributional ambiguity in stochastic systems. While most existing works address inaccurate distributional information in fully observable settings, we consider a partially observable DRC problem for discrete-time linear systems using the Wasserstein metric. For a tractable solution, we propose a novel approximation method exploiting the Gelbrich bound of Wasserstein distance. Using techniques from modern distributionally robust optimization, we derive a closed-form expression for the optimal control policy and a tractable semidefinite programming problem for the worst-case distribution policy in both finite-horizon and infinite-horizon average-cost settings. The proposed method features several salient theoretical properties, such as a guaranteed cost property and a probabilistic out-of-sample performance guarantee, demonstrating the distributional robustness of our controller. Furthermore, the resulting controller is shown to ensure the closed-loop stability of the mean-state system. The empirical performance of our method is tested through numerical experiments on a power system frequency control problem.

eess.SY↗

arXiv2022

Anderson Acceleration for Partially Observable Markov Decision Processes: A Maximum Entropy Approach

Partially observable Markov decision processes (POMDPs) is a rich mathematical framework that embraces a large class of complex sequential decision-making problems under uncertainty with limited observations. However, the complexity of POMDPs poses various computational challenges, motivating the need for an efficient algorithm that rapidly finds a good enough suboptimal solution. In this paper, we propose a novel accelerated offline POMDP algorithm exploiting Anderson acceleration (AA) that is capable of efficiently solving fixed-point problems using previous solution estimates. Our algorithm is based on the Q-function approximation (QMDP) method to alleviate the scalability issue inherent in POMDPs. Inspired by the quasi-Newton interpretation of AA, we propose a maximum entropy variant of QMDP, which we call soft QMDP, to fully benefit from AA. We prove that the overall algorithm converges to the suboptimal solution obtained by soft QMDP. Our algorithm can also be implemented in a model-free manner using simulation data. Provable error bounds on the residual and the solution are provided to examine how the simulation errors are propagated through the proposed algorithm. Finally, the performance of our algorithm is tested on several benchmark problems. According to the results of our experiments, the proposed algorithm converges significantly faster without degrading the solution quality compared to its standard counterparts.

eess.SY↗

arXiv2022

Wasserstein Distributionally Robust Control of Partially Observable Linear Systems: Tractable Approximation and Performance Guarantee

Wasserstein distributionally robust control (WDRC) is an effective method for addressing inaccurate distribution information about disturbances in stochastic systems. It provides various salient features, such as an out-of-sample performance guarantee, while most of the existing methods use full-state observations. In this paper, we develop a computationally tractable WDRC method for discrete-time partially observable linear-quadratic (LQ) control problems. The key idea is to reformulate the WDRC problem as a novel minimax control problem with an approximate Wasserstein penalty. We derive a closed-form expression of the optimal control policy of the approximate problem using a nontrivial Riccati equation. We further show the guaranteed cost property of the resulting controller and identify a provable bound for the optimality gap. Finally, we evaluate the performance of our method through numerical experiments using both Gaussian and non-Gaussian disturbances.

eess.SY↗

arXiv2022

On Affine Policies for Wasserstein Distributionally Robust Unit Commitment

This paper proposes a unit commitment (UC) model based on data-driven Wasserstein distributionally robust optimization (WDRO) for power systems under uncertainty of renewable generation as well as its tractable exact reformulation. The proposed model is formulated as a WDRO problem relying on an affine policy, which nests an infinite-dimensional worst-case expectation problem and satisfies the non-anticipativity constraint. To reduce conservativeness, we develop a novel technique that defines a subset of the uncertainty set with a probabilistic guarantee. Subsequently, the proposed model is recast as a semi-infinite programming problem that can be efficiently solved using existing algorithms. Notably, the scale of this reformulation is invariant with the sample size. As a result, a number of samples are easily incorporated without using sophisticated decomposition algorithms. Numerical simulations on 6- and 24-bus test systems demonstrate the economic and computational efficiency of the proposed model.

eess.SY↗

arXiv2022

Infusing model predictive control into meta-reinforcement learning for mobile robots in dynamic environments

The successful operation of mobile robots requires them to adapt rapidly to environmental changes. To develop an adaptive decision-making tool for mobile robots, we propose a novel algorithm that combines meta-reinforcement learning (meta-RL) with model predictive control (MPC). Our method employs an off-policy meta-RL algorithm as a baseline to train a policy using transition samples generated by MPC when the robot detects certain events that can be effectively handled by MPC, with its explicit use of robot dynamics. The key idea of our method is to switch between the meta-learned policy and the MPC controller in a randomized and event-triggered fashion to make up for suboptimal MPC actions caused by the limited prediction horizon. During meta-testing, the MPC module is deactivated to significantly reduce computation time in motion control. We further propose an online adaptation scheme that enables the robot to infer and adapt to a new task within a single trajectory. The performance of our method has been demonstrated through simulations using a nonlinear car-like vehicle model with (i) synthetic movements of obstacles, and (ii) real-world pedestrian motion data. The simulation results indicate that our method outperforms other algorithms in terms of learning efficiency and navigation quality.

cs.RO↗

arXiv2022

Risk-sensitive safety analysis using Conditional Value-at-Risk

This paper develops a safety analysis method for stochastic systems that is sensitive to the possibility and severity of rare harmful outcomes. We define risk-sensitive safe sets as sub-level sets of the solution to a non-standard optimal control problem, where a random maximum cost is assessed via Conditional Value-at-Risk (CVaR). The objective function represents the maximum extent of constraint violation of the state trajectory, averaged over a given percentage of worst cases. This problem is well-motivated but difficult to solve tractably because the temporal decomposition for CVaR is history-dependent. Our primary theoretical contribution is to derive computationally tractable under-approximations to risk-sensitive safe sets. Our method provides a novel, theoretically guaranteed, parameter-dependent upper bound to the CVaR of a maximum cost without the need to augment the state space. For a fixed parameter value, the solution to only one Markov decision process problem is required to obtain the under-approximations for any family of risk-sensitivity levels. In addition, we propose a second definition for risk-sensitive safe sets and provide a tractable method for their estimation without using a parameter-dependent upper bound. The second definition is expressed in terms of a new coherent risk functional, which is inspired by CVaR. We demonstrate our primary theoretical contribution via numerical examples.

eess.SY↗

arXiv2022

On representation formulas for optimal control: A Lagrangian perspective

In this paper, we study representation formulas for finite-horizon optimal control problems with or without state constraints, unifying two different viewpoints: the Lagrangian and dynamic programming (DP) frameworks. In a recent work [1], the generalized Lax formula is obtained via DP for optimal control problems with state constraints and nonlinear systems. We revisit the formula from the Lagrangian perspective to provide a unified framework for understanding and implementing the nontrivial representation of the value function. Our simple derivation makes direct use of the Lagrangian formula from the theory of Hamilton-Jacobi (HJ) equations. We also discuss a rigorous way to construct an optimal control using a $δ$-net, as well as a numerical scheme for controller synthesis via convex optimization.

math.OC↗

arXiv2021

Training Wasserstein GANs without gradient penalties

We propose a stable method to train Wasserstein generative adversarial networks. In order to enhance stability, we consider two objective functions using the $c$-transform based on Kantorovich duality which arises in the theory of optimal transport. We experimentally show that this algorithm can effectively enforce the Lipschitz constraint on the discriminator while other standard methods fail to do so. As a consequence, our method yields an accurate estimation for the optimal discriminator and also for the Wasserstein distance between the true distribution and the generated one. Our method requires no gradient penalties nor corresponding hyperparameter tuning and is computationally more efficient than other methods. At the same time, it yields competitive generators of synthetic images based on the MNIST, F-MNIST, and CIFAR-10 datasets.

cs.LG↗

arXiv2021

Distributionally robust risk map for learning-based motion planning and control: A semidefinite programming approach

This paper proposes a novel safety specification tool, called the distributionally robust risk map (DR-risk map), for a mobile robot operating in a learning-enabled environment. Given the robot's position, the map aims to reliably assess the conditional value-at-risk (CVaR) of collision with obstacles whose movements are inferred by Gaussian process regression (GPR). Unfortunately, the inferred distribution is subject to errors, making it difficult to accurately evaluate the CVaR of collision. To overcome this challenge, this tool measures the risk under the worst-case distribution in a so-called ambiguity set that characterizes allowable distribution errors. To resolve the infinite-dimensionality issue inherent in the construction of the DR-risk map, we derive a tractable semidefinite programming formulation that provides an upper bound of the risk, exploiting techniques from modern distributionally robust optimization. As a concrete application for motion planning, a distributionally robust RRT* algorithm is considered using the risk map that addresses distribution errors caused by GPR. Furthermore, a motion control method is devised using the DR-risk map in a learning-based model predictive control (MPC) formulation. In particular, a neural network approximation of the risk map is proposed to reduce the computational cost in solving the MPC problem. The performance and utility of the proposed risk map are demonstrated through simulation studies that show its ability to ensure the safety of mobile robots despite learning errors.

cs.RO↗

arXiv2021

On Anderson acceleration for partially observable Markov decision processes

This paper proposes an accelerated method for approximately solving partially observable Markov decision process (POMDP) problems offline. Our method carefully combines two existing tools: Anderson acceleration (AA) and the fast informed bound (FIB) method. Adopting AA, our method rapidly solves an approximate Bellman equation with an efficient combination of previous solution estimates. Furthermore, the use of FIB alleviates the scalability issue inherent in POMDPs. We show the convergence of the overall algorithm to the suboptimal solution obtained by FIB. We further consider a simulation-based method and prove that the approximation error is bounded explicitly. The performance of our algorithm is evaluated on several benchmark problems. The results of our experiments demonstrate that the proposed algorithm converges significantly faster without degrading the quality of the solution compared to its standard counterpart.

eess.SY↗

arXiv2021

Distributional robustness in minimax linear quadratic control with Wasserstein distance

To address the issue of inaccurate distributions in practical stochastic systems, a minimax linear-quadratic control method is proposed using the Wasserstein metric. Our method aims to construct a control policy that is robust against errors in an empirical distribution of underlying uncertainty, by adopting an adversary that selects the worst-case distribution. The opponent receives a Wasserstein penalty proportional to the amount of deviation from the empirical distribution. A closed-form expression of the finite-horizon optimal policy pair is derived using a Riccati equation. The result is then extended to the infinite-horizon average cost setting by identifying conditions under which the Riccati recursion converges to the unique positive semi-definite solution to an algebraic Riccati equation. Our method is shown to possess several salient features including closed-loop stability, and an out-of-sample performance guarantee. We also discuss how to optimize the penalty parameter for enhancing the distributional robustness of our control policy. Last but not least, a theoretical connection to the classical $H_\infty$-method is identified from the perspective of distributional robustness.

eess.SY↗

arXiv2020

Wasserstein Distributionally Robust Stochastic Control: A Data-Driven Approach

Standard stochastic control methods assume that the probability distribution of uncertain variables is available. Unfortunately, in practice, obtaining accurate distribution information is a challenging task. To resolve this issue, we investigate the problem of designing a control policy that is robust against errors in the empirical distribution obtained from data. This problem can be formulated as a two-player zero-sum dynamic game problem, where the action space of the adversarial player is a Wasserstein ball centered at the empirical distribution. We propose computationally tractable value and policy iteration algorithms with explicit estimates of the number of iterations required for constructing an $ε$-optimal policy. We show that the contraction property of associated Bellman operators extends a single-stage out-of-sample performance guarantee, obtained using a measure concentration inequality, to the corresponding multi-stage guarantee without any degradation in the confidence level. In addition, we characterize an explicit form of the optimal distributionally robust control policy and the worst-case distribution policy for linear-quadratic problems with Wasserstein penalty. Our study indicates that dynamic programming and Kantorovich duality play a critical role in solving and analyzing the Wasserstein distributionally robust stochastic control problems.

math.OC↗

arXiv2020

Hamilton-Jacobi Deep Q-Learning for Deterministic Continuous-Time Systems with Lipschitz Continuous Controls

In this paper, we propose Q-learning algorithms for continuous-time deterministic optimal control problems with Lipschitz continuous controls. Our method is based on a new class of Hamilton-Jacobi-Bellman (HJB) equations derived from applying the dynamic programming principle to continuous-time Q-functions. A novel semi-discrete version of the HJB equation is proposed to design a Q-learning algorithm that uses data collected in discrete time without discretizing or approximating the system dynamics. We identify the condition under which the Q-function estimated by this algorithm converges to the optimal Q-function. For practical implementation, we propose the Hamilton-Jacobi DQN, which extends the idea of deep Q-networks (DQN) to our continuous control setting. This approach does not require actor networks or numerical solutions to optimization problems for greedy actions since the HJB equation provides a simple characterization of optimal controls via ordinary differential equations. We empirically demonstrate the performance of our method through benchmark tasks and high-dimensional linear-quadratic problems.

cs.LG↗

arXiv2020

Hamilton-Jacobi-Bellman Equations for Maximum Entropy Optimal Control

Maximum entropy reinforcement learning (RL) methods have been successfully applied to a range of challenging sequential decision-making and control tasks. However, most of existing techniques are designed for discrete-time systems. As a first step toward their extension to continuous-time systems, this paper considers continuous-time deterministic optimal control problems with entropy regularization. Applying the dynamic programming principle, we derive a novel class of Hamilton-Jacobi-Bellman (HJB) equations and prove that the optimal value function of the maximum entropy control problem corresponds to the unique viscosity solution of the HJB equation. Our maximum entropy formulation is shown to enhance the regularity of the viscosity solution and to be asymptotically consistent as the effect of entropy regularization diminishes. A salient feature of the HJB equations is computational tractability. Generalized Hopf-Lax formulas can be used to solve the HJB equations in a tractable grid-free manner without the need for numerically optimizing the Hamiltonian. We further show that the optimal control is uniquely characterized as Gaussian in the case of control affine systems and that, for linear-quadratic problems, the HJB equation is reduced to a Riccati equation, which can be used to obtain an explicit expression of the optimal control. Lastly, we discuss how to extend our results to continuous-time model-free RL by taking an adaptive dynamic programming approach. To our knowledge, the resulting algorithms are the first data-driven control methods that use an information theoretic exploration mechanism in continuous time.

math.OC↗

arXiv2020

A Convex Optimization Approach to Dynamic Programming in Continuous State and Action Spaces

In this paper, a convex optimization-based method is proposed for numerically solving dynamic programs in continuous state and action spaces. The key idea is to approximate the output of the Bellman operator at a particular state by the optimal value of a convex program. The approximate Bellman operator has a computational advantage because it involves a convex optimization problem in the case of control-affine systems and convex costs. Using this feature, we propose a simple dynamic programming algorithm to evaluate the approximate value function at pre-specified grid points by solving convex optimization problems in each iteration. We show that the proposed method approximates the optimal value function with a uniform convergence property in the case of convex optimal value functions. We also propose an interpolation-free design method for a control policy, of which performance converges uniformly to the optimum as the grid resolution becomes finer. When a nonlinear control-affine system is considered, the convex optimization approach provides an approximate policy with a provable suboptimality bound. For general cases, the proposed convex formulation of dynamic programming operators can be modified as a nonconvex bi-level program, in which the inner problem is a linear program, without losing uniform convergence properties.

math.OC↗

arXiv2020

Risk-sensitive safety specifications for stochastic systems using Conditional Value-at-Risk

This paper proposes a safety analysis method that facilitates a tunable balance between the worst-case and risk-neutral perspectives. First, we define a risk-sensitive safe set to specify the degree of safety attained by a stochastic system. This set is defined as a sublevel set of the solution to an optimal control problem that is expressed using the Conditional Value-at-Risk (CVaR) measure. This problem does not satisfy Bellman's Principle, thus our next contribution is to show how risk-sensitive safe sets can be under-approximated by the solution to a CVaR-Markov Decision Process. We adopt an existing value iteration algorithm to find an approximate solution to the reduced problem for a class of linear systems. Then, we develop a realistic numerical example of a stormwater system to show that this approach can be applied to non-linear systems. Finally, we compare the CVaR criterion to the exponential disutility criterion. The latter allocates control effort evenly across the cost distribution to reduce variance, while the CVaR criterion focuses control effort on a given worst-case quantile--where it matters most for safety.

eess.SY↗

arXiv2020

Multi-Objective Predictive Taxi Dispatch via Network Flow Optimization

In this paper, we discuss a large-scale fleet management problem in a multi-objective setting. We aim to seek a receding horizon taxi dispatch solution that serves as many ride requests as possible while minimizing the cost of relocating vehicles. To obtain the desired solution, we first convert the multi-objective taxi dispatch problem into a network flow problem, which can be solved using the classical minimum cost maximum flow (MCMF) algorithm. We show that a solution obtained using the MCMF algorithm is integer-valued; thus, it does not require any additional rounding procedure that may introduce undesirable numerical errors. Furthermore, we prove the time-greedy property of the proposed solution, which justifies the use of receding horizon optimization. For computational efficiency, we propose a linear programming method to obtain an optimal solution in near real time. The results of our simulation studies using real-world data for the metropolitan area of Seoul, South Korea indicate that the performance of the proposed predictive method is almost as good as that of the oracle that foresees the future.

eess.SY↗