arXiv subjects

Andrew Lamperski

Publications and source records attributed to Andrew Lamperski.

At least 37 records · Page 2Linked to original sources

arXiv2020

First-Order Algorithms for Constrained Nonlinear Dynamic Games

This paper presents algorithms for non-zero sum nonlinear constrained dynamic games with full information. Such problems emerge when multiple players with action constraints and differing objectives interact with the same dynamic system. They model a wide range of applications including economics, defense, and energy systems. We show how to exploit the temporal structure in projected gradient and Douglas-Rachford (DR) splitting methods. The resulting algorithms converge locally to open-loop Nash equilibria (OLNE) at linear rates. Furthermore, we extend stagewise Newton method to find a local feedback policy around an OLNE. In the of linear dynamics and polyhedral constraints, we show that this local feedback controller is an approximated feedback Nash equilibrium (FNE). Numerical examples are provided.

eess.SY↗

arXiv2020

Newton's Method and Differential Dynamic Programming for Unconstrained Nonlinear Dynamic Games

Dynamic games arise when multiple agents with differing objectives control a dynamic system. They model a wide variety of applications in economics, defense, energy systems and etc. However, compared to single-agent control problems, the computational methods for dynamic games are relatively limited. As in the single-agent case, only specific dynamic games can be solved exactly, so approximation algorithms are required. In this paper, we show how to extend a recursive Newton's algorithm and the popular differential dynamic programming (DDP) for single-agent optimal control to the case of full-information non-zero sum dynamic games. In the single-agent case, the convergence of DDP is proved by comparison with Newton's method, which converges locally at a quadratic rate. We show that the iterates of Newton's method and DDP are sufficiently close for the DDP to inherit the quadratic convergence rate of Newton's method. We also prove both methods result in an open-loop Nash equilibrium and a local feedback $O(ε^2)$-Nash equilibrium. Numerical examples are provided.

eess.SY↗

arXiv2019

Trading-Off Static and Dynamic Regret in Online Least-Squares and Beyond

Recursive least-squares algorithms often use forgetting factors as a heuristic to adapt to non-stationary data streams. The first contribution of this paper rigorously characterizes the effect of forgetting factors for a class of online Newton algorithms. For exp-concave and strongly convex objectives, the algorithms achieve the dynamic regret of $\max\{O(\log T),O(\sqrt{TV})\}$, where $V$ is a bound on the path length of the comparison sequence. In particular, we show how classic recursive least-squares with a forgetting factor achieves this dynamic regret bound. By varying $V$, we obtain a trade-off between static and dynamic regret. In order to obtain more computationally efficient algorithms, our second contribution is a novel gradient descent step size rule for strongly convex functions. Our gradient descent rule recovers the order optimal dynamic regret bounds described above. For smooth problems, we can also obtain static regret of $O(T^{1-β})$ and dynamic regret of $O(T^βV^*)$, where $β\in (0,1)$ and $V^*$ is the path length of the sequence of minimizers. By varying $β$, we obtain a trade-off between static and dynamic regret.

cs.LG↗

arXiv2019

Non-asymptotic Closed-Loop System Identification using Autoregressive Processes and Hankel Model Reduction

One of the primary challenges of system identification is determining how much data is necessary to adequately fit a model. Non-asymptotic characterizations of the performance of system identification methods provide this knowledge. Such characterizations are available for several algorithms performing open-loop identification. Often times, however, data is collected in closed-loop. Application of open-loop identification methods to closed-loop data can result in biased estimates. One method used by subspace identification techniques to eliminate these biases involves first fitting a long-horizon autoregressive model, then performing model reduction. The asymptotic behavior of such algorithms is well characterized, but the non-asymptotic behavior is not. This work provides a non-asymptotic characterization of one particular variant of these algorithms. More specifically, we provide non-asymptotic upper bounds on the generalization error of the produced model, as well as high probability bounds on the difference between the produced model and the finite horizon Kalman Filter.

eess.SY↗

arXiv2019

Simple Algorithms for Dueling Bandits

In this paper, we present simple algorithms for Dueling Bandits. We prove that the algorithms have regret bounds for time horizon T of order O(T^rho ) with 1/2 <= rho <= 3/4, which importantly do not depend on any preference gap between actions, Delta. Dueling Bandits is an important extension of the Multi-Armed Bandit problem, in which the algorithm must select two actions at a time and only receives binary feedback for the duel outcome. This is analogous to comparisons in which the rater can only provide yes/no or better/worse type responses. We compare our simple algorithms to the current state-of-the-art for Dueling Bandits, ISS and DTS, discussing complexity and regret upper bounds, and conducting experiments on synthetic data that demonstrate their regret performance, which in some cases exceeds state-of-the-art.

cs.LG↗

arXiv2019

Online Adaptive Principal Component Analysis and Its extensions

We propose algorithms for online principal component analysis (PCA) and variance minimization for adaptive settings. Previous literature has focused on upper bounding the static adversarial regret, whose comparator is the optimal fixed action in hindsight. However, static regret is not an appropriate metric when the underlying environment is changing. Instead, we adopt the adaptive regret metric from the previous literature and propose online adaptive algorithms for PCA and variance minimization, that have sub-linear adaptive regret guarantees. We demonstrate both theoretically and experimentally that the proposed algorithms can adapt to the changing environments.

cs.LG↗

arXiv2019

Online convex optimization for cumulative constraints

We propose the algorithms for online convex optimization which lead to cumulative squared constraint violations of the form $\sum\limits_{t=1}^T\big([g(x_t)]_+\big)^2=O(T^{1-β})$, where $β\in(0,1)$. Previous literature has focused on long-term constraints of the form $\sum\limits_{t=1}^Tg(x_t)$. There, strictly feasible solutions can cancel out the effects of violated constraints. In contrast, the new form heavily penalizes large constraint violations and cancellation effects cannot occur. Furthermore, useful bounds on the single step constraint violation $[g(x_t)]_+$ are derived. For convex objectives, our regret bounds generalize existing bounds, and for strongly convex objectives we give improved regret bounds. In numerical experiments, we show that our algorithm closely follows the constraint boundary leading to low cumulative violation.

cs.LG↗

arXiv2018

Differential Dynamic Programming for Nonlinear Dynamic Games

Dynamic games arise when multiple agents with differing objectives choose control inputs to a dynamic system. Dynamic games model a wide variety of applications in economics, defense, and energy systems. However, compared to single-agent control problems, the computational methods for dynamic games are relatively limited. As in the single-agent case, only very specialized dynamic games can be solved exactly, and so approximation algorithms are required. This paper extends the differential dynamic programming algorithm from single-agent control to the case of non-zero sum full-information dynamic games. The method works by computing quadratic approximations to the dynamic programming equations. The approximation results in static quadratic games which are solved recursively. Convergence is proved by showing that the algorithm iterates sufficiently close to iterates of Newton's method to inherit its convergence properties. A numerical example is provided.

math.OC↗

arXiv2018

Moment Analysis of Stochastic Hybrid Systems Using Semidefinite Programming

This paper proposes a semidefinite programming based method for estimating moments of a stochastic hybrid system (SHS). For polynomial SHSs -- which consist of polynomial continuous vector fields, reset maps, and transition intensities -- the dynamics of moments evolve according to a system of linear ordinary differential equations. However, it is generally not possible to solve the system exactly since time evolution of a specific moment may depend upon moments of order higher than it. One way to overcome this problem is to employ so-called moment closure methods that give point approximations to moments, but these are limited in that accuracy of the estimations is unknown. We find lower and upper bounds on a moment of interest via a semidefinite program that includes linear constraints obtained from moment dynamics, along with semidefinite constraints that arise from the non-negativity of moment matrices. These bounds are further shown to improve as the size of semidefinite program is increased. The key insight in the method is a reduction from stochastic hybrid systems with multiple discrete modes to a single-mode hybrid system with algebraic constraints. We further extend the scope of the proposed method to a class of non-polynomial SHSs which can be recast to polynomial SHSs via augmentation of additional states. Finally, we illustrate the applicability of results via examples of SHSs drawn from different disciplines.

math.OC↗

arXiv2017

Estimating stationary characteristic functions of stochastic systems via semidefinite programming

This paper proposes a methodology to estimate characteristic functions of stochastic differential equations that are defined over polynomials and driven by Lévy noise. For such systems, the time evolution of the characteristic function is governed by a partial differential equation; consequently, the stationary characteristic function can be obtained by solving an ordinary differential equation (ODE). However, except for a few special cases such as linear systems, the solution to the ODE consists of unknown coefficients. These coefficients are closely related with the stationary moments of the process, and bounds on these can be obtained by utilizing the fact that the characteristic function is positive definite. These bounds can be further used to find bounds on other higher order stationary moments and also estimate the stationary characteristic function itself. The method is finally illustrated via examples.

math.OC↗

arXiv2017

Approximate moment dynamics for polynomial and trigonometric stochastic systems

Stochastic dynamical systems often contain nonlinearities which make it hard to compute probability density functions or statistical moments of these systems. For the moment computations, nonlinearities in the dynamics lead to unclosed moment dynamics; in particular, the time evolution of a moment of a specific order may depend both on moments of order higher than it and on some nonlinear function of other moments. The moment closure techniques are used to find an approximate, close system of equations the moment dynamics. In this work, we extend a moment closure technique based on derivative matching that was originally proposed for polynomial stochastic systems with discrete states to continuous state stochastic systems to continuous state stochastic differential equations, with both polynomial and trigonometric nonlinearities. We validate the technique using two examples of nonlinear stochastic systems.

math.OC↗

arXiv2017

Analysis and Control of Stochastic Systems using Semidefinite Programming over Moments

This paper develops a unified methodology for probabilistic analysis and optimal control design for jump diffusion processes defined by polynomials. For such systems, the evolution of the moments of the state can be described via a system of linear ordinary differential equations. Typically, however, the moments are not closed and an infinite system of equations is required to compute statistical moments exactly. Existing methods for stochastic analysis, known as closure methods, focus on approximating this infinite system of equations with a finite dimensional system. This work develops an alternative approach in which the higher order terms, which are approximated in closure methods, are viewed as inputs to a finite-dimensional linear control system. Under this interpretation, upper and lower bounds of statistical moments can be computed via convex linear optimal control problems with semidefinite constraints. For analysis of steady-state distributions, this optimal control problem reduces to a static semidefinite program. These same optimization problems extend automatically to stochastic optimal control problems. For minimization problems, the methodology leads to guaranteed lower bounds on the true optimal value. Furthermore, we show how an approximate optimal control strategy can be constructed from the solution of the semidefinite program. The results are illustrated using numerous examples.

math.OC↗

arXiv2016

Bounds on stationary moments in stochastic chemical kinetics

In the stochastic formulation of chemical kinetics, the stationary moments of the population count of species can be described via a set of linear equations. However, except for some specific cases such as systems with linear reaction propensities, the moment equations are underdetermined as a lower order moment might depend upon a higher order moment. Here, we propose a method to find lower, and upper bounds on stationary moments of molecular counts in a chemical reaction system. The method exploits the fact that statistical moments of any positive-valued random variable must satisfy some constraints. Such constraints can be expressed as nonlinear inequalities on moments in terms of their lower order moments, and solving them in conjugation with the stationary moment equations results in bounds on the moments. Using two examples of biochemical systems, we illustrate that not only one obtains upper and lower bounds on a given stationary moment, but these bounds also improve as one uses more moment equations and utilizes the inequalities for the corresponding higher order moments. Our results provide avenues for development of moment approximations that provide explicit bounds on moment dynamics for systems whose dynamics are otherwise intractable.

q-bio.QM↗

arXiv2016

Stochastic optimal control using semidefinite programming for moment dynamics

This paper presents a method to approximately solve stochastic optimal control problems in which the cost function and the system dynamics are polynomial. For stochastic systems with polynomial dynamics, the moments of the state can be expressed as a, possibly infinite, system of deterministic linear ordinary differential equations. By casting the problem as a deterministic control problem in moment space, semidefinite programming is used to find a lower bound on the optimal solution. The constraints in the semidefinite program are imposed by the ordinary differential equations for moment dynamics and semidefiniteness of the outer product of moments. From the solution to the semidefinite program, an approximate optimal control strategy can be constructed using a least squares method. In the linear quadratic case, the method gives an exact solution to the optimal control problem. In more complex problems, an infinite number of moment differential equations would be required to compute the optimal control law. In this case, we give a procedure to increase the size of the semidefinite program, leading to increasingly accurate approximations to the true optimal control strategy.

math.OC↗

arXiv2014

Optimal Decentralized State-Feedback Control with Sparsity and Delays

This work presents the solution to a class of decentralized linear quadratic state-feedback control problems, in which the plant and controller must satisfy the same combination of delay and sparsity constraints. Using a novel decomposition of the noise history, the control problem is split into independent subproblems that are solved using dynamic programming. The approach presented herein both unifies and generalizes many existing results.

eess.SY↗

arXiv2014

The H2 Control Problem for Quadratically Invariant Systems with Delays

This paper gives a new solution to the output feedback H2 problem for quadratically invariant communication delay patterns. A characterization of all stabilizing controllers satisfying the delay constraints is given and the decentralized H2 problem is cast as a convex model matching problem. The main result shows that the model matching problem can be reduced to a finite-dimensional quadratic program. A recursive state-space method for computing the optimal controller based on vectorization is given.

eess.SY↗

arXiv2014

Stability of Asynchronous Networked Control Systems with Probabilistic Clocks

This paper studies the stability of sampled and networked control systems with sampling and communication times governed by probabilistic clocks. The clock models have few restrictions, and can be used to model numerous phenomena such as deterministic sampling, jitter, and transmission times of packet dropping networks. Moreover, the stability theory can be applied to an arbitrary number of clocks with different distributions, operating asynchronously. The paper gives Lyapunov-type sufficient conditions for stochastic stability of nonlinear networked systems. For linear systems, the paper gives necessary and sufficient conditions for exponential mean square stability, based on linear matrix inequalities. In both the linear and nonlinear cases, the Lyapunov inequalities are constructed from a simple linear combination of the classical inequalities from continuous and discrete time. Crucially, the stability theorems only depend on the mean sampling intervals. Thus, they can be applied with only limited statistical information about the clocks. The Lyapunov theorems are then applied to systems with multirate sampling, asynchronous communication, delays, and packet losses.

eess.SY↗

arXiv2014

Optimal Two Player LQR State Feedback With Varying Delay

This paper presents an explicit solution to a two player distributed LQR problem in which communication between controllers occurs across a communication link with varying delay. We extend known dynamic programming methods to accommodate this varying delay, and show that under suitable assumptions, the optimal control actions are linear in their information, and that the resulting controller has piecewise linear dynamics dictated by the current effective delay regime.

math.OC↗