arXiv subjects

Maryam Kamgarpour

Publications and source records attributed to Maryam Kamgarpour.

At least 37 records · Page 2Linked to original sources

arXiv2025

Identifying Time-varying Costs in Finite-horizon Linear Quadratic Gaussian Games

We address cost identification in a finite-horizon linear quadratic Gaussian game. We characterize the set of cost parameters that generate a given Nash equilibrium policy. We propose a backpropagation algorithm to identify the time-varying cost parameters. We derive a probabilistic error bound when the cost parameters are identified from finite trajectories. We test our method in numerical and driving simulations. Our algorithm identifies the cost parameters that can reproduce the Nash equilibrium policy and trajectory observations.

eess.SY↗

arXiv2025

On Linear Quadratic Potential Games

Our paper addresses characterizing conditions for a linear quadratic (LQ) game to be a potential game. The desired properties of potential games in finite action settings, such as convergence of learning dynamics to Nash equilibria, and the challenges of learning Nash equilibria in continuous state and action settings motivate us to characterize LQ potential games. Our first contribution is to show that the set of LQ games with full-state feedback that are potential games is very limited, essentially differing only slightly from an identical interest game. Given this finding, we restrict the class of LQ games to those with decoupled dynamics and decoupled state information structure. For this subclass, we show that the set of potential games strictly includes non-identical interest games and characterize conditions for the LQ games in this subclass to be potential. We further derive their corresponding potential function and prove the existence of a Nash equilibrium. Meanwhile, we highlight the challenges in the characterization and computation of Nash equilibrium for this class of potential LQ games.

math.OC↗

arXiv2025

A learning-based approach to stochastic optimal control under reach-avoid constraint

We develop a model-free approach to optimally control stochastic, Markovian systems subject to a reach-avoid constraint. Specifically, the state trajectory must remain within a safe set while reaching a target set within a finite time horizon. Due to the time-dependent nature of these constraints, we show that, in general, the optimal policy for this constrained stochastic control problem is non-Markovian, which increases the computational complexity. To address this challenge, we apply the state-augmentation technique from arXiv:2402.19360, reformulating the problem as a constrained Markov decision process (CMDP) on an extended state space. This transformation allows us to search for a Markovian policy, avoiding the complexity of non-Markovian policies. To learn the optimal policy without a system model, and using only trajectory data, we develop a log-barrier policy gradient approach. We prove that under suitable assumptions, the policy parameters converge to the optimal parameters, while ensuring that the system trajectories satisfy the stochastic reach-avoid constraint with high probability.

math.OC↗

arXiv2025

Bridging Finite and Infinite-Horizon Nash Equilibria in Linear Quadratic Games

Finite-horizon linear quadratic (LQ) games admit a unique Nash equilibrium, while infinite-horizon settings may have multiple. We clarify the relationship between these two cases by interpreting the finite-horizon equilibrium as a nonlinear dynamical system. Within this framework, we prove that its fixed points are exactly the infinite-horizon equilibria and that any such equilibrium can be recovered by an appropriate choice of terminal costs. We further show that periodic orbits of the dynamical system, when they arise, correspond to periodic Nash equilibria, and we provide numerical evidence of convergence to such cycles. Finally, simulations reveal three asymptotic regimes: convergence to stationary equilibria, convergence to periodic equilibria, and bounded non-convergent trajectories. These findings offer new insights and tools for tuning finite-horizon LQ games using infinite-horizon.

cs.MA↗

arXiv2025

Chance-constrained Linear Quadratic Gaussian Games for Multi-robot Interaction under Uncertainty

We address safe multi-robot interaction under uncertainty. In particular, we formulate a chance-constrained linear quadratic Gaussian game with coupling constraints and system uncertainties. We find a tractable reformulation of the game and propose a dual ascent algorithm. We prove that the algorithm converges to a feedback generalized Nash equilibrium of the reformulated game, ensuring the satisfaction of the chance constraints. We test our method in driving simulations and real-world robot experiments. Our method ensures safety under uncertainty and generates less conservative trajectories than single-agent model predictive control.

cs.RO↗

arXiv2025

Convergence Rate of Generalized Nash Equilibrium Learning in Strongly Monotone Games with Linear Constraints

We consider payoff-based learning of a generalized Nash equilibrium (GNE) in multi-agent systems. Our focus is on games with jointly convex constraints of a linear structure and strongly monotone pseudo-gradients. We present a convergent procedure based on a partial regularization technique and establish the convergence rate of its iterates under one- and two-point payoff-based feedback. To the best of our knowledge, this work is the first one characterizing the convergence speed of iterates to a variational GNE in the class of games under consideration.

math.OC↗

arXiv2025

Nash equilibria in scalar discrete-time linear quadratic games

An open problem in linear quadratic (LQ) games has been characterizing the Nash equilibria. This problem has renewed relevance given the surge of work on understanding the convergence of learning algorithms in dynamic games. This paper investigates scalar discrete-time infinite-horizon LQ games with two agents. Even in this arguably simple setting, there are no results for finding $\textit{all}$ Nash equilibria. By analyzing the best response map, we formulate a polynomial system of equations characterizing the linear feedback Nash equilibria. This enables us to bring in tools from algebraic geometry, particularly the Gröbner basis, to study the roots of this polynomial system. Consequently, we can not only compute all Nash equilibria numerically, but we can also characterize their number with explicit conditions. For instance, we prove that the LQ games under consideration admit at most three Nash equilibria. We further provide sufficient conditions for the existence of at most two Nash equilibria and sufficient conditions for the uniqueness of the Nash equilibrium. Our numerical experiments demonstrate the tightness of our bounds and showcase the increased complexity in settings with more than two agents.

cs.GT↗

arXiv2025

Chance-Constrained Trajectory Planning with Multimodal Environmental Uncertainty

We tackle safe trajectory planning under Gaussian mixture model (GMM) uncertainty. Specifically, we use a GMM to model the multimodal behaviors of obstacles' uncertain states. Then, we develop a mixed-integer conic approximation to the chance-constrained trajectory planning problem with deterministic linear systems and polyhedral obstacles. When the GMM moments are estimated via finite samples, we develop a tight concentration bound to ensure the chance constraint with a desired confidence. Moreover, to limit the amount of constraint violation, we develop a Conditional Value-at-Risk (CVaR) approach corresponding to the chance constraints and derive a tractable approximation for known and estimated GMM moments. We verify our methods with state-of-the-art trajectory prediction algorithms and autonomous driving datasets.

cs.RO↗

arXiv2025

Recursively Feasible Chance-constrained Model Predictive Control under Gaussian Mixture Model Uncertainty

We present a chance-constrained model predictive control (MPC) framework under Gaussian mixture model (GMM) uncertainty. Specifically, we consider the uncertainty that arises from predicting future behaviors of moving obstacles, which may exhibit multiple modes (for example, turning left or right). To address the multi-modal uncertainty distribution, we propose three MPC formulations: nominal chance-constrained planning, robust chance-constrained planning, and contingency planning. We prove that closed-loop trajectories generated by the three planners are safe. The approaches differ in conservativeness and performance guarantee. In particular, the robust chance-constrained planner is recursively feasible under certain assumptions on the propagation of prediction uncertainty. On the other hand, the contingency planner generates a less conservative closed-loop trajectory than the nominal planner. We validate our planners using state-of-the-art trajectory prediction algorithms in autonomous driving simulators.

eess.SY↗

arXiv2025

A safe exploration approach to constrained Markov decision processes

We consider discounted infinite-horizon constrained Markov decision processes (CMDPs), where the goal is to find an optimal policy that maximizes the expected cumulative reward while satisfying expected cumulative constraints. Motivated by the application of CMDPs in online learning for safety-critical systems, we focus on developing a model-free and \emph{simulator-free} algorithm that ensures \emph{constraint satisfaction during learning}. To this end, we employ the LB-SGD algorithm proposed in \cite{usmanova2022log}, which utilizes an interior-point approach based on the log-barrier function of the CMDP. Under the commonly assumed conditions of relaxed Fisher non-degeneracy and bounded transfer error in policy parameterization, we establish the theoretical properties of the LB-SGD algorithm. In particular, unlike existing CMDP approaches that ensure policy feasibility only upon convergence, the LB-SGD algorithm guarantees feasibility throughout the learning process and converges to the $\varepsilon$-optimal policy with a sample complexity of $\tilde{\mathcal{O}}(\varepsilon^{-6})$. Compared to the state-of-the-art policy gradient-based algorithm, C-NPG-PDA \cite{bai2022achieving2}, the LB-SGD algorithm requires an additional $\mathcal{O}(\varepsilon^{-2})$ samples to ensure policy feasibility during learning with the same Fisher non-degenerate parameterization.

cs.LG↗

arXiv2025

Convergence of a model-free entropy-regularized inverse reinforcement learning algorithm

Given a dataset of expert demonstrations, inverse reinforcement learning (IRL) aims to recover a reward for which the expert is optimal. This work proposes a model-free algorithm to solve entropy-regularized IRL problem. In particular, we employ a stochastic gradient descent update for the reward and a stochastic soft policy iteration update for the policy. Assuming access to a generative model, we prove that our algorithm is guaranteed to recover a reward for which the expert is $\varepsilon$-optimal using $\mathcal{O}(1/\varepsilon^{2})$ samples of the Markov decision process (MDP). Furthermore, with $\mathcal{O}(1/\varepsilon^{4})$ samples we prove that the optimal policy corresponding to the recovered reward is $\varepsilon$-close to the expert policy in total variation distance.

cs.LG↗

arXiv2025

Learning in Zero-Sum Markov Games: Relaxing Strong Reachability and Mixing Time Assumptions

We address payoff-based decentralized learning in infinite-horizon zero-sum Markov games. In this setting, each player makes decisions based solely on received rewards, without observing the opponent's strategy or actions nor sharing information. Prior works established finite-time convergence to an approximate Nash equilibrium under strong reachability and mixing time assumptions. We propose a convergent algorithm that significantly relaxes these assumptions, requiring only the existence of a single policy (not necessarily known) with bounded reachability and mixing time. Our key technical novelty is introducing Tsallis entropy regularization to smooth the best-response policy updates. By suitably tuning this regularization, we ensure sufficient exploration, thus bypassing previous stringent assumptions on the MDP. By establishing novel properties of the value and policy updates induced by the Tsallis entropy regularizer, we prove finite-time convergence to an approximate Nash equilibrium.

cs.GT↗

arXiv2025

Towards the Transferability of Rewards Recovered via Regularized Inverse Reinforcement Learning

Inverse reinforcement learning (IRL) aims to infer a reward from expert demonstrations, motivated by the idea that the reward, rather than the policy, is the most succinct and transferable description of a task [Ng et al., 2000]. However, the reward corresponding to an optimal policy is not unique, making it unclear if an IRL-learned reward is transferable to new transition laws in the sense that its optimal policy aligns with the optimal policy corresponding to the expert's true reward. Past work has addressed this problem only under the assumption of full access to the expert's policy, guaranteeing transferability when learning from two experts with the same reward but different transition laws that satisfy a specific rank condition [Rolland et al., 2022]. In this work, we show that the conditions developed under full access to the expert's policy cannot guarantee transferability in the more practical scenario where we have access only to demonstrations of the expert. Instead of a binary rank condition, we propose principal angles as a more refined measure of similarity and dissimilarity between transition laws. Based on this, we then establish two key results: 1) a sufficient condition for transferability to any transition laws when learning from at least two experts with sufficiently different transition laws, and 2) a sufficient condition for transferability to local changes in the transition law when learning from a single expert. Furthermore, we also provide a probably approximately correct (PAC) algorithm and an end-to-end analysis for learning transferable rewards from demonstrations of multiple experts.

cs.LG↗

arXiv2024

Convergence Rate of Payoff-based Generalized Nash Equilibrium Learning

We consider generalized Nash equilibrium (GNE) problems in games with strongly monotone pseudo-gradients and jointly linear coupling constraints. We establish the convergence rate of a payoff-based approach intended to learn a variational GNE (v-GNE) in such games. While convergent algorithms have recently been proposed in this setting given full or partial information of the gradients, rate of convergence in the payoff-based information setting has been an open problem. Leveraging properties of a game extended from the original one by a dual player, we establish a convergence rate of $O(\frac{1}{t^{4/7}})$ to a v-GNE of the game.

math.OC↗

arXiv2024

A Policy Iteration Algorithm for N-player General-Sum Linear Quadratic Dynamic Games

We present a policy iteration algorithm for the infinite-horizon N-player general-sum deterministic linear quadratic dynamic games and compare it to policy gradient methods. We demonstrate that the proposed policy iteration algorithm is distinct from the Gauss-Newton policy gradient method in the N-player game setting, in contrast to the single-player setting where under suitable choice of step size they are equivalent. We illustrate in numerical experiments that the convergence rate of the proposed policy iteration algorithm significantly surpasses that of the Gauss-Newton policy gradient method and other policy gradient variations. Furthermore, our numerical results indicate that, compared to policy gradient methods, the convergence performance of the proposed policy iteration algorithm is less sensitive to the initial policy and changes in the number of players.

math.OC↗

arXiv2024

Safe Zeroth-Order Optimization Using Quadratic Local Approximations

This paper addresses black-box smooth optimization problems, where the objective and constraint functions are not explicitly known but can be queried. The main goal of this work is to generate a sequence of feasible points converging towards a KKT primal-dual pair. Assuming to have prior knowledge on the smoothness of the unknown objective and constraints, we propose a novel zeroth-order method that iteratively computes quadratic approximations of the constraint functions, constructs local feasible sets and optimizes over them. Under some mild assumptions, we prove that this method returns an $η$-KKT pair (a property reflecting how close a primal-dual pair is to the exact KKT condition) within $O({1}/{η^{2}})$ iterations. Moreover, we numerically show that our method can achieve faster convergence compared with some state-of-the-art zeroth-order approaches. The effectiveness of the proposed approach is also illustrated by applying it to nonconvex optimization problems in optimal control and power system operation.

math.OC↗

arXiv2024

Learning to Bid in Forward Electricity Markets Using a No-Regret Algorithm

It is a common practice in the current literature of electricity markets to use game-theoretic approaches for strategic price bidding. However, they generally rely on the assumption that the strategic bidders have prior knowledge of rival bids, either perfectly or with some uncertainty. This is not necessarily a realistic assumption. This paper takes a different approach by relaxing such an assumption and exploits a no-regret learning algorithm for repeated games. In particular, by using the \emph{a posteriori} information about rivals' bids, a learner can implement a no-regret algorithm to optimize her/his decision making. Given this information, we utilize a multiplicative weight-update algorithm, adapting bidding strategies over multiple rounds of an auction to minimize her/his regret. Our numerical results show that when the proposed learning approach is used the social cost and the market-clearing prices can be higher than those corresponding to the classical game-theoretic approaches. The takeaway for market regulators is that electricity markets might be exposed to greater market power of suppliers than what classical analysis shows.

cs.GT↗

arXiv2024

Convergence Rate of Learning a Strongly Variationally Stable Equilibrium

We derive the rate of convergence to the strongly variationally stable Nash equilibrium in a convex game, for a zeroth-order learning algorithm. Though we do not assume strong monotonicity of the game, our rates for the one-point feedback and for the two-point feedback match the best known rates for strongly monotone games under zeroth-order information.

math.OC↗