arXiv subjects

Maryam Kamgarpour

Publications and source records attributed to Maryam Kamgarpour.

At least 19 recordsLinked to original sources

arXiv2026

Finite-Time Guarantees for Constrained Reinforcement Learning under Partial Observability

We study reinforcement learning in constrained partially observable Markov decision processes (CPOMDPs), where an agent maximizes reward subject to a cost constraint using only incomplete information about the underlying state. The lack of state observability may require optimal policies to depend on arbitrarily long observation-action histories, making planning and learning challenging. We leverage recent advances in finite-memory approximations of POMDPs and represent finite history windows as states of a constrained MDP. We then develop a primal--dual algorithm that provides the first non-asymptotic guarantees that jointly bound average reward suboptimality and constraint violation in the original CPOMDP. A key challenge is that this finite-window approximation affects both reward values and the set of feasible policies. Under a filter-stability assumption, we derive bounds relating optimality and feasibility in the finite-state model to those in the original CPOMDP, with approximation errors that decay geometrically with the window length. Experiments in partially observable gridworlds demonstrate the effectiveness of our approach.

cs.LG↗

arXiv2026

Selling Information to a Competitor

Selling information to a competitor can generate revenue but may also help the rival choose better actions in the market they compete in. We study this trade-off when the buyer's demand for supplemental information depends on its own private information. The buyer selects from a menu of information products offered by the seller, who incurs a competitive loss proportional to the increase in the buyer's probability of choosing the correct downstream action. In a common-prior model with binary states and actions, we characterize profit-maximizing menus for two buyer types and for a continuum of types. We show that selling information can remain profitable despite the competitive loss, but withholding is optimal when this loss exceeds the buyer's valuation of the information. Stronger competition weakly reduces the profitability of selling information and the buyer's overall probability of choosing correctly downstream, though it need not make individual information products less informative.

cs.GT↗

arXiv2026

Provably Safe Sim-to-Real Transfer

We address safe sim-to-real transfer, in which an agent leverages an imperfect simulator and limited real-world interaction while ensuring safety throughout data collection in the real system. This problem arises in applications such as robotics and healthcare: simulators provide cheap data, but sim-to-real mismatch makes direct transfer unreliable, and collecting real-world data to correct this mismatch must itself be safe. Moreover, deployment objectives may vary across tasks, making it costly to collect new data for each reward function. We therefore formulate safe sim-to-real transfer as a reward-free safe reinforcement learning (RL) problem, in which data are collected once and reused to plan for arbitrary reward functions. We develop a computationally efficient algorithm that identifies where the simulator and real dynamics differ, uses certified simulator transitions where they are reliable, and estimates mismatched transitions from safely collected data. With high probability, every policy deployed during learning is feasible, and the collected data support the computation of a feasible and near-optimal policy for any reward function. When the simulator is uninformative, our algorithm recovers online reward-free safe RL while improving the best-known sample complexity by a factor of $\widetildeΘ(H/ξ^2)$, where $ξ$ is the safety margin of a baseline policy. When the simulator is accurate on most transitions, this improvement grows to $\widetildeΘ(H^2|\mc S||\mc A|/(ξ^2|\mc B|))$, where $|\mc B|$ denotes the size of the sim-to-real mismatch region.

cs.LG↗

arXiv2026

Inverse Linear Quadratic Gaussian Games: Constrained Setting and Transferability

This work addresses finite-horizon inverse linear quadratic Gaussian games. In a constrained setting, we characterize the set of cost parameters and optimal dual values that generate a given generalized Nash equilibrium, and we propose an algorithm to compute these parameters. In an unconstrained setting, we address transferability, namely, we bound the cost value perturbation between two policies: one induced by the identified cost parameters, the other by the expert parameters, under a set of different dynamics. This cost value perturbation scales linearly with the deviations in the dynamics and the identified cost parameters. Through numerical simulations, we show that, in a constrained setting, our algorithm identifies the cost parameters and dual values that can reproduce the policy and trajectories corresponding to the observed generalized Nash equilibrium. In an unconstrained setting, we show with a traffic simulation and real-robot experiments that the identified cost parameters can be used to control sufficiently close dynamics, with performance degrading linearly with the deviations in the dynamics and the identified cost parameters.

eess.SY↗

arXiv2026

Independent Learning of Nash Equilibria in Partially Observable Markov Potential Games with Decoupled Dynamics

We study Nash equilibrium learning in partially observable Markov games (POMGs), a multi-agent reinforcement learning framework in which agents cannot fully observe the underlying state. Prior work in this setting relies on centralization or information sharing, and suffers from sample and computational complexity that scales exponentially in the number of players. We focus on a subclass of POMGs with independent state transitions, where agents remain coupled through their rewards, and assume that the underlying fully observed Markov game is a Markov potential game. For this class, we present an independent learning algorithm in which players, observing only their own actions and observations and without communication, jointly converge to an approximate Nash equilibrium. Due to partial observability, optimal policies may in general depend on the full action-observation history. Under a filter stability assumption, we show that policies based on finite history windows provide sufficient approximation guarantees. This enables us to approximate the POMG by a surrogate Markov game that is near-potential, leading to quasi-polynomial sample and computational complexity for independent Nash equilibrium learning in the underlying POMG.

cs.GT↗

arXiv2026

Learning to build covering structures with continuous adjustments

Robotic construction offers the potential to use materials more efficiently and create complex geometries, but current methods rely on rigid, high-precision plans that cannot accommodate the tolerances, inaccuracies, and unexpected changes inherent in physical fabrication. In this work, we introduce a reinforcement learning approach that forgoes predefined plans entirely, instead generating construction sequences adaptively as the structure is built. Our method operates on graph-structured state representations and a mixed (parameterized) action space, requiring both discrete block selection and continuous placement parameters. Because the stability simulation of a structure is computationally heavy, we develop an efficient exploration strategy by incorporating unilateral edges into graph neural networks, extending soft actor-critic (SAC) to this hybrid setting. We evaluate our algorithm, HSAC, against the prior method hybrid-PPO (HPPO), demonstrating significantly higher asymptotic performance and good sample efficiency. We also demonstrate HSAC's robustness to hyperparameter choices and its exploration capability, handling up to 10 discrete actions without performance degradation. Finally, we validate our approach on a physical two-robot setup, successfully building a spanning arch with 3D-printed blocks in closed-loop execution, confirming that policies trained in simulation transfer to real hardware.

cs.RO↗

arXiv2026

Refundable Deposits: How to Restore Cooperation in Finitely Repeated Games

While infinitely repeated games admit a rich set of Nash equilibria, finitely repeated games typically have a much smaller and often inefficient one. We show how to enlarge this set using deposits: in each period a player may place a refundable sum with a neutral intermediary, returned when the game ends and forfeited following a deviation. Paying these deposits is voluntary and incentive compatible at every stage, so no commitment by the players is assumed, the only commitment required being that of the intermediary to a refund rule fixed before play begins. The mechanism sustains payoff profiles more efficient than those of the standard equilibria, without altering the underlying game and without transfers between players. We demonstrate it on the prisoner's dilemma, a congestion game, and a public goods game, all settings where cooperation cannot emerge in the standard finitely repeated version. We also apply it to a dynamic common-pool resource, suggesting that the construction extends beyond repeated stage-games.

cs.GT↗

arXiv2026

Model-Based Learning of Near-Optimal Finite-Window Policies in POMDPs

We study model-based learning of finite-window policies in tabular partially observable Markov decision processes (POMDPs). A common approach to learning under partial observability is to approximate unbounded history dependencies using finite action-observation windows. This induces a finite-state Markov decision process (MDP) over histories, referred to as the superstate MDP. Once a model of this superstate MDP is available, standard MDP algorithms can be used to compute optimal policies, motivating the need for sample-efficient model estimation. Estimating the superstate MDP model is challenging because trajectories are generated by interaction with the original POMDP, creating a mismatch between the sampling process and target model. We propose a model estimation procedure for tabular POMDPs and analyze its sample complexity. Our analysis exploits a connection between filter stability and concentration inequalities for weakly dependent random variables. As a result, we obtain tight sample complexity guarantees for estimating the superstate MDP model from a single trajectory. Combined with value iteration, this yields approximately optimal finite-window policies for the POMDP.

cs.LG↗

arXiv2026

On the characterization of constrained correlated equilibria in Markov games

Markov games with coupling constraints model constrained dynamical decision-making involving self-interested agents, where the feasibility of an individual agent's strategy depends on the joint strategies of the others. Such games arise in numerous real-world applications involving safety requirements and budget caps, for example, in environmental management, electricity markets, and transportation systems. In unconstrained dynamical decision-making, the correlated equilibrium has emerged as a desired solution concept, due to its computational tractability and amenability to learning algorithms. Understanding how coupling constraints shape correlated equilibria is a crucial step towards computing solutions in constrained Markov games. In this paper, we formalize and characterize the notion of constrained correlated equilibria for Markov games, defined as feasible joint policies where any unilateral deviation is either unprofitable or infeasible.

cs.GT↗

arXiv2026

Fast Rates for Inverse Reinforcement Learning

We establish novel structural and statistical results for entropy-regularized min-max inverse reinforcement learning (Min-Max-IRL) in finite-horizon MDPs with Borel state and action spaces. We show that maximum likelihood estimation (MLE) and Min-Max-IRL are equivalent at the population level, and at the empirical level under deterministic dynamics. For linear reward classes, we leverage pseudo-self-concordance of the Min-Max-IRL loss to prove that both the excess trajectory-level KL divergence and the squared parameter error in the Hessian norm decay at the fast rate $O(n^{-1})$, where $n$ is the number of expert trajectories. A local minimax lower bound matches the parameter-error rate up to logarithmic factors in the well-specified deterministic setting. Our guarantees apply under misspecification and require no uniform state-coverage assumption. We further extend reward-identifiability results to general Borel spaces and compare our results with MLE-based guarantees.

cs.LG↗

arXiv2026

Eliciting Truthful Feedback for Preference-Based Learning via the VCG Mechanism

We study resource allocation problems in which a central planner allocates resources among strategic agents with private cost functions in order to minimize a social cost, defined as an aggregate of the agents' costs. This setting poses two main challenges: (i) the agents' cost functions may be unknown to them or difficult to specify explicitly, and (ii) agents may misreport their costs strategically. To address these challenges, we propose an algorithm that combines preference-based learning with Vickrey-Clarke-Groves (VCG) payments to incentivize truthful reporting. Our algorithm selects informative preference queries via D-optimal design, estimates cost parameters through maximum likelihood, and computes VCG allocations and payments based on these estimates. In a one-shot setting, we prove that the mechanism is approximately truthful, individually rational, and efficient up to an error of $\tilde{\mathcal O}(K^{-1/2})$ for $K$ preference queries per agent. In an online setting, these guarantees hold asymptotically with sublinear regret at a rate of $\tilde{\mathcal O}(T^{2/3})$ after $T$ rounds. Finally, we validate our approach through a numerical case study on demand response in local electricity markets.

cs.GT↗

arXiv2026

Uncertainty and Autarky: Cooperative Game Theory for Stable Local Energy Market Partitioning

Local energy markets empower prosumers in distribution grids to form coalitions for collective self-consumption. An open question is to analyze the scale and composition of local energy market coalitions formed by strategic prosumers in distribution grids. This analysis must account for grid constraints, stochasticity of load and generation, as well as the interaction between possibly multiple local energy markets in the distribution grid. In this work, we present a cooperative game theoretic framework to study distribution grid partitioning into local energy markets under uncertain prosumption, grid constraints, and coalitional externalities. We formulate the optimal stable partitioning problem to balance the interests of the grid operator with that of strategic prosumers. Under deterministic load and generation, we show that the largest market coalition is the optimal stable partition. Under high levels of grid congestion, we show that individual self-consumption corresponds to the optimal stable partition. For the general case of stochastic prosumption and moderate grid congestion levels, we provide an algorithm to evaluate the optimal stable partition. We validate our algorithm and theory using numerical experiments on benchmark and real world distribution grids. Our results help in understanding the impact of prosumption uncertainty and grid constraints on coalition formation.

eess.SY↗

arXiv2026

Constrained Meta Reinforcement Learning with Provable Test-Time Safety

Meta reinforcement learning (RL) allows agents to leverage experience across a distribution of tasks on which the agent can train at will, enabling faster learning of optimal policies on new test tasks. Despite its success in improving sample complexity on test tasks, many real-world applications, such as robotics and healthcare, impose safety constraints during testing. Constrained meta RL provides a promising framework for integrating safety into meta RL. An open question in constrained meta RL is how to ensure safety of the policy on the real-world test task, while reducing the sample complexity and thus, enabling faster learning of optimal policies. To address this gap, we propose an algorithm that refines policies learned during training, with provable safety and sample complexity guarantees for learning a near optimal policy on the test tasks. We further derive a matching lower bound, showing that this sample complexity is tight.

cs.LG↗

arXiv2026

Nash Equilibria in Games with Playerwise Concave Coupling Constraints: Existence and Computation

We study the existence and computation of Nash equilibria in concave games where the players' admissible strategies are subject to shared coupling constraints. Under playerwise concavity of constraints, we prove existence of Nash equilibria. Our proof leverages topological fixed point theory and novel structural insights into the contractibility of feasible sets, and relaxes strong assumptions for existence in prior work. Having established existence, we address the question of whether in the presence of coupling constraints, playerwise independent learning dynamics have convergence guarantees. We address this positively for the class of potential games by designing a convergent algorithm. To account for the possibly nonconvex feasible region, we employ a log barrier regularized gradient ascent with adaptive stepsizes. Starting from an initial feasible strategy profile and under exact gradient feedback, the proposed method converges to an $ε$-approximate constrained Nash equilibrium within $\mathcal{O}(ε^{-3})$ iterations.

cs.GT↗

arXiv2026

Finite-time convergence to an $ε$-efficient Nash equilibrium in potential games

This paper investigates the convergence time of log-linear learning to an $ε$-efficient Nash equilibrium in potential games, where an efficient Nash equilibrium is defined as the maximizer of the potential function. Previous literature provides asymptotic convergence rates to efficient Nash equilibria, and existing finite-time rates are limited to potential games with further assumptions such as the interchangeability of players. We prove the first finite-time convergence to an $ε$-efficient Nash equilibrium in general potential games. Our bounds depend polynomially on $1/ε$, an improvement over previous bounds for subclasses of potential games that are exponential in $1/ε$. We then strengthen our convergence result in two directions: first, we show that a variant of log-linear learning requiring a constant factor less feedback on the utility per round enjoys a similar convergence time; second, we demonstrate the robustness of our convergence guarantee if log-linear learning is subject to small perturbations such as alterations in the learning rule or noise-corrupted utilities.

cs.MA↗

arXiv2025

Scheduling the Charge of Temporally Flexible Electric Vehicles: a Market-based Approach

The increasing electrification of human activities and the rapid integration of variable renewable energy sources strain the power grid. A solution to address the need for more grid storage is to use the battery of electric vehicles as a back-up capacity. However, drivers tend to disconnect their electric vehicle when its battery is needed the most. We propose a charge scheduler that incentivizes drivers to delay their disconnection to improve vehicle-to-grid services. We also leverage drivers' temporal flexibility to alleviate congestion in oversubscribed charging stations. We formulate the computation of an optimal flexible schedule as a mixed-integer quadratic problem. We tractably approximate its solution using the Alternating Direction Method of Multipliers. Considering the possibility that strategic drivers misreport their charging preferences to the station coordinator, we then propose a Vickrey-Clarke-Groves mechanism that incentivizes truthful reporting. We conclude with a simulated case study using real-world data to quantitatively assess the added value of drivers' temporal flexibility for enhancing vehicle-to-grid services and reducing station congestion.

eess.SY↗

arXiv2025

No-Regret Learning in Stackelberg Games with an Application to Electric Ride-Hailing

We consider the problem of efficiently learning to play single-leader multi-follower Stackelberg games when the leader lacks knowledge of the lower-level game. Such games arise in hierarchical decision-making problems involving self-interested agents. For example, in electric ride-hailing markets, a central authority aims to learn optimal charging prices to shape fleet distributions and charging patterns of ride-hailing companies. Existing works typically apply gradient-based methods to find the leader's optimal strategy. Such methods are impractical as they require that the followers share private utility information with the leader. Instead, we treat the lower-level game as a black box, assuming only that the followers' interactions approximate a Nash equilibrium while the leader observes the realized cost of the resulting approximation. Under kernel-based regularity assumptions on the leader's cost function, we develop a no-regret algorithm that converges to an $ε$-Stackelberg equilibrium in $O(\sqrt{T})$ rounds. Finally, we validate our approach through a numerical case study on optimal pricing in electric ride-hailing markets.

eess.SY↗

arXiv2025

Efficient Preference-Based Reinforcement Learning: Randomized Exploration Meets Experimental Design

We study reinforcement learning from human feedback in general Markov decision processes, where agents learn from trajectory-level preference comparisons. A central challenge in this setting is to design algorithms that select informative preference queries to identify the underlying reward while ensuring theoretical guarantees. We propose a meta-algorithm based on randomized exploration, which avoids the computational challenges associated with optimistic approaches and remains tractable. We establish both regret and last-iterate guarantees under mild reinforcement learning oracle assumptions. To improve query complexity, we introduce and analyze an improved algorithm that collects batches of trajectory pairs and applies optimal experimental design to select informative comparison queries. The batch structure also enables parallelization of preference queries, which is relevant in practical deployment as feedback can be gathered concurrently. Empirical evaluation confirms that the proposed method is competitive with reward-based reinforcement learning while requiring a small number of preference queries.

cs.LG↗