arXiv subjects

Mehrdad Moharrami

Publications and source records attributed to Mehrdad Moharrami.

14 recordsLinked to original sources

arXiv2026

Offline Constrained RLHF with Multiple Preference Oracles

We study offline constrained reinforcement learning from human feedback with multiple preference oracles. Motivated by applications that trade off performance with safety or fairness, we aim to maximize target population utility subject to a minimum protected group welfare constraint. From pairwise comparisons collected under a reference policy, we estimate oracle-specific rewards via maximum likelihood and analyze how statistical uncertainty propagates through the dual program. We cast the constrained objective as a KL-regularized Lagrangian whose primal optimizer is a Gibbs policy, reducing learning to a convex dual problem. We propose a dual-only algorithm that ensures high-probability constraint satisfaction and provide the first finite-sample performance guarantees for offline constrained preference learning. Finally, we extend our theoretical analysis to accommodate multiple constraints and general f-divergence regularization.

cs.LG↗

arXiv2026

Tail Distribution of Regret in Optimistic Reinforcement Learning

We derive instance-dependent tail bounds for the regret of optimism-based reinforcement learning in finite-horizon tabular Markov decision processes with unknown transition dynamics. We first study a UCBVI-type (model-based) algorithm and characterize the tail distribution of the cumulative regret $R_K$ over $K$ episodes via explicit bounds on $P(R_K \ge x)$, going beyond analyses limited to $E[R_K]$ or a single high-probability quantile. We analyze two natural exploration-bonus schedules for UCBVI: (i) a $K$-dependent scheme that explicitly incorporates the total number of episodes $K$, and (ii) a $K$-independent (anytime) scheme that depends only on the current episode index. We then complement the model-based results with an analysis of optimistic Q-learning (model-free) under a $K$-dependent bonus schedule. Across both the model-based and model-free settings, we obtain upper bounds on $P(R_K \ge x)$ with a distinctive two-regime structure: a sub-Gaussian tail starting from an instance-dependent scale up to a transition threshold, followed by a sub-Weibull tail beyond that point. We further derive corresponding instance-dependent bounds on the expected regret $E[R_K]$. The proposed algorithms depend on a tuning parameter $α$, which balances the expected regret and the range over which the regret exhibits sub-Gaussian decay.

cs.LG↗

arXiv2025

Learning to Admit Optimally in an $M/M/k/k+N$ Queueing System with Unknown Service Rate

Motivated by applications of the Erlang-B blocking model and the extended $M/M/k/k+N$ model that allows for some queueing, beyond communication networks to sizing and pricing in production, messaging, and app-based parking systems, we study admission control for such systems with unknown service rate. In our model, a dispatcher either admits every arrival into the system (when there is room) or blocks it. Every served job yields a fixed reward but incurs a per unit time holding cost which includes the waiting time in the queue to get service if there is any. We aim to design a dispatching policy that maximizes the long-term average reward by observing arrival times and system state at arrivals, a realistic decision-event driven sampling of such systems. The dispatcher observes neither service times nor departure epochs, which excludes the use of reward-based reinforcement learning approaches. We develop our learning-based dispatch scheme as a parametric learning problem a'la self-tuning adaptive control. In our problem, certainty equivalent control switches between always admit if room (explore infinitely often), and never admit (terminate learning), so at judiciously chosen times we avoid the never admit recommendation. We prove that our proposed policy asymptotically converges to the optimal policy and present finite-time regret guarantees. The extreme contrast in the control policies shows up in our regret bounds for different parameter regimes: constant in one versus logarithmic in another.

eess.SY↗

arXiv2025

The Planted Spanning Tree Problem

We study the problem of detecting and recovering a planted spanning tree $M_n^*$ hidden within a complete, randomly weighted graph $G_n$. Specifically, each edge $e$ has a non-negative weight drawn independently from $P_n$ if $e \in M_n^*$ and from $Q_n$ otherwise, where $P_n \equiv P$ is fixed and $Q_n$ scales with $n$ such that its density at the origin satisfies $\lim_{n\to\infty} n Q'_n(0)=1.$ We consider two representative cases: when $M_n^*$ is either a uniform spanning tree or a uniform Hamiltonian path. We analyze the recovery performance of the minimum spanning tree (MST) algorithm and derive a fixed-point equation that characterizes the asymptotic fraction of edges in $M_n^*$ successfully recovered by the MST as $n \to \infty.$ Furthermore, we establish the asymptotic mean weight of the MST, extending Frieze's $ζ(3)$ result to the planted model. Leveraging this result, we design an efficient test based on the MST weight and show that it can distinguish the planted model from the unplanted model with vanishing testing error as $n \to \infty.$ Our analysis relies on an asymptotic characterization of the local structure of the planted model, employing the framework of local weak convergence.

cs.DS↗

arXiv2024

A Stackelberg Game Model of Flocking

We study a Stackelberg game to examine how two agents determine to cooperate while competing with each other. Each selects an arrival time to a destination, the earlier one fetching a higher reward. There is, however, an inherent penalty in arriving too early as well as a risk in traveling alone. This gives rise to the possibility of the agents cooperating by traveling together while competing for the reward. In our prior work [1] we studied this problem as a sequential game among a set of $N$ competing agents in continuous time, and defined the formation of a group traveling together as arriving at exactly the same time. In the present study, we relax this definition to allow arrival times within a small window, and study a 2-agent game in both continuous and discrete time, referred to as the flock formation game. We derive and examine the properties of the subgame perfect equilibrium (SPE) of this game.

cs.GT↗

arXiv2024

On the Convergence of Modified Policy Iteration in Risk Sensitive Exponential Cost Markov Decision Processes

Modified policy iteration (MPI) is a dynamic programming algorithm that combines elements of policy iteration and value iteration. The convergence of MPI has been well studied in the context of discounted and average-cost MDPs. In this work, we consider the exponential cost risk-sensitive MDP formulation, which is known to provide some robustness to model parameters. Although policy iteration and value iteration have been well studied in the context of risk sensitive MDPs, MPI is unexplored. We provide the first proof that MPI also converges for the risk-sensitive problem in the case of finite state and action spaces. Since the exponential cost formulation deals with the multiplicative Bellman equation, our main contribution is a convergence proof which is quite different than existing results for discounted and risk-neutral average-cost problems as well as risk sensitive value and policy iteration approaches. We conclude our analysis with simulation results, assessing MPI's performance relative to alternative dynamic programming methods like value iteration and policy iteration across diverse problem parameters. Our findings highlight risk-sensitive MPI's enhanced computational efficiency compared to both value and policy iteration techniques.

cs.LG↗

arXiv2023

Backward and Forward Inference in Interacting Independent-Cascade Processes: A Scalable and Convergent Message-Passing Approach

We study the problems of estimating the past and future evolutions of two diffusion processes that spread concurrently on a network. Specifically, given a known network $G=(V, \overrightarrow{E})$ and a (possibly noisy) snapshot $\mathcal{O}_n$ of its state taken at (a possibly unknown) time $W$, we wish to determine the posterior distributions of the initial state of the network and the infection times of its nodes. These distributions are useful in finding source nodes of epidemics and rumors -- $\textit{backward inference}$ -- , and estimating the spread of a fixed set of source nodes -- $\textit{forward inference}$. To model the interaction between the two processes, we study an extension of the independent-cascade (IC) model where, when a node gets infected with either process, its susceptibility to the other one changes. First, we derive the exact joint probability of the initial state of the network and the observation-snapshot $\mathcal{O}_n$. Then, using the machinery of factor-graphs, factor-graph transformations, and the generalized distributive-law, we derive a Belief-Propagation (BP) based algorithm that is scalable to large networks and can converge on graphs of arbitrary topology (at a likely expense in approximation accuracy).

cs.SI↗

arXiv2023

Performance Bounds for Policy-Based Average Reward Reinforcement Learning Algorithms

Many policy-based reinforcement learning (RL) algorithms can be viewed as instantiations of approximate policy iteration (PI), i.e., where policy improvement and policy evaluation are both performed approximately. In applications where the average reward objective is the meaningful performance metric, discounted reward formulations are often used with the discount factor being close to $1,$ which is equivalent to making the expected horizon very large. However, the corresponding theoretical bounds for error performance scale with the square of the horizon. Thus, even after dividing the total reward by the length of the horizon, the corresponding performance bounds for average reward problems go to infinity. Therefore, an open problem has been to obtain meaningful performance bounds for approximate PI and RL algorithms for the average-reward setting. In this paper, we solve this open problem by obtaining the first finite-time error bounds for average-reward MDPs, and show that the asymptotic error goes to zero in the limit as policy evaluation and policy improvement errors go to zero.

cs.LG↗

arXiv2023

The Erlang Weighted Tree, A New Branching Process

In this paper, we study a new discrete tree and the resulting branching process, which we call the \textbf{E}rlang \textbf{W}eighted \textbf{T}ree(\textbf{EWT}). The EWT appears as the local weak limit of a random graph model proposed in~\cite{La2015}. In contrast to the local weak limit of well-known random graph models, the EWT has an interdependent structure. In particular, its vertices encode a multi-type branching process with uncountably many types. We derive the main properties of the EWT, such as the probability of extinction, growth rate, etc. We show that the probability of extinction is the smallest fixed point of an operator. We then take a point process perspective and analyze the growth rate operator. We derive the Krein--Rutman eigenvalue $β_0$ and the corresponding eigenfunctions of the growth operator, and show that the probability of extinction equals one if and only if $β_0 \leq 1$.

math.PR↗

arXiv2022

Rarest-First with Probabilistic-Mode-Suppression

Recent studies suggested that the BitTorrent's rarest-first protocol, owing to its work-conserving nature, can become unstable in the presence of non-persistent users. Consequently, for any provably stable protocol, many peers, at some point, would have to be endogenously forced to hold off their file-download activity. In this work, we propose a tunable piece-selection policy that minimizes this (undesirable) requisite by combining the (work-conserving but not stabilizing) rarest-first protocol with only an appropriate share of the (non-work conserving and stabilizing) mode-suppression protocol. We refer to this policy as ``Rarest-First with Probabilistic Mode-Suppression'' or simply RFwPMS. We study RFwPMS using a stochastic abstraction of the BitTorrent network that is general enough to capture a multiple swarm setting of non-persistent users -- each swarm having its own altruistic preferences that may or may not overlap with those of other swarms. Using Lyapunov drift analysis, we show that for all kinds of inter-swarm behaviors and all arrival-rate configurations, RFwPMS is stable. Then, using the Kingman's moment bound technique, we further show that the expected steady-state sojourn time of RFwPMS is independent of the arrival-rate in the single-swarm case (under a mild additional assumption). Finally, our simulation-based performance evaluation confirms our theoretical findings and shows that the steady-state expected sojourn time is linear in the file-size (compared to our loose estimate of a polynomial with degree 6). Overall, an improved performance is observed in comparison to previously proposed stabilizing schemes like mode-suppression (MS).

eess.SY↗

arXiv2022

A Policy Gradient Algorithm for the Risk-Sensitive Exponential Cost MDP

We study the risk-sensitive exponential cost MDP formulation and develop a trajectory-based gradient algorithm to find the stationary point of the cost associated with a set of parameterized policies. We derive a formula that can be used to compute the policy gradient from (state, action, cost) information collected from sample paths of the MDP for each fixed parameterized policy. Unlike the traditional average-cost problem, standard stochastic approximation theory cannot be used to exploit this formula. To address the issue, we introduce a truncated and smooth version of the risk-sensitive cost and show that this new cost criterion can be used to approximate the risk-sensitive cost and its gradient uniformly under some mild assumptions. We then develop a trajectory-based gradient algorithm to minimize the smooth truncated estimation of the risk-sensitive cost and derive conditions under which a sequence of truncations can be used to solve the original, untruncated cost problem.

eess.SY↗

arXiv2022

Impact of Community Structure on Cascades

We study cascades under the threshold model on sparse random graphs with community structure. In this model, individuals adopt the new behavior based on how many neighbors have already chosen it. Specifically, we consider the permanent adoption model wherein individuals that have adopted the new behavior (or opinion) cannot change their state. We present a differential-equation-based tight approximation to the stochastic process of adoption and prove the validity of the mean-field equations. In addition, we characterize both necessary and sufficient conditions for contagion to happen no matter how small the set of initial adopters is. Finally, we study the problem of optimum seeding given budget constraints and propose a gradient-based heuristic seeding strategy. Our algorithm, numerically, dispels commonly held beliefs in the literature that suggest the best seeding strategy is to seed over the vertices with the highest number of neighbors.

math.PR↗

arXiv2021

Structural Stability of a Family of Group Formation Games

We introduce and study a group formation game in which individuals/agents, driven by self-interest, team up in disjoint groups so as to be in groups of high collective strength. This strength could be group identity, reputation, or protection, and is equally shared by all group members. The group's access to resources, obtained from its members, is traded off against the geographic dispersion of the group: spread-out groups are more costly to maintain. We seek to understand the stability and structure of such partitions. We define two types of equilibria: Acceptance Equilibria (AE), in which no agent will unilaterally change group affiliation, either because the agent cannot increase her utility by switching, or because the intended receiving group is unwilling to accept her (i.e., the utility of existing members would decrease if she joined); and Strong Acceptance Equilibria (SAE), in which no subset of any group will change group affiliations (move together) for the same reasons given above. We show that under natural assumptions on the group utility functions, both an AE and SAE always exist, and that any sequence of improving deviations by agents (resp., subsets of agents in the same group) converges to an AE (resp., SAE). We then characterize the properties of the AEs. We show that an "encroachment" relationship - which groups have members in the territory of other groups - always gives rise to a directed acyclic graph (DAG); conversely, given any DAG, we can construct a game with suitable conditions on the utility function that has an AE with the encroachment structure specified by the given graph.

cs.GT↗

arXiv2020

The Planted Matching Problem: Phase Transitions and Exact Results

We study the problem of recovering a planted matching in randomly weighted complete bipartite graphs $K_{n,n}$. For some unknown perfect matching $M^*$, the weight of an edge is drawn from one distribution $P$ if $e \in M^*$ and another distribution $Q$ if $e \notin M^*$. Our goal is to infer $M^*$, exactly or approximately, from the edge weights. In this paper we take $P=\exp(λ)$ and $Q=\exp(1/n)$, in which case the maximum-likelihood estimator of $M^*$ is the minimum-weight matching $M_{\text{min}}$. We obtain precise results on the overlap between $M^*$ and $M_{\text{min}}$, i.e., the fraction of edges they have in common. For $λ\ge 4$ we have almost perfect recovery, with overlap $1-o(1)$ with high probability. For $λ< 4$ the expected overlap is an explicit function $α(λ) < 1$: we compute it by generalizing Aldous' celebrated proof of the $ζ(2)$ conjecture for the un-planted model, using local weak convergence to relate $K_{n,n}$ to a type of weighted infinite tree, and then deriving a system of differential equations from a message-passing algorithm on this tree.

cs.DS↗