arXiv subjects

Dheeraj Narasimha

Publications and source records attributed to Dheeraj Narasimha.

5 recordsLinked to original sources

arXiv2026

Model Predictive Control is almost Optimal for Heterogeneous Restless Multi-armed Bandits

We consider a general infinite horizon Heterogeneous Restless multi-armed Bandit (RMAB). Heterogeneity is a fundamental problem for many real-world systems largely because it resists many concentration arguments. In this paper, we assume that each of the $N$ arms can have different model parameters. Model predictive control is a well-known control strategy that repeatedly solves a finite-horizon optimization problem of length $τ$ to produce a policy that can be applied to an infinite-horizon setting. In this paper, we adopt this approach by repeatedly solving a finite linear program, yielding what we call the LP-update policy for the infinite-horizon problem. Under a mild assumption of uniform ergodicity, we show an $\mathcal{O}\left(\sqrt{1/N}\right)$ suboptimality gap on this well-known algorithm that works very well in practice. In addition to the LP-update policy we are able to derive a finite-horizon policy (LP-update with recomputation) that segments the infinite time horizon into finite horizon problems that allow us to explicitly connect the length of computation time to the acceptable error tolerance. Our simulations demonstrate that our algorithm works extremely well even when this finite-horizon, $τ$, is very small (in our case $5$), which makes it computationally efficient. Our theoretical results draw on techniques from the model predictive control literature by invoking the concept of \emph{dissipativity} and generalize quite easily to the more general weakly coupled heterogeneous Markov Decision Process setting. In addition, we draw a parallel between our own policy and the LP-index policy by showing that the LP-index policy corresponds to $τ=1$.

math.OC↗

arXiv2025

PITA: Preference-Guided Inference-Time Alignment for LLM Post-Training

Inference-time alignment enables large language models (LLMs) to generate outputs aligned with end-user preferences without further training. Recent post-training methods achieve this by using small guidance models to modify token generation during inference. These methods typically optimize a reward function KL-regularized by the original LLM taken as the reference policy. A critical limitation, however, is their dependence on a pre-trained reward model, which requires fitting to human preference feedback--a potentially unstable process. In contrast, we introduce PITA, a novel framework that integrates preference feedback directly into the LLM's token generation, eliminating the need for a reward model. PITA learns a small preference-based guidance policy to modify token probabilities at inference time without LLM fine-tuning, reducing computational cost and bypassing the pre-trained reward model dependency. The problem is framed as identifying an underlying preference distribution, solved through stochastic search and iterative refinement of the preference-based guidance model. We evaluate PITA across diverse tasks, including mathematical reasoning and sentiment classification, demonstrating its effectiveness in aligning LLM outputs with user preferences.

cs.AI↗

arXiv2025

Model Predictive Control is Almost Optimal for Restless Bandit

We consider the discrete time infinite horizon average reward restless markovian bandit (RMAB) problem. We propose a \emph{model predictive control} based non-stationary policy with a rolling computational horizon $τ$. At each time-slot, this policy solves a $τ$ horizon linear program whose first control value is kept as a control for the RMAB. Our solution requires minimal assumptions and quantifies the loss in optimality in terms of $τ$ and the number of arms, $N$. We show that its sub-optimality gap is $O(1/\sqrt{N})$ in general, and $\exp(-Ω(N))$ under a local-stability condition. Our proof is based on a framework from dynamic control known as \emph{dissipativity}. Our solution easy to implement and performs very well in practice when compared to the state of the art. Further, both our solution and our proof methodology can easily be generalized to more general constrained MDP settings and should thus, be of great interest to the burgeoning RMAB community.

math.OC↗

arXiv2025

CONGO: Compressive Online Gradient Optimization

We address the challenge of zeroth-order online convex optimization where the objective function's gradient exhibits sparsity, indicating that only a small number of dimensions possess non-zero gradients. Our aim is to leverage this sparsity to obtain useful estimates of the objective function's gradient even when the only information available is a limited number of function samples. Our motivation stems from the optimization of large-scale queueing networks that process time-sensitive jobs. Here, a job must be processed by potentially many queues in sequence to produce an output, and the service time at any queue is a function of the resources allocated to that queue. Since resources are costly, the end-to-end latency for jobs must be balanced with the overall cost of the resources used. While the number of queues is substantial, the latency function primarily reacts to resource changes in only a few, rendering the gradient sparse. We tackle this problem by introducing the Compressive Online Gradient Optimization framework which allows compressive sensing methods previously applied to stochastic optimization to achieve regret bounds with an optimal dependence on the time horizon without the full problem dimension appearing in the bound. For specific algorithms, we reduce the samples required per gradient estimate to scale with the gradient's sparsity factor rather than its full dimensionality. Numerical simulations and real-world microservices benchmarks demonstrate CONGO's superiority over gradient descent approaches that do not account for sparsity.

cs.LG↗

arXiv2014

Autoregressive Cascades on Random Networks

This paper considers a model for cascades on random networks in which the cascade propagation at any node depends on the load at the failed neighbor, the degree of the neighbor as well as the load at that node. Each node in the network bears an initial load that is below the capacity of the node. The trigger for the cascade emanates at a single node or a small fraction of the nodes from some external shock. Upon failure, the load at the failed node gets divided randomly and added to the existing load at those neighboring nodes that have not yet failed. Subsequently, a neighboring node fails if its accumulated load exceeds its capacity. The failed node then plays no further part in the process. The cascade process stops as soon as the accumulated load at all nodes that have not yet failed is below their respective capacities. The model is shown to operate in two regimes, one in which the cascade terminates with only a finite number of node failures. In the other regime there is a positive probability that the cascade continues indefinitely. Bounds are obtained on the critical parameter where the phase transition occurs.

physics.soc-ph↗