arXiv subjects

Rahul V. Kulkarni

Publications and source records attributed to Rahul V. Kulkarni.

At least 19 recordsLinked to original sources

arXiv2026

Thermodynamics of Reinforcement Learning Curricula

Connections between statistical mechanics and machine learning have repeatedly proven fruitful, providing insight into optimization, generalization, and representation learning. In this work, we follow this tradition by leveraging results from non-equilibrium thermodynamics to formalize curriculum learning in reinforcement learning (RL). In particular, we propose a geometric framework for RL by interpreting reward parameters as coordinates on a task manifold. We show that, by minimizing the excess thermodynamic work, optimal curricula correspond to geodesics in this task space. As an application of this framework, we provide an algorithm, "MEW" (Minimum Excess Work), to derive a principled schedule for temperature annealing in maximum-entropy RL.

cs.LG↗

arXiv2026

Maximum Entropy Exploration Without the Rollouts

Efficient exploration remains a central challenge in reinforcement learning, serving as a useful pretraining objective for data collection, particularly when an external reward function is unavailable. A principled formulation of the exploration problem is to find policies that maximize the entropy of their induced steady-state visitation distribution, thereby encouraging uniform long-run coverage of the state space. Many existing exploration approaches require estimating state visitation frequencies through repeated on-policy rollouts, which can be computationally expensive. In this work, we instead consider an intrinsic average-reward formulation in which the reward is derived from the visitation distribution itself, so that the optimal policy maximizes steady-state entropy. An entropy-regularized version of this objective admits a spectral characterization: the relevant stationary distributions can be computed from the dominant eigenvectors of a problem-dependent transition matrix. This insight leads to a novel algorithm for solving the maximum entropy exploration problem, EVE (EigenVector-based Exploration), which avoids explicit rollouts and distribution estimation, instead computing the solution through iterative updates, similar to a value-based approach. To address the original unregularized objective, we employ a posterior-policy iteration (PPI) approach, which monotonically improves the entropy and converges in value. We prove convergence of EVE under standard assumptions and demonstrate empirically that it efficiently produces policies with high steady-state entropy, achieving competitive exploration performance relative to rollout-based baselines in deterministic grid-world environments.

cs.LG↗

arXiv2025

Average-Reward Soft Actor-Critic

The average-reward formulation of reinforcement learning (RL) has drawn increased interest in recent years for its ability to solve temporally-extended problems without relying on discounting. Meanwhile, in the discounted setting, algorithms with entropy regularization have been developed, leading to improvements over deterministic methods. Despite the distinct benefits of these approaches, deep RL algorithms for the entropy-regularized average-reward objective have not been developed. While policy-gradient based approaches have recently been presented for the average-reward literature, the corresponding actor-critic framework remains less explored. In this paper, we introduce an average-reward soft actor-critic algorithm to address these gaps in the field. We validate our method by comparing with existing average-reward algorithms on standard RL benchmarks, achieving superior performance for the average-reward criterion.

cs.LG↗

arXiv2025

Bootstrapped Reward Shaping

In reinforcement learning, especially in sparse-reward domains, many environment steps are required to observe reward information. In order to increase the frequency of such observations, "potential-based reward shaping" (PBRS) has been proposed as a method of providing a more dense reward signal while leaving the optimal policy invariant. However, the required "potential function" must be carefully designed with task-dependent knowledge to not deter training performance. In this work, we propose a "bootstrapped" method of reward shaping, termed BSRS, in which the agent's current estimate of the state-value function acts as the potential function for PBRS. We provide convergence proofs for the tabular setting, give insights into training dynamics for deep RL, and show that the proposed method improves training speed in the Atari suite.

cs.LG↗

arXiv2025

EVAL: EigenVector-based Average-reward Learning

In reinforcement learning, two objective functions have been developed extensively in the literature: discounted and averaged rewards. The generalization to an entropy-regularized setting has led to improved robustness and exploration for both of these objectives. Recently, the entropy-regularized average-reward problem was addressed using tools from large deviation theory in the tabular setting. This method has the advantage of linearity, providing access to both the optimal policy and average reward-rate through properties of a single matrix. In this paper, we extend that framework to more general settings by developing approaches based on function approximation by neural networks. This formulation reveals new theoretical insights into the relationship between different objectives used in RL. Additionally, we combine our algorithm with a posterior policy iteration scheme, showing how our approach can also solve the average-reward RL problem without entropy-regularization. Using classic control benchmarks, we experimentally find that our method compares favorably with other algorithms in terms of stability and rate of convergence.

cs.LG↗

arXiv2024

Boosting Soft Q-Learning by Bounding

An agent's ability to leverage past experience is critical for efficiently solving new tasks. Prior work has focused on using value function estimates to obtain zero-shot approximations for solutions to a new task. In soft Q-learning, we show how any value function estimate can also be used to derive double-sided bounds on the optimal value function. The derived bounds lead to new approaches for boosting training performance which we validate experimentally. Notably, we find that the proposed framework suggests an alternative method for updating the Q-function, leading to boosted performance.

cs.LG↗

arXiv2023

Bounding the Optimal Value Function in Compositional Reinforcement Learning

In the field of reinforcement learning (RL), agents are often tasked with solving a variety of problems differing only in their reward functions. In order to quickly obtain solutions to unseen problems with new reward functions, a popular approach involves functional composition of previously solved tasks. However, previous work using such functional composition has primarily focused on specific instances of composition functions whose limiting assumptions allow for exact zero-shot composition. Our work unifies these examples and provides a more general framework for compositionality in both standard and entropy-regularized RL. We find that, for a broad class of functions, the optimal solution for the composite task of interest can be related to the known primitive task solutions. Specifically, we present double-sided inequalities relating the optimal composite value function to the value functions for the primitive tasks. We also show that the regret of using a zero-shot policy can be bounded for this class of functions. The derived bounds can be used to develop clipping approaches for reducing uncertainty during training, allowing agents to quickly adapt to new tasks.

cs.LG↗

arXiv2023

Entropy Regularized Reinforcement Learning Using Large Deviation Theory

Reinforcement learning (RL) is an important field of research in machine learning that is increasingly being applied to complex optimization problems in physics. In parallel, concepts from physics have contributed to important advances in RL with developments such as entropy-regularized RL. While these developments have led to advances in both fields, obtaining analytical solutions for optimization in entropy-regularized RL is currently an open problem. In this paper, we establish a mapping between entropy-regularized RL and research in non-equilibrium statistical mechanics focusing on Markovian processes conditioned on rare events. In the long-time limit, we apply approaches from large deviation theory to derive exact analytical results for the optimal policy and optimal dynamics in Markov Decision Process (MDP) models of reinforcement learning. The results obtained lead to a novel analytical and computational framework for entropy-regularized RL which is validated by simulations. The mapping established in this work connects current research in reinforcement learning and non-equilibrium statistical mechanics, thereby opening new avenues for the application of analytical and computational approaches from one field to cutting-edge problems in the other.

cs.LG↗

arXiv2022

Utilizing Prior Solutions for Reward Shaping and Composition in Entropy-Regularized Reinforcement Learning

In reinforcement learning (RL), the ability to utilize prior knowledge from previously solved tasks can allow agents to quickly solve new problems. In some cases, these new problems may be approximately solved by composing the solutions of previously solved primitive tasks (task composition). Otherwise, prior knowledge can be used to adjust the reward function for a new problem, in a way that leaves the optimal policy unchanged but enables quicker learning (reward shaping). In this work, we develop a general framework for reward shaping and task composition in entropy-regularized RL. To do so, we derive an exact relation connecting the optimal soft value functions for two entropy-regularized RL problems with different reward functions and dynamics. We show how the derived relation leads to a general result for reward shaping in entropy-regularized RL. We then generalize this approach to derive an exact relation connecting optimal value functions for the composition of multiple tasks in entropy-regularized RL. We validate these theoretical contributions with experiments showing that reward shaping and task composition lead to faster learning in various settings.

cs.LG↗

arXiv2019

Stochastic modeling of phenotypic switching and chemoresistance in cancer cell populations

Phenotypic heterogeneity in cancer cells is widely observed and is often linked to drug resistance. In several cases, such heterogeneity in drug sensitivity of tumors is driven by stochastic and reversible acquisition of a drug tolerant phenotype by individual cells even in an isogenic population. Accumulating evidence further suggests that cell-fate transitions such as the epithelial to mesenchymal transition (EMT) are associated with drug resistance. In this study, we analyze stochastic models of phenotypic switching to provide a framework for analyzing cell-fate transitions such as EMT as a source of phenotypic variability in drug sensitivity. Motivated by our cell-culture based experimental observations connecting phenotypic switching in EMT and drug resistance, we analyze a coarse-grained model of phenotypic switching between two states in the presence of cytotoxic stress from chemotherapy. We derive analytical results for time-dependent probability distributions that provide insights into the rates of phenotypic switching and characterize initial phenotypic heterogeneity of cancer cells. The results obtained can also shed light on fundamental questions relating to adaptation and selection scenarios in tumor response to cytotoxic therapy.

q-bio.MN↗

arXiv2017

Stochastic gene expression conditioned on large deviations

The intrinsic stochasticity of gene expression can give rise to large fluctuations and rare events that drive phenotypic variation in a population of genetically identical cells. Characterizing the fluctuations that give rise to such rare events motivates the analysis of large deviations in stochastic models of gene expression. Recent developments in non-equilibrium statistical mechanics have led to a framework for analyzing Markovian processes conditioned on rare events and for representing such processes by conditioning-free driven Markovian processes. We use this framework, in combination with approaches based on queueing theory, to analyze a general class of stochastic models of gene expression. Modeling gene expression as a Batch Markovian Arrival Process (BMAP), we derive exact analytical results quantifying large deviations of time-integrated random variables such as promoter activity fluctuations. We find that the conditioning-free driven process can also be represented by a BMAP that has the same form as the original process, but with renormalized parameters. The results obtained can be used to quantify the likelihood of large deviations, to characterize system fluctuations conditional on rare events and to identify combinations of model parameters that can give rise to dynamical phase transitions in system dynamics.

cond-mat.stat-mech↗

arXiv2014

Transcriptional bursting in gene expression: analytical results for general stochastic models

Gene expression in individual cells is highly variable and sporadic, often resulting in the synthesis of mRNAs and proteins in bursts. Bursting in gene expression is known to impact cell-fate in diverse systems ranging from latency in HIV-1 viral infections to cellular differentiation. It is generally assumed that bursts are geometrically distributed and that they arrive according to a Poisson process. On the other hand, recent single-cell experiments provide evidence for complex burst arrival processes, highlighting the need for more general stochastic models. To address this issue, we invoke a mapping between general models of gene expression and systems studied in queueing theory to derive exact analytical expressions for the moments associated with mRNA/protein steady-state distributions. These moments are then used to derive explicit conditions, based entirely on experimentally measurable quantities, that determine if the burst distributions deviate from the geometric distribution or if burst arrival deviates from a Poisson process. For non-Poisson arrivals, we develop approaches for accurate estimation of burst parameters.

q-bio.MN↗

arXiv2014

Exact distributions for stochastic gene expression models with bursting and feedback

Stochasticity in gene expression can give rise to fluctuations in protein levels and lead to phenotypic variation across a population of genetically identical cells. Recent experiments indicate that bursting and feedback mechanisms play important roles in controlling noise in gene expression and phenotypic variation. A quantitative understanding of the impact of these factors requires analysis of the corresponding stochastic models. However, for stochastic models of gene expression with feedback and bursting, exact analytical results for protein distributions have not been obtained so far. Here, we analyze a model of gene expression with bursting and feedback regulation and obtain exact results for the corresponding protein steady-state distribution. The results obtained provide new insights into the role of bursting and feedback in noise regulation and optimization. Furthermore, for a specific choice of parameters, the system studied maps on to a two-state biochemical switch driven by a bursty input noise source. The analytical results derived thus provide quantitative insights into diverse cellular processes involving noise in gene expression and biochemical switching.

q-bio.MN↗

arXiv2013

Exact protein distributions for stochastic models of gene expression using partitioning of Poisson processes

Stochasticity in gene expression gives rise to fluctuations in protein levels across a population of genetically identical cells. Such fluctuations can lead to phenotypic variation in clonal populations, hence there is considerable interest in quantifying noise in gene expression using stochastic models. However, obtaining exact analytical results for protein distributions has been an intractable task for all but the simplest models. Here, we invoke the partitioning property of Poisson processes to develop a mapping that significantly simplifies the analysis of stochastic models of gene expression. The mapping leads to exact protein distributions using results for mRNA distributions in models with promoter-based regulation. Using this approach, we derive exact analytical results for steady-state and time-dependent distributions for the basic 2-stage model of gene expression. Furthermore, we show how the mapping leads to exact protein distributions for extensions of the basic model that include the effects of post-transcriptional and post-translational regulation. The approach developed in this work is widely applicable and can contribute to a quantitative understanding of stochasticity in gene expression and its regulation.

q-bio.MN↗

arXiv2011

On the structural properties of small-world networks with finite range of shortcut links

We explore a new variant of Small-World Networks (SWNs), in which an additional parameter ($r$) sets the length scale over which shortcuts are uniformly distributed. When $r=0$ we have an ordered network, whereas $r=1$ corresponds to the original SWN model. These short-range SWNs have a similar degree distribution and scaling properties as the original SWN model. We observe the small-world phenomenon for $r \ll 1$ indicating that global shortcuts are not necessary for the small-world effect. For short-range SWNs, the average path length changes nonmonotonically with system size, whereas for the original SWN model it increases monotonically. We propose an expression for the average path length for short-range SWNs based on numerical simulations and analytical approximations.

physics.soc-ph↗

arXiv2011

Connecting protein and mRNA burst distributions for stochastic models of gene expression

The intrinsic stochasticity of gene expression can lead to large variability in protein levels for genetically identical cells. Such variability in protein levels can arise from infrequent synthesis of mRNAs which in turn give rise to bursts of protein expression. Protein expression occurring in bursts has indeed been observed experimentally and recent studies have also found evidence for transcriptional bursting, i.e. production of mRNAs in bursts. Given that there are distinct experimental techniques for quantifying the noise at different stages of gene expression, it is of interest to derive analytical results connecting experimental observations at different levels. In this work, we consider stochastic models of gene expression for which mRNA and protein production occurs in independent bursts. For such models, we derive analytical expressions connecting protein and mRNA burst distributions which show how the functional form of the mRNA burst distribution can be inferred from the protein burst distribution. Additionally, if gene expression is repressed such that observed protein bursts arise only from single mRNAs, we show how observations of protein burst distributions (repressed and unrepressed) can be used to completely determine the mRNA burst distribution. Assuming independent contributions from individual bursts, we derive analytical expressions connecting means and variances for burst and steady-state protein distributions. Finally, we validate our general analytical results by considering a specific reaction scheme involving regulation of protein bursts by small RNAs. For a range of parameters, we derive analytical expressions for regulated protein distributions that are validated using stochastic simulations. The analytical results obtained in this work can thus serve as useful inputs for a broad range of studies focusing on stochasticity in gene expression.

physics.bio-ph↗

arXiv2011

Regulation by small RNAs via coupled degradation: mean-field and variational approaches

Regulatory genes called small RNAs (sRNAs) are known to play critical roles in cellular responses to changing environments. For several sRNAs, regulation is effected by coupled stoichiometric degradation with messenger RNAs (mRNAs). The nonlinearity inherent in this regulatory scheme indicates that exact analytical solutions for the corresponding stochastic models are intractable. Here, we present a variational approach to analyze a well-studied stochastic model for regulation by sRNAs via coupled degradation. The proposed approach is efficient and provides accurate estimates of mean mRNA levels as well as higher order terms. Results from the variational ansatz are in excellent agreement with data from stochastic simulations for a wide range of parameters, including regions of parameter space where mean-field approaches break down. The proposed approach can be applied to quantitatively model stochastic gene expression in complex regulatory networks.

physics.bio-ph↗

arXiv2011

Stochastic modeling of regulation of gene expression by multiple small RNAs

A wealth of new research has highlighted the critical roles of small RNAs (sRNAs) in diverse processes such as quorum sensing and cellular responses to stress. The pathways controlling these processes often have a central motif comprising of a master regulator protein whose expression is controlled by multiple sRNAs. However, the regulation of stochastic gene expression of a single target gene by multiple sRNAs is currently not well understood. To address this issue, we analyze a stochastic model of regulation of gene expression by multiple sRNAs. For this model, we derive exact analytic results for the regulated protein distribution including compact expressions for its mean and variance. The derived results provide novel insights into the roles of multiple sRNAs in fine-tuning the noise in gene expression. In particular, we show that, in contrast to regulation by a single sRNA, multiple sRNAs provide a mechanism for independently controlling the mean and variance of the regulated protein distribution.

q-bio.MN↗