arXiv subjects

Peter W. Glynn

Publications and source records attributed to Peter W. Glynn.

At least 19 recordsLinked to original sources

arXiv2026

Sequential Batch Learning in Finite-Action Linear Contextual Bandits

We study the sequential batch learning problem in linear contextual bandits with finite action sets, where the decision maker is constrained to split incoming individuals into (at most) a fixed number of batches and can only observe outcomes for the individuals within a batch at the batch's end. Compared with both standard online contextual-bandit learning and offline policy learning in contextual bandits, this sequential batch learning problem provides a finer-grained formulation of many personalized sequential decision making problems in practical applications, including medical treatment in clinical trials, product recommendation in e-commerce and adaptive experiment design in crowdsourcing. We study two settings of the problem: one where the contexts are arbitrarily generated and the other where the context vectors are mutually independent across actions and time and follow a common Gaussian distribution. In each setting, we establish a regret lower bound and provide an algorithm, whose regret upper bound nearly matches the lower bound. As an important insight revealed therefrom, in the former setting, we show that the number of batches required to achieve the fully online performance is polynomial in the time horizon, while for the latter setting, a pure-exploitation algorithm with a judicious batch partition scheme achieves the fully online performance even when the number of batches is less than logarithmic in the time horizon. In the stochastic context setting, we additionally provide tight margin-based (i.e. instance-dependent) upper and lower regret bounds that delineate performance in terms of how difficult the problem instance is. Together, our results provide a near-complete characterization of sequential decision making in linear contextual bandits when batch constraints are present.

cs.LG↗

arXiv2026

Diffusion Approximations for Thompson Sampling in the Small Gap Regime

We study the process-level dynamics of Thompson sampling and related sampling-based bandit algorithms in the ``small gap'' regime, where the gaps between the arm means are of order $\sqrtγ$ or smaller and the time horizon is of order $1/γ$, with $γ\downarrow 0$. In this regime, as $γ\downarrow 0$, we show that the process-level dynamics of such algorithms converge weakly to the solutions to certain stochastic differential equations and stochastic ordinary differential equations. Our weak convergence theory is developed using the Continuous Mapping Theorem, which provides a direct and modular theoretical approach that can be adapted to analyze a variety of sampling-based bandit algorithms and handle weakly dependent reward processes. A central finding is an algorithmic invariance principle: in the small gap regime, the limit dynamics of a broad class of sampling-based algorithms -- including Thompson sampling with general single-parameter exponential family likelihoods, as well as non-parametric bandit algorithms based on bootstrap re-sampling -- all coincide with those of Thompson sampling with Gaussian likelihoods. Moreover, in the small gap regime, the regret performance of these algorithms is generally insensitive to model mis-specification, changing continuously with increasing degrees of mis-specification.

cs.LG↗

arXiv2025

The Law of Large Numbers and CLT for Non-stationary Markov Jump Processes Exhibiting Time-of-Day Effects

In this paper, we develop a general law of large numbers and central limit theorem for cumulative reward processes associated with finite state Markov jump processes with non-stationary transition rates. Such models commonly arise in service operations and manufacturing applications in which time-of-day, day-of-week, and secular effects are of first-order importance in predicting system behavior. Our theorems allow for non-stationary reward environments that continuously accumulate reward, while also including contributions from non-stationary lump-sum rewards of random size that are collected at either jump times of the underlying process, jump times of a Poisson process modulated by the underlying process, or scheduled deterministic times. As part of our development, we also obtain a new central limit theorem for the special case in which the jump process transition rates and reward structure are periodic (as may occur over a weekly time interval), as well as for jump process models with resetting. We include a simulation study illustrating the quality of our CLT approximations for several non-stationary stochastic models.

math.PR↗

arXiv2025

Change-Point Testing for Risk Measures in Time Series

We propose novel methods for change-point testing for nonparametric estimators of expected shortfall and related risk measures in weakly dependent time series. We can detect general multiple structural changes in the tails of marginal distributions of time series under general assumptions. Self-normalization allows us to avoid the issues of standard error estimation. The theoretical foundations for our methods are functional central limit theorems, which we develop under weak assumptions. An empirical study of S&P 500 and US Treasury bond returns illustrates the practical use of our methods in detecting and quantifying instability in the tails of financial time series.

econ.EM↗

arXiv2025

Asymptotic Product-form Steady-state Distribution for Semimartingale Reflecting Brownian Motion in Multi-scaling Regime

Inspired by Dai et al. [2023], we develop a novel multi-scaling asymptotic regime for semimartingale reflecting Brownian motion (SRBM). In this regime, we establish the steady-state convergence of SRBM to a product-form limit with exponentially distributed components by assuming the P-reflection matrix and a uniform moment bound condition. We further demonstrate that the uniform moment bound condition holds in several subclasses of P-matrices. Our proof approach is rooted in the basic adjoint relationship (BAR) for SRBM proposed by Harrison and Williams [1987a].

math.PR↗

arXiv2025

Controllable Coupled Image Generation via Diffusion Models

We provide an attention-level control method for the task of coupled image generation, where "coupled" means that multiple simultaneously generated images are expected to have the same or very similar backgrounds. While backgrounds coupled, the centered objects in the generated images are still expected to enjoy the flexibility raised from different text prompts. The proposed method disentangles the background and entity components in the model's cross-attention modules, attached with a sequence of time-varying weight control parameters depending on the time step of sampling. We optimize this sequence of weight control parameters with a combined objective that assesses how coupled the backgrounds are as well as text-to-image alignment and overall visual quality. Empirical results demonstrate that our method outperforms existing approaches across these criteria.

cs.CV↗

arXiv2025

A Regeneration-based a Posteriori Error Bound for a Markov Chain Stationary Distribution Truncation Algorithm

When the state space of a discrete state space positive recurrent Markov chain is infinite or very large, it becomes necessary to truncate the state space in order to facilitate numerical computation of the stationary distribution. This paper develops a new approach for bounding the truncation error that arises when computing approximations to the stationary distribution. This rigorous a posteriori error bound exploits the regenerative structure of the chain and assumes knowledge of a Lyapunov function. Because the bound is a posteriori (and leverages the computations done to calculate the stationary distribution itself), it tends to be much tighter than a priori bounds. The bound decomposes the regenerative cycle into a random number of excursions from a set $K$ defined in terms of the Lyapunov function into the complement of the truncation set $A$. The bound can be easily computed, and does not (for example) involve a linear program, as do some other error bounds.

math.PR↗

arXiv2025

Approximation of Markov Chain Expectations and the Key Role of Stationary Distribution Convergence

Consider a sequence $P_n$ of positive recurrent transition matrices or kernels that approximate a limiting infinite state matrix or kernel $P_{\infty}$. Such approximations arise naturally when one truncates an infinite state Markov chain and replaces it with a finite state approximation. It also describes the situation in which $P_{\infty}$ is a simplified limiting approximation to $P_n$ when $n$ is large. In both settings, it is often verified that the approximation $P_n$ has the characteristic that its stationary distribution $π_n$ converges to the stationary distribution $π_{\infty}$ associated with the limit. In this paper, we show that when the state space is countably infinite, this stationary distribution convergence implies that $P_n^m$ can be approximated uniformly in $m$ by $P_{\infty}^m$ when n is large. We show that this ability to approximate the marginal distributions at all time scales $m$ fails in continuous state space, but is valid when the convergence is in total variation or when we have weak convergence and the kernels are suitably Lipschitz. When the state space is discrete (as in the truncation setting), we further show that stationary distribution convergence also implies that all the expectations that are computable via first transition analysis (e.g. mean hitting times, expected infinite horizon discounted rewards) converge to those associated with the limit $P_{\infty}$. Simply put, we show that once one has established stationary distribution convergence, one immediately can infer convergence for a huge range of other expectations.

math.PR↗

arXiv2025

Moments of polynomial functionals of spectrally positive Lévy processes

Let $J(\cdot)$ be a compound Poisson process with rate $λ>0$ and a jumps distribution $G(\cdot)$ concentrated on $(0,\infty)$. In addition, let $V$ be a random variable which is distributed according to $G(\cdot)$ and independent from $J(\cdot)$. Define a new process $W(t)\equiv W_V(t)\equiv V+J(t)-t$, $t\geqslant 0$ and let $τ_V$ be the first time that $W(\cdot)$ hits the origin. A long-standing open problem due to Iglehart (1971) and Cohen (1979) is to derive the moments of the functional $\int_0^τW(t)\,{\rm d}t$ in terms of the moments of $G(\cdot)$ and $λ$. In the current work, we solve this problem in much greater generality, i.e., first by letting $J(\cdot)$ belong to a wide class of spectrally positive \color{black} Lévy processes and secondly, by considering more general class of functionals. We also supply several applications of the existing results, e.g., in studying the process $x\mapsto \int_0^{τ_x}W_x(t)\,{\rm d}t$ defined on $x\in[0,\infty)$.

math.PR↗

arXiv2025

Computable Bounds on the Solution to Poisson's Equation for General Harris Chains

Poisson's equation is fundamental to the study of Markov chains, and arises in connection with martingale representations and central limit theorems for additive functionals, perturbation theory for stationary distributions, and average reward Markov decision process problems. In this paper, we develop a new probabilistic representation for the solution of Poisson's equation, and use Lyapunov functions to bound this solution representation explicitly. In contrast to most prior work on this problem, our bounds are computable. Our contribution is closely connected to recent work of Herve and Ledoux (2025), in which they focus their study on a special class of Harris chains satisfying a particular small set condition. However, our theory covers general Harris chains, and often provides a tighter bound. In addition to the new bound and representation, we also develop a computable uniform bound on marginal expectations for Harris chains, and a computable bound on the potential kernel representation of the solution to Poisson's equation.

math.PR↗

arXiv2025

Linear Algebraic Truncation Algorithm with A Posteriori Error Bounds for Computing Markov Chain Equilibrium Gradients

The numerical computation of equilibrium reward gradients for Markov chains appears in many applications for example within the policy improvement step arising in connection with average reward stochastic dynamic programming. When the state space is large or infinite, one will typically need to truncate the state space in order to arrive at a numerically tractable formulation. In this paper, we derive the first computable a posteriori error bounds for equilibrium reward gradients that account for the error induced by the truncation. Our approach uses regeneration to express equilibrium quantities in terms of the expectations of cumulative rewards over regenerative cycles. Lyapunov functions are then used to bound the contributions to these cumulative rewards and their gradients from path excursions that take the chain outside the truncation set. Our numerical results indicate that our approach can provide highly accurate bounds with truncation sets of moderate size. We further extend our approach to Markov jump processes.

math.OC↗

arXiv2024

The Fragility of Optimized Bandit Algorithms

Much of the literature on optimal design of bandit algorithms is based on minimization of expected regret. It is well known that designs that are optimal over certain exponential families can achieve expected regret that grows logarithmically in the number of arm plays, at a rate governed by the Lai-Robbins lower bound. In this paper, we show that when one uses such optimized designs, the regret distribution of the associated algorithms necessarily has a very heavy tail, specifically, that of a truncated Cauchy distribution. Furthermore, for $p>1$, the $p$'th moment of the regret distribution grows much faster than poly-logarithmically, in particular as a power of the total number of arm plays. We show that optimized UCB bandit designs are also fragile in an additional sense, namely when the problem is even slightly mis-specified, the regret can grow much faster than the conventional theory suggests. Our arguments are based on standard change-of-measure ideas, and indicate that the most likely way that regret becomes larger than expected is when the optimal arm returns below-average rewards in the first few arm plays, thereby causing the algorithm to believe that the arm is sub-optimal. To alleviate the fragility issues exposed, we show that UCB algorithms can be modified so as to ensure a desired degree of robustness to mis-specification. In doing so, we also show a sharp trade-off between the amount of UCB exploration and the heaviness of the resulting regret distribution tail.

cs.LG↗

arXiv2024

Online Linear Programming with Batching

We study Online Linear Programming (OLP) with batching. The planning horizon is cut into $K$ batches, and the decisions on customers arriving within a batch can be delayed to the end of their associated batch. Compared with OLP without batching, the ability to delay decisions brings better operational performance, as measured by regret. Two research questions of interest are: (1) What is a lower bound of the regret as a function of $K$? (2) What algorithms can achieve the regret lower bound? These questions have been analyzed in the literature when the distribution of the reward and the resource consumption of the customers have finite support. By contrast, this paper analyzes these questions when the conditional distribution of the reward given the resource consumption is continuous, and we show the answers are different under this setting. When there is only a single type of resource and the decision maker knows the total number of customers, we propose an algorithm with a $O(\log K)$ regret upper bound and provide a $Ω(\log K)$ regret lower bound. We also propose algorithms with $O(\log K)$ regret upper bound for the setting in which there are multiple types of resource and the setting in which customers arrive following a Poisson process. All these regret upper and lower bounds are independent of the length of the planning horizon, and all the proposed algorithms delay decisions on customers arriving in only the first and the last batch. We also take customer impatience into consideration and establish a way of selecting an appropriate batch size.

cs.LG↗

arXiv2024

Minimax Optimal Estimation of Stability Under Distribution Shift

The performance of decision policies and prediction models often deteriorates when applied to environments different from the ones seen during training. To ensure reliable operation, we analyze the stability of a system under distribution shift, which is defined as the smallest change in the underlying environment that causes the system's performance to deteriorate beyond a permissible threshold. In contrast to standard tail risk measures and distributionally robust losses that require the specification of a plausible magnitude of distribution shift, the stability measure is defined in terms of a more intuitive quantity: the level of acceptable performance degradation. We develop a minimax optimal estimator of stability and analyze its convergence rate, which exhibits a fundamental phase shift behavior. Our characterization of the minimax convergence rate shows that evaluating stability against large performance degradation incurs a statistical cost. Empirically, we demonstrate the practical utility of our stability framework by using it to compare system designs on problems where robustness to distribution shift is critical.

stat.ML↗

arXiv2024

When are Unbiased Monte Carlo Estimators More Preferable than Biased Ones?

Due to the potential benefits of parallelization, designing unbiased Monte Carlo estimators, primarily in the setting of randomized multilevel Monte Carlo, has recently become very popular in operations research and computational statistics. However, existing work primarily substantiates the benefits of unbiased estimators at an intuitive level or using empirical evaluations. The intuition being that unbiased estimators can be replicated in parallel enabling fast estimation in terms of wall-clock time. This intuition ignores that, typically, bias will be introduced due to impatience because most unbiased estimators necesitate random completion times. This paper provides a mathematical framework for comparing these methods under various metrics, such as completion time and overall computational cost. Under practical assumptions, our findings reveal that unbiased methods typically have superior completion times - the degree of superiority being quantifiable through the tail behavior of their running time distribution - but they may not automatically provide substantial savings in overall computational costs. We apply our findings to Markov Chain Monte Carlo and Multilevel Monte Carlo methods to identify the conditions and scenarios where unbiased methods have an advantage, thus assisting practitioners in making informed choices between unbiased and biased methods.

stat.CO↗

arXiv2024

A Numerical Truncation Approximation with A Posteriori Error Bounds for the Solution of Poisson's Equation

The solution to Poisson's equation arise in many Markov chain and Markov jump process settings, including that of the central limit theorem, value functions for average reward Markov decision processes, and within the gradient formula for equilibrium Markovian rewards. In this paper, we consider the problem of numerically computing the solution to Poisson's equation when the state space is infinite or very large. In such settings, the state space must be truncated in order to make the problem computationally tractable. In this paper, we provide the first truncation approximation solution to Poisson's equation that comes with provable and computable a posteriori error bounds. Our theory applies to both discrete-time chains and continuous-time jump processes. Through numerical experiments, we show our method can provide highly accurate solutions and tight bounds.

math.PR↗

arXiv2023

On the Maximization of Long-Run Reward CVaR for Markov Decision Processes

This paper studies the optimization of Markov decision processes (MDPs) from a risk-seeking perspective, where the risk is measured by conditional value-at-risk (CVaR). The objective is to find a policy that maximizes the long-run CVaR of instantaneous rewards over an infinite horizon across all history-dependent randomized policies. By establishing two optimality inequalities of opposing directions, we prove that the maximum of long-run CVaR of MDPs over the set of history-dependent randomized policies can be found within the class of stationary randomized policies. In contrast to classical MDPs, we find that there may not exist an optimal stationary deterministic policy for maximizing CVaR. Instead, we prove the existence of an optimal stationary randomized policy that requires randomizing over at most two actions. Via a convex optimization representation of CVaR, we convert the long-run CVaR maximization MDP into a minimax problem, where we prove the interchangeability of minimum and maximum and the related existence of saddle point solutions. Furthermore, we propose an algorithm that finds the saddle point solution by solving two linear programs. These results are then extended to objectives that involve maximizing some combination of mean and CVaR of rewards simultaneously. Finally, we conduct numerical experiments to demonstrate the main results.

math.OC↗

arXiv2023

Overlapping Batch Confidence Intervals on Statistical Functionals Constructed from Time Series: Application to Quantiles, Optimization, and Estimation

We propose a general purpose confidence interval procedure (CIP) for statistical functionals constructed using data from a stationary time series. The procedures we propose are based on derived distribution-free analogues of the $χ^2$ and Student's $t$ random variables for the statistical functional context, and hence apply in a wide variety of settings including quantile estimation, gradient estimation, M-estimation, CVAR-estimation, and arrival process rate estimation, apart from more traditional statistical settings. Like the method of subsampling, we use overlapping batches of time series data to estimate the underlying variance parameter; unlike subsampling and the bootstrap, however, we assume that the implied point estimator of the statistical functional obeys a central limit theorem (CLT) to help identify the weak asymptotics (called OB-x limits, x=I,II,III) of batched Studentized statistics. The OB-x limits, certain functionals of the Wiener process parameterized by the size of the batches and the extent of their overlap, form the essential machinery for characterizing dependence, and consequently the correctness of the proposed CIPs. The message from extensive numerical experimentation is that in settings where a functional CLT on the point estimator is in effect, using \emph{large overlapping batches} alongside OB-x critical values yields confidence intervals that are often of significantly higher quality than those obtained from more generic methods like subsampling or the bootstrap. We illustrate using examples from CVaR estimation, ARMA parameter estimation, and NHPP rate estimation; R and MATLAB code for OB-x critical values is available at~\texttt{web.ics.purdue.edu/~pasupath/}.

math.ST↗