arXiv subjects

Jiashuo Jiang

Publications and source records attributed to Jiashuo Jiang.

At least 19 recordsLinked to original sources

arXiv2026

Pareto-Aware Hierarchical Reinforcement Learning for Online Resource Allocation in RIS-assisted Large-Scale IoT Systems

With the rapid evolution of 5G and emerging 6G networks, reconfigurable intelligent surfaces (RIS) have become a critical technology for enhancing wireless communication scenarios. However, optimizing RIS-assisted multi-user systems typically introduces high-dimensional physical layer variables and non-convex Pareto-optimal rate sets, posing severe computational challenges for real-time applications. To address these limitations, this paper proposes a dimension-reduced, hierarchical reinforcement learning (RL) framework, termed Pareto-aware autoencoder-assisted RL (PAAERL), to optimize online resource allocation in RIS-assisted Internet of Things (IoT) networks. Our approach first substitutes high-dimensional continuous RIS beamforming variables with lower-dimensional weight vectors that strictly represent the Pareto-optimal frontier, theoretically avoiding geometric information loss across both convex and non-convex rate regions. To further mitigate the curse of dimensionality in dense networks, an autoencoder architecture is integrated to execute a secondary, data-driven compression phase, mapping the priority space into a highly condensed continuous latent action space. Extensive simulations conducted across practical communication scenarios, including multi-user mobile edge computing (MEC) networks, demonstrate that the proposed PAAERL framework drastically reduces offline training times, accelerates online policy convergence, and significantly decreases overall network costs compared to state-of-the-art benchmarks, underscoring its exceptional scalability and practical viability for next-generation intelligent IoT environments.

cs.NI↗

arXiv2026

Competitive Analysis of Stock-based Thresholds via Prophet Inequalities in Continuous Time

We study a continuous-time $K$-unit online resource allocation problem with nonhomogeneous Poisson arrivals and time-varying valuation distributions. While the optimal dynamic policy generally depends on both the remaining inventory and the time left in the horizon, we focus on a simpler and practically appealing class of stock-based threshold policies, whose limited number of thresholds depend only on the number of units remaining. We evaluate these policies against the multi-unit prophet benchmark, which selects the best $K$ realized values in hindsight. Our main contribution is a new competitive-analysis framework for stock-based thresholds in continuous time. We first reformulate the problem through a type-covering dual. The central challenge for analyzing the dual is that the dual is both infinite-dimensional and non-convex: the adversary can choose time-varying arrival and valuation processes, while the policy performance depends nonlinearly on the stochastic inventory trajectory. We overcome these challenges by reducing the continuous-time adversarial problem to a Poisson optimization $PoisOPTRe_K$, and then proving sharp structural properties of its worst-case solutions. In particular, adversarial arrivals admit cutoff and late-filling structures, which yield an exact four-parameter formulation for two thresholds and a finite nested-interval representation for general thresholds. These reductions make the guarantees directly computable. For example, for two thresholds, we obtain a competitive ratio $0.6269$ for $K=2$; with three thresholds, we obtain the ratio $0.6816$ for $K=3$. In this way, we show that simple stock-based thresholds achieve strong prophet-inequality guarantees despite ignoring calendar time.

math.OC↗

arXiv2026

Cross-Epoch Adaptive Rollout Optimization for RL Post-Training

LLM post-training often relies on reinforcement learning methods that sample multiple rollouts per prompt, yet most existing approaches use a fixed rollout budget for every prompt, despite large differences in the training signal different prompts provide. In this paper, we study adaptive rollout allocation under a fixed global budget and formulate the problem as online resource allocation with prompt-level diminishing returns. Our method, CERO, maintains a Beta posterior over each prompt's success probability and uses the posterior expected Bernoulli variance as a Bayesian estimate of the value of additional rollouts. We use this estimate to construct a concave, saturating utility over cumulative allocations, yielding an objective in which decisions across prompts and epochs are coupled by the global budget. Since the resulting objective is temporally nonseparable, we derive a Fenchel-dual reformulation and update both prompt-level and budget-level dual variables via projected online gradient descent. Under fixed prompt utilities, we prove an $O(\sqrt{K})$ regret bound against the offline allocation benchmark. Experiments on mathematical-reasoning problems show that CERO consistently outperforms GRPO across multiple open-weight LLMs and benchmarks, demonstrating that adaptive rollout budgeting can improve sample efficiency.

cs.LG↗

arXiv2026

Adaptive Inference for Resource-Constrained Dynamic Pricing

We study dynamic pricing over a finite selling horizon when limited resource capacity determines revenue and the observations available for inference at a prespecified price. Resource depletion can remove the target neighborhood from the feasible price set, changing the experiment generated by the pricing policy. We develop inference-aware re-solving controllers that check target-band feasibility before current covariates arrive and log the pricing mixture. Target-reserved and smooth controllers take population mean-pair geometry as a predeployment input; learned barycentric re-solving instead estimates stationary mean-consumption vectors of predeclared component kernels. On an affine binding-capacity family, an exact-input target-reserved controller assigning mass $t^{-\gamma}$ obtains an information clock of order $T^{1-\gamma}$ in probability, radius $O_p\{T^{-(1-\gamma)/2}\}$, and, under an exposed-face reward identity, a signed fluid-benchmark gap bounded above by $O(\log T+T^{1-\gamma})$. Under the exogenous affine-face condition, predeclared target support, and polynomial error spending with exponent greater than one, learned barycentric re-solving has a linear information clock in probability and an $O(\log T)$ signed-gap upper bound; centered local pricing has the same orders under slack capacity and global target optimality. An exact-input, target-compatible smooth alternative without reservation gives a linear clock in probability, an $O_p(T^{-1/2})$ deterministic-envelope radius with unconditional coverage and reporting probability tending to one, and an $O(\log^2 T)$ signed-gap upper bound. Boundary results show when physical support is lost and why a $1/t$ target branch yields only $O_p(1)$ information if it is the sole target-local source. The policy reports an interval when its prespecified support and information conditions hold and otherwise abstains.

stat.ML↗

arXiv2026

Adaptive Bidding Policies for First-Price Auctions with Budget Constraints under Non-stationarity

In this paper, we study how a budget-constrained bidder should learn to bid adaptively in repeated first-price auctions to maximize cumulative payoff. This problem arises from the recent industry-wide shift from second-price auctions to first-price auctions in display advertising, which renders truthful bidding suboptimal. We propose a simple dual-gradient-descent-based bidding policy that maintains a dual variable for the budget constraint as the bidder consumes the budget. We analyze two settings based on the bidder's knowledge of future private values: (i) an uninformative setting where all distributional knowledge (potentially non-stationary) is entirely unknown, and (ii) an informative setting where a prediction of budget allocation is available in advance. We characterize the performance loss (regret) relative to an optimal policy with complete information. For uninformative setting, we show that the regret is ~O(sqrt(T)) plus a Wasserstein-based variation term capturing non-stationarity, which is order-optimal. In the informative setting, the variation term can be eliminated using predictions, yielding a regret of ~O(sqrt(T)) plus the prediction error. Furthermore, we go beyond the global budget constraint by introducing a refined benchmark based on a per-period budget allocation plan, achieving exactly ~O(sqrt(T)) regret. We also establish robustness guarantees when the baseline policy deviates from the planned allocation, covering both ideal and adversarial deviations.

cs.GT↗

arXiv2026

The Value of Information in Resource-Constrained Pricing

Firms that price perishable resources -- airline seats, hotel rooms, seasonal inventory -- now routinely use demand predictions, but these predictions vary widely in quality. Under hard capacity constraints, acting on an inaccurate prediction can irreversibly deplete inventory needed for future periods. We study how prediction uncertainty propagates into dynamic pricing decisions with linear demand, stochastic noise, and finite capacity. A certified demand forecast with known error bound~$ε^0$ specifies where the system should operate: it shifts regret from $O(\sqrt{T})$ to $O(\log T)$ when $ε^0 \lesssim T^{-1/4}$, and we prove this threshold is tight. A misspecified surrogate model -- biased but correlated with true demand -- cannot set prices directly but reduces learning variance by a factor of $(1-ρ^2)$ through control variates. The two mechanisms compose: the forecast determines the regret regime; the surrogate tightens estimation within it. All algorithms rest on a boundary attraction mechanism that stabilizes pricing near degenerate capacity boundaries without requiring non-degeneracy assumptions. Experiments confirm the phase transition threshold, the variance reduction from surrogates, and robustness across problem instances.

math.OC↗

arXiv2026

Online Semi-infinite Linear Programming: Efficient Algorithms via Function Approximation

We consider the dynamic resource allocation problem where the decision space is finite-dimensional, yet the solution must satisfy a large or even infinite number of constraints revealed via streaming data or oracle feedback. We model this challenge as an Online Semi-infinite Linear Programming (OSILP) problem and develop a novel LP formulation to solve it approximately. Specifically, we employ function approximation to reduce the number of constraints to a constant $q$. This addresses a key limitation of traditional online LP algorithms, whose regret bounds typically depend on the number of constraints, leading to poor performance in this setting. We propose a dual-based algorithm to solve our new formulation, which offers broad applicability through the selection of appropriate potential functions. We analyze this algorithm under two classical input models-stochastic input and random permutation-establishing regret bounds of $O(q\sqrt{T})$ and $O\left(\left(q+q\log{T})\sqrt{T}\right)\right)$ respectively. Note that both regret bounds are independent of the number of constraints, which demonstrates the potential of our approach to handle a large or infinite number of constraints. Furthermore, we investigate the potential to improve upon the $O(q\sqrt{T})$ regret and propose a two-stage algorithm, achieving $O(q\log{T} + q/ε)$ regret under more stringent assumptions. We also extend our algorithms to the general function setting. A series of experiments validates that our algorithms outperform existing methods when confronted with a large number of constraints.

cs.LG↗

arXiv2026

Online Bidding for Contextual First-Price Auctions with Budgets under One-Sided Information Feedback

In this paper, we study the problem of learning to bid in repeated first-price auctions with budget constraints. In each period, the decision maker needs to submit a bid to win the auction and maximize the total collected reward, subject to a budget constraint throughout the horizon. We focus on the setting with one-sided information feedback where only the winning bid is revealed to the decision maker at each period. Different from previous papers that assume homogeneous competitors' bids, we assume that the highest bid of other bidders depends on the context of the impression, which is initially unknown and needs to be learned over time. To tackle the learning difficulty, we propose a novel robust regression method based on conditional quantile invariance to learn the contextual parameter. Further combined with a dual update procedure, we develop a new bidding algorithm and prove that our algorithm achieves $\widetilde{O}(\sqrt{T})$ regret, which is order-optimal. We further extend our approach to the multi-dimensional setting and demonstrate the practical efficiency of our algorithm through numerical experiments.

math.OC↗

arXiv2026

Online Order Fulfillment with Replenishment

In modern e-commerce and service operations, firms must jointly manage inventory replenishment and real-time order fulfillment to maximize profit under demand uncertainty. While each component has been studied extensively in isolation, their interaction remains underexplored. This paper investigates a fundamental operational question: which lever plays a more decisive role in overall system performance, replenishment or fulfillment? We model the system as a one-location online order fulfillment problem with lost sales and stochastic customer arrivals, each offering heterogeneous rewards. Replenishment follows either a base-stock or constant-order policy, while real-time fulfillment decisions are made using online algorithms. Our core performance metric is the expected average profit per replenishment cycle, evaluated across all combinations of these policies and algorithms. Our main theoretical result shows that when the replenishment cycle is long, the cumulative regret of online fulfillment remains of the same order as in a corresponding single-cycle problem, even under repeated replenishment, revealing a form of regret stability. This phenomenon also extends to a multi-location setting. We further develop a regret-based framework that quantitatively compares the value of improving replenishment versus improving fulfillment, and we characterize regimes in which optimizing replenishment yields a larger revenue impact than refining the online fulfillment algorithm (and vice versa). Motivated by examples where myopic algorithms underperform, we introduce a novel look-ahead online algorithm that anticipates future replenishment and demand. Numerical experiments verify that this algorithm outperforms myopic baselines. Overall, our results provide both theoretical and managerial insights into situations where inventory replenishment policies are more influential and vice versa.

math.OC↗

arXiv2026

Non-Stationary Online Resource Allocation: Learning from a Single Sample

We study online resource allocation under non-stationary demand with a minimum offline data requirement. In this problem, a decision-maker must allocate multiple types of resources to sequentially arriving queries over a finite horizon. Each query belongs to a finite set of types with fixed resource consumption and a stochastic reward drawn from an unknown, type-specific distribution. Critically, the environment exhibits arbitrary non-stationarity -- arrival distributions may shift unpredictably-while the algorithm requires only one historical sample per period to operate effectively. We distinguish two settings based on sample informativeness: (i) reward-observed samples containing both query type and reward realization, and (ii) the more challenging type-only samples revealing only query type information. We propose a novel type-dependent quantile-based meta-policy that decouples the problem into modular components: reward distribution estimation, optimization of target service probabilities via fluid relaxation, and real-time decisions through dynamic acceptance thresholds. For reward-observed samples, our static threshold policy achieves $\tilde{O}(\sqrt{T})$ regret. For type-only samples, we first establish that sublinear regret is impossible without additional structure; under a mild minimum-arrival-probability assumption, we design both a partially adaptive policy attaining the same $\tilde{O}({T})$ bound and, more significantly, a fully adaptive resolving policy with careful rounding that achieves the first poly-logarithmic regret guarantee of $O((\log T)^3)$ for non-stationary multi-resource allocation. Our framework advances prior work by operating with minimal offline data (one sample per period), handling arbitrary non-stationarity without variation-budget assumptions, and supporting multiple resource constraints.

cs.LG↗

arXiv2026

An LP-Based Approach for Bilinear Saddle Point Problem with Instance-dependent Guarantee and Noisy Feedback

In this work, we study the sample complexity of obtaining a Nash equilibrium (NE) estimate in two-player zero-sum matrix games with noisy feedback. Specifically, we propose a novel algorithm that repeatedly solves linear programs (LPs) to obtain an NE estimate with bias at most $\varepsilon$ with a sample complexity of $O\left(\frac{m_1 m_2}{\varepsilon\min\{δ^2,σ_0^2,σ^3\}} \log\frac{m_1 m_2}{\varepsilon}\right)$ for general $m_1 \times m_2$ game matrices, where $σ$, $σ_0$, $δ$ are some problem-dependent constants. To our knowledge, this is the first instance-dependent sample complexity bound for finding an NE estimate with $\varepsilon$ bias in general-dimension matrix games with noisy feedback and potentially non-unique equilibria. Our algorithm builds on recent advances in online resource allocation and operates in two stages: (1) identifying the support set of an NE, and (2) computing the unique NE restricted to this support. Both stages rely on a careful analysis of LP solutions derived from noisy samples.

math.OC↗

arXiv2026

Decentralized Multi-product Pricing: Diagonal Dominance, Nash Equilibrium, and Price of Anarchy

Decentralized decision making in multi--product firms can lead to efficiency losses when autonomous decision makers fail to internalize cross--product demand interactions. This paper quantifies the magnitude of such losses by analyzing the Price of Anarchy in a pricing game in which each decision maker independently sets prices to maximize its own product--level revenue. We model demand using a linear system that captures both substitution and complementarity effects across products. We first establish existence and uniqueness of a pure--strategy Nash equilibrium under economically standard diagonal dominance conditions. Our main contribution is the derivation of a tight worst--case lower bound on the ratio between decentralized revenue and the optimal centralized revenue. We show that this efficiency loss is governed by a single scalar parameter, denoted by $μ$, which measures the aggregate strength of cross--price effects relative to own--price sensitivities. In particular, we prove that the revenue ratio is bounded below by $4(1-μ)/(2-μ)^2$, and we demonstrate the tightness of this bound by constructing a symmetric market topology in which the bound is exactly attained. We further refine the analysis by providing an instance--exact characterization of efficiency loss based on the spectral properties of the demand interaction matrix. Together, these results offer a quantitative framework for assessing the trade--off between centralized pricing and decentralized autonomy in multi--product firms.

cs.GT↗

arXiv2026

Online Scheduling for LLM Inference with KV Cache Constraints

Large Language Model (LLM) inference, where a trained model generates text one word at a time in response to user prompts, is a computationally intensive process requiring efficient scheduling to optimize latency and resource utilization. A key challenge in LLM inference is the management of the Key-Value (KV) cache, which reduces redundant computations but introduces memory constraints. In this work, we model LLM inference with KV cache constraints theoretically and propose a novel batching and scheduling algorithm that minimizes inference latency while effectively managing the KV cache's memory. More specifically, we make the following contributions. First, to evaluate the performance of online algorithms for scheduling in LLM inference, we introduce a hindsight optimal benchmark, formulated as an integer program that computes the minimum total inference latency under full future information. Second, we prove that no deterministic online algorithm can achieve a constant competitive ratio when the arrival process is arbitrary. Third, motivated by the computational intractability of solving the integer program at scale, we propose a polynomial-time online scheduling algorithm and show that under certain conditions it can achieve a constant competitive ratio. We also demonstrate our algorithm's strong empirical performance by comparing it to the hindsight optimal in a synthetic dataset. Finally, we conduct empirical evaluations on a real-world public LLM inference dataset, simulating the Llama2-70B model on A100 GPUs, and show that our algorithm significantly outperforms the benchmark algorithms. Overall, our results offer a path toward more sustainable and cost-effective LLM deployment.

cs.LG↗

arXiv2025

Ask, Clarify, Optimize: Human-LLM Agent Collaboration for Smarter Inventory Control

Inventory management remains a challenge for many small and medium-sized businesses that lack the expertise to deploy advanced optimization methods. This paper investigates whether Large Language Models (LLMs) can help bridge this gap. We show that employing LLMs as direct, end-to-end solvers incurs a significant "hallucination tax": a performance gap arising from the model's inability to perform grounded stochastic reasoning. To address this, we propose a hybrid agentic framework that strictly decouples semantic reasoning from mathematical calculation. In this architecture, the LLM functions as an intelligent interface, eliciting parameters from natural language and interpreting results while automatically calling rigorous algorithms to build the optimization engine. To evaluate this interactive system against the ambiguity and inconsistency of real-world managerial dialogue, we introduce the Human Imitator, a fine-tuned "digital twin" of a boundedly rational manager that enables scalable, reproducible stress-testing. Our empirical analysis reveals that the hybrid agentic framework reduces total inventory costs by 32.1% relative to an interactive baseline using GPT-4o as an end-to-end solver. Moreover, we find that providing perfect ground-truth information alone is insufficient to improve GPT-4o's performance, confirming that the bottleneck is fundamentally computational rather than informational. Our results position LLMs not as replacements for operations research, but as natural-language interfaces that make rigorous, solver-based policies accessible to non-experts.

cs.AI↗

arXiv2025

Achieving Instance-dependent Sample Complexity for Constrained Markov Decision Process

We consider the reinforcement learning problem for the constrained Markov decision process (CMDP), which plays a central role in satisfying safety or resource constraints in sequential learning and decision-making. In this problem, we are given finite resources and a MDP with unknown transition probabilities. At each stage, we take an action, collecting a reward and consuming some resources, all assumed to be unknown and need to be learned over time. In this work, we take the first step towards deriving optimal problem-dependent guarantees for the CMDP problems. We derive a logarithmic regret bound, which translates into a $O(\frac{1}{Δ\cdotε}\cdot\log^2(1/ε))$ sample complexity bound, with $Δ$ being a problem-dependent parameter, yet independent of $ε$. Our sample complexity bound improves upon the state-of-art $O(1/ε^2)$ sample complexity for CMDP problems established in the previous literature, in terms of the dependency on $ε$. To achieve this advance, we develop a new framework for analyzing CMDP problems. To be specific, our algorithm operates in the primal space and we resolve the primal LP for the CMDP problem at each period in an online manner, with adaptive remaining resource capacities. The key elements of our algorithm are: i) a characterization of the instance hardness via LP basis, ii) an eliminating procedure that identifies one optimal basis of the primal LP, and; iii) a resolving procedure that is adaptive to the remaining resources and sticks to the characterized optimal basis.

cs.LG↗

arXiv2025

A Lyapunov Drift-Plus-Penalty Method Tailored for Reinforcement Learning with Queue Stability

With the proliferation of Internet of Things (IoT) devices, the demand for addressing complex optimization challenges has intensified. The Lyapunov Drift-Plus-Penalty algorithm is a widely adopted approach for ensuring queue stability, and some research has preliminarily explored its integration with reinforcement learning (RL). In this paper, we investigate the adaptation of the Lyapunov Drift-Plus-Penalty algorithm for RL applications, deriving an effective method for combining Lyapunov Drift-Plus-Penalty with RL under a set of common and reasonable conditions through rigorous theoretical analysis. Unlike existing approaches that directly merge the two frameworks, our proposed algorithm, termed Lyapunov drift-plus-penalty method tailored for reinforcement learning with queue stability (LDPTRLQ) algorithm, offers theoretical superiority by effectively balancing the greedy optimization of Lyapunov Drift-Plus-Penalty with the long-term perspective of RL. Simulation results for multiple problems demonstrate that LDPTRLQ outperforms the baseline methods using the Lyapunov drift-plus-penalty method and RL, corroborating the validity of our theoretical derivations. The results also demonstrate that our proposed algorithm outperforms other benchmarks in terms of compatibility and stability.

cs.LG↗

arXiv2025

Regret Minimization and Statistical Inference in Online Decision Making with High-dimensional Covariates

This paper investigates regret minimization, statistical inference, and their interplay in high-dimensional online decision-making based on the sparse linear context bandit model. We integrate the $\varepsilon$-greedy bandit algorithm for decision-making with a hard thresholding algorithm for estimating sparse bandit parameters and introduce an inference framework based on a debiasing method using inverse propensity weighting. Under a margin condition, our method achieves either $O(T^{1/2})$ regret or classical $O(T^{1/2})$-consistent inference, indicating an unavoidable trade-off between exploration and exploitation. If a diverse covariate condition holds, we demonstrate that a pure-greedy bandit algorithm, i.e., exploration-free, combined with a debiased estimator based on average weighting can simultaneously achieve optimal $O(\log T)$ regret and $O(T^{1/2})$-consistent inference. We also show that a simple sample mean estimator can provide valid inference for the optimal policy's value. Numerical simulations and experiments on Warfarin dosing data validate the effectiveness of our methods.

cs.LG↗

arXiv2025

Adaptive Resolving Methods for Markov Decision Processes with Function Approximations

Learning the optimal policy for Markov decision process problems (MDPs) from samples is a fundamental problem in online and data-driven decision-making. Function approximations are usually deployed to handle large or infinite state-action space. In our work, we consider the MDP problems with function approximation and we develop a new algorithm to solve it efficiently. Our algorithm is based on a linear programming (LP) reformulation and repeatedly resolves the identified reduced linear system as new transition samples arrive. After the optimal basis is identified, we show that, after $N$ resolving rounds, the expected averaged iterate achieves an instance-dependent $\widetilde O(C_{\mathrm{inst}}/N)$ objective shortfall and signed constraint residual. We separately account for the historical samples used for basis identification and the $d_2$ transition queries used in each resolving round, which yields the corresponding total transition-query complexity. We further complement our result with a \textit{robust} $O(1/\sqrt{N})$ bound that is independent of $\Delta$. In comparison to the guarantees established in the previous literature, our instance dependent guarantee is tighter when the underlying instance is favorable, and the numerical experiments also reveal the wide applications and efficient empirical performances of our algorithms.

cs.LG↗