arXiv subjects

Qizhen Jia

Publications and source records attributed to Qizhen Jia.

3 recordsLinked to original sources

arXiv2026

From Relaxed Indexability to Exact Indexability: A $t$-Step Approach for Partially Observable Restless Bandits

Whittle index policies offer a scalable method for restless multi-armed bandits, but under partial observability even determining the indifference subsidy at a single belief requires solving an infinite-horizon belief-state problem with no closed-form value function. Liu [10] addresses this difficulty by linearizing the unknown decision boundary, leading to a linear system and a closed-form approximate Whittle index. However, the resulting threshold uses only a one-step active--passive comparison and does not account for longer-horizon continuation values. We extend this framework to a \emph{$t$-step lookahead threshold policy}. For each subsidy $m$, the threshold is defined by the active-minus-passive advantage under $t$-step finite-horizon value iteration. At $t=1$, the threshold is $m$-independent and recovers the linear threshold of Liu [10]; for $t>1$, it becomes subsidy-dependent through the induced first-crossing structure and tracks the exact decision boundary more closely. The proposed algorithmic framework does not require indexability as an input and includes an indexability verification. Under the original Whittle indexability, we prove that the $t$-step approximate Whittle index converges geometrically to the exact Whittle index, \[ |\widehat W_t(ω)-W(ω)|=O(β^t). \] Numerically, all 2,715 tested three-state instances are verified with computable priority index functions according to the proposed criterion. The P95 index error decreases from $2.18\times10^{-2}$ at $t=1$ to $8.93\times10^{-4}$ at $t=8$. In an exact-comparable instance with $β=0.9999$, $t=2$ already recovers the exact Whittle-index ordering. Moderate-depth threshold policies also outperform the one-step baseline and remain close to the optimal dynamic-programming benchmark, while runtime grows mildly with $t$.

cs.LG↗

arXiv2026

Efficient Algorithm Design of Dynamic Spectrum Access by Whittle Index

Dynamic spectrum access problem is an important problem that allows a wireless sub-network to use channels temporarily unoccupied by the parent network for minimizing the spectrum waste. Previous work has shown that the sequential channel allocation problem for the sub-network can be formulated within the restless multi-armed bandits (RMAB) framework. The objective is to maximize the expected long-term return over an infinite horizon while minimizing interference to the parent network. Different from the previous work that exploits a binary feedback (e.g., ACK/NAK) to compensate for sensing errors, we leverage the finer and more robust channel quality indicator (CQI) feedback to update the information state (belief vector) of the sub-network. However, the implementation of CQI-based observation model yields significantly more complex belief transition behaviors in an infinite state space and worsens the curse of dimensionality of dynamic programming. To overcome this challenge, we dive into the rich structures of the value functions and obtain tight bounds on their derivatives. These results lead to the proof of optimality of threshold policies on a single-channel problem with subsidy and subsequently a closed-form channel index function using an iterative method to approximate the well-known Whittle index policy, which offers a low-complexity solution for ranking the currently available channels whose states are never directly observable. Through extensive numerical studies, we demonstrate the superior performance and robustness of our proposed algorithm.

math.OC↗

arXiv2025

General Formulation and PCL-Analysis for Restless Bandits with Limited Observability

In this paper, we consider a general observation model for restless multi-armed bandit problems. The operation of the player is based on the past observation history that is limited (partial) and error-prone due to resource constraints or environmental or intrinsic noises. By establishing a general probabilistic model for dynamics of the observation process, we formulate the problem as a restless bandit with an infinite high-dimensional belief state space. We apply the achievable region method with partial conservation law (PCL) to the infinite-state problem and analyze its indexability and priority index (Whittle index). Finally, we propose an approximation process to transform the problem into which the AG algorithm of Niño-Mora (2001) for finite-state problems can be applied. Numerical experiments show that our algorithm has excellent performance.

stat.ML↗