arXiv subjects

Rong J. B. Zhu

Publications and source records attributed to Rong J. B. Zhu.

7 recordsLinked to original sources

arXiv2026

From Weighting to Modeling: A Nonparametric Estimator for Off-Policy Evaluation

We study off-policy evaluation in the setting of contextual bandits, where we aim to evaluate a new policy using historical data that consists of contexts, actions and received rewards. This historical data typically does not faithfully represent action distribution of the new policy accurately. A common approach, inverse probability weighting (IPW), adjusts for these discrepancies in action distributions. However, this method often suffers from high variance due to the probability being in the denominator. The doubly robust (DR) estimator reduces variance through modeling reward but does not directly address variance from IPW. In this work, we address the limitation of IPW by proposing a Nonparametric Weighting (NW) approach that constructs weights using a nonparametric model. Our NW approach achieves low bias like IPW but typically exhibits significantly lower variance. To further reduce variance, we incorporate reward predictions -- similar to the DR technique -- resulting in the Model-assisted Nonparametric Weighting (MNW) approach. The MNW approach yields accurate value estimates by explicitly modeling and mitigating bias from reward modeling, without aiming to guarantee the standard doubly robust property. Extensive empirical comparisons show that our approaches consistently outperform existing techniques, achieving lower variance in value estimation while maintaining low bias.

cs.LG↗

arXiv2026

Synthetic Regressing Control

Estimating weights in the synthetic control method, typically resulting in sparse weights where only a few control units have non-zero weights, involves an optimization procedure that selects and combines control units to closely match the treated unit. However, it is not uncommon for the linear combination of pre-treatment period outcomes for the control units, using nonnegative weights with the constraint that their sum equals one, to inadequately approximate the pre-treatment outcomes for the treated unit. To address the issue, this paper proposes a simple and effective method called Synthetic Regressing Control (SRC). The SRC method begins by performing the univariate linear regression to appropriately align the pre-treatment periods of the control units with the treated unit. Subsequently, a SRC estimator is obtained by synthesizing the regressed controls. To determine the weights in the synthesis procedure, we propose an approach that utilizes a criterion of an unbiased risk estimator. Theoretically, we show that the synthesis way is asymptotically optimal in the sense of achieving the minimum loss of the infeasible best possible synthetic estimator. Extensive numerical experiments highlight the advantages of the SRC method.

econ.EM↗

arXiv2025

Empirical Bayesian Multi-Bandit Learning

Multi-task learning in contextual bandits has attracted significant research interest due to its potential to enhance decision-making across multiple related tasks by leveraging shared structures and task-specific heterogeneity. In this article, we propose a novel hierarchical Bayesian framework for learning in various bandit instances. This framework captures both the heterogeneity and the correlations among different bandit instances through a hierarchical Bayesian model, enabling effective information sharing while accommodating instance-specific variations. Unlike previous methods that overlook the learning of the covariance structure across bandits, we introduce an empirical Bayesian approach to estimate the covariance matrix of the prior distribution. This enhances both the practicality and flexibility of learning across multi-bandits. Building on this approach, we develop two efficient algorithms: ebmTS (Empirical Bayesian Multi-Bandit Thompson Sampling) and ebmUCB (Empirical Bayesian Multi-Bandit Upper Confidence Bound), both of which incorporate the estimated prior into the decision-making process. We provide the frequentist regret upper bounds for the proposed algorithms, thereby filling a research gap in the field of multi-bandit problems. Extensive experiments on both synthetic and real-world datasets demonstrate the superior performance of our algorithms, particularly in complex environments. Our methods achieve lower cumulative regret compared to existing techniques, highlighting their effectiveness in balancing exploration and exploitation across multi-bandits.

cs.LG↗

arXiv2024

Semiparametric Inference for Regression-Discontinuity Designs

Treatment effects in regression discontinuity designs (RDDs) are often estimated using local regression methods. \cite{Hahn:01} demonstrated that the identification of the average treatment effect at the cutoff in RDDs relies on the unconfoundedness assumption and that, without this assumption, only the local average treatment effect at the cutoff can be identified. In this paper, we propose a semiparametric framework tailored for identifying the average treatment effect in RDDs, eliminating the need for the unconfoundedness assumption. Our approach globally conceptualizes the identification as a partially linear modeling problem, with the coefficient of a specified polynomial function of propensity score in the linear component capturing the average treatment effect. This identification result underpins our semiparametric inference for RDDs, employing the $P$-spline method to approximate the nonparametric function and establishing a procedure for conducting inference within this framework. Through theoretical analysis, we demonstrate that our global approach achieves a faster convergence rate compared to the local method. Monte Carlo simulations further confirm that the proposed method consistently outperforms alternatives across various scenarios. Furthermore, applications to real-world datasets illustrate that our global approach can provide more reliable inference for practical problems.

econ.EM↗

arXiv2024

UCB Exploration for Fixed-Budget Bayesian Best Arm Identification

We study best-arm identification (BAI) in the fixed-budget setting. Adaptive allocations based on upper confidence bounds (UCBs), such as UCBE, are known to work well in BAI. However, it is well-known that its optimal regret is theoretically dependent on instances, which we show to be an artifact in many fixed-budget BAI problems. In this paper we propose an UCB exploration algorithm that is both theoretically and empirically efficient for the fixed budget BAI problem under a Bayesian setting. The key idea is to learn prior information, which can enhance the performance of UCB-based BAI algorithm as it has done in the cumulative regret minimization problem. We establish bounds on the failure probability and the simple regret for the Bayesian BAI problem, providing upper bounds of order $\tilde{O}(\sqrt{K/n})$, up to logarithmic factors, where $n$ represents the budget and $K$ denotes the number of arms. Furthermore, we demonstrate through empirical results that our approach consistently outperforms state-of-the-art baselines.

cs.LG↗

arXiv2024

High dimensional Bayesian Optimization via Condensing-Expansion Projection

In high-dimensional settings, Bayesian optimization (BO) can be expensive and infeasible. The random embedding Bayesian optimization algorithm is commonly used to address high-dimensional BO challenges. However, this method relies on the effective subspace assumption on the optimization problem's objective function, which limits its applicability. In this paper, we introduce Condensing-Expansion Projection Bayesian optimization (CEPBO), a novel random projection-based approach for high-dimensional BO that does not reply on the effective subspace assumption. The approach is both simple to implement and highly practical. We present two algorithms based on different random projection matrices: the Gaussian projection matrix and the hashing projection matrix. Experimental results demonstrate that both algorithms outperform existing random embedding-based algorithms in most cases, achieving superior performance on high-dimensional BO problems. The code is available in \url{https://anonymous.4open.science/r/CEPBO-14429}.

cs.LG↗

arXiv2022

Gradient Descent Temporal Difference-difference Learning

Off-policy algorithms, in which a behavior policy differs from the target policy and is used to gain experience for learning, have proven to be of great practical value in reinforcement learning. However, even for simple convex problems such as linear value function approximation, these algorithms are not guaranteed to be stable. To address this, alternative algorithms that are provably convergent in such cases have been introduced, the most well known being gradient descent temporal difference (GTD) learning. This algorithm and others like it, however, tend to converge much more slowly than conventional temporal difference learning. In this paper we propose gradient descent temporal difference-difference (Gradient-DD) learning in order to improve GTD2, a GTD algorithm, by introducing second-order differences in successive parameter updates. We investigate this algorithm in the framework of linear value function approximation, theoretically proving its convergence by applying the theory of stochastic approximation. %analytically showing its improvement over GTD2. Studying the model empirically on the random walk task, the Boyan-chain task, and the Baird's off-policy counterexample, we find substantial improvement over GTD2 and, in several cases, better performance even than conventional TD learning.

cs.LG↗