Hadi El-Amine | Searcharxiv

arXiv2026

Consistent Variance Estimation for Q-Function Estimators in Finite-Horizon MDP Tree Search

We study the variance of Q-function estimators in finite-horizon, finite-state Markov decision process (MDP) tree search. We show that the variance decomposes into three components attributed to the immediate reward collected, probabilistic state transitions, and uncertainty in future state value function estimates. Using this decomposition, we show that the sample variance estimator based on the assumption of i.i.d. paths is biased, underestimating the true variance, and the bias does not vanish in the limit. We then propose a recursive variance estimator that is consistent. To enable efficient storage and computation, we derive an equivalent implementation of the recursive estimator using only node-local statistics that can be iteratively updated. This consistent variance estimator is integrated into two Monte Carlo Tree Search (MCTS) sampling procedures for finite-horizon MDPs. In numerical examples from inventory control and kidney paired donation matching, the new estimator improves the performance of the MCTS algorithm relative to a baseline that uses the i.i.d.-based sample variance estimator.

eess.SY↗

arXiv2021

Computational Efficiency in Multivariate Adversarial Risk Analysis Models

In this paper we address the computational feasibility of the class of decision theoretic models referred to as adversarial risk analyses (ARA). These are models where a decision must be made with consideration for how an intelligent adversary may behave and where the decision-making process of the adversary is unknown, and is elicited by analyzing the adversary's decision problem using priors on his utility function and beliefs. The motivation of this research was to develop a computational algorithm that can be applied across a broad range of ARA models; to the best of our knowledge, no such algorithm currently exists. Using a two-person sequential model, we incrementally increase the size of the model and develop a simulation-based approximation of the true optimum where an exact solution is computationally impractical. In particular, we begin with a relatively large decision space by considering a theoretically continuous space that must be discretized. Then, we incrementally increase the number of strategic objectives which causes the decision space to grow exponentially. The problem is exacerbated by the presence of an intelligent adversary who also must solve an exponentially large decision problem according to some unknown decision-making process. Nevertheless, using a stylized example that can be solved analytically we show that our algorithm not only solves large ARA models quickly but also accurately selects to the true optimal solution. Furthermore, the algorithm is sufficiently general that it can be applied to any ARA model with a large, yet finite, decision space.

econ.GN↗