arXiv subjects

Daniel Adelman

Publications and source records attributed to Daniel Adelman.

4 recordsLinked to original sources

arXiv2026

Optimizing Treatment Allocation to Maximize the Health of a Population

Recent shifts in global health priorities have positioned Population Health Management (PHM) as a central area of focus. However, optimizing PHM strategies presents several challenges: managing high-dimensional patient covariates, tracking their evolution and long-term response to interventions, and accounting for the inflow and outflow of individuals within the population. In this paper, we propose a novel approach based on Measurized MDPs that integrates these components. We consider a setting in which a treatment with population-level benefits is available but scarce, and model an MDP that optimizes the long-term distribution of the healthcare population under expected capacity constraints. This formulation allows us to bypass both the dimensionality and practical challenges of handling and tracking individual patient covariates across the population. To ensure ethical compliance, we introduce a non-maleficence constraint that limits the allowable mortality rate. To solve the resulting infinite-dimensional problem, we use ADP and reduce the task to identifying a finite set of high-performing treated and untreated patients. Despite the complexity of the underlying structure, our approach yields a simple, clinically implementable index policy: a patient is selected for treatment if their adjusted impactability exceeds a specified threshold. The adjusted impactability captures the long-term consequences of receiving or not receiving treatment. While straightforward to apply, the policy remains flexible and can incorporate general machine learning models. Using CMS data, we show that our policy yields a statistically significant improvement over a myopic benchmark. This advantage increases with the time horizon, consistent with the forward-looking nature of our policy. At the longest horizon tested, this corresponds to over 1,500 additional home days annually per 1,000 patients.

math.OC↗

arXiv2025

Dynamic Basis Function Generation for Network Revenue Management

This paper introduces an algorithm that dynamically generates basis functions to approximate the value function in Network Revenue Management. Unlike existing algorithms sampling the parameters of new basis functions, this Nonlinear Incremental Algorithm (NLIAlg) iteratively refines the value function approximation by optimizing these parameters. For larger instances, the Two-Phase Incremental Algorithm (2PIAlg) modifies NLIAlg to leverage the efficiency of LP solvers. It reduces the size of a large-dimensional nonlinear problem and transforms it into an LP by fixing the basis function parameters, which are then optimized in a second phase using the flow imbalance ideas from Adelman and Klabjan (2012). This marks the first application of these techniques in a stochastic setting. The algorithms can operate in two modes: (1) Standalone mode, to construct a value function approximation from scratch, and (2) Add-on mode, to refine an existing approximation. Our numerical experiments indicate that while NLIAlg and 2PIAlg in standalone mode are only feasible for small-scale problems, the heuristic version of 2PIAlg (H-2PIAlg) in add-on mode, using the Affine Approximation and exponential ridge basis functions, can handle extremely large instances that may cause benchmark network revenue management methods to run out of memory. In these scenarios, H-2PIAlg delivers substantially better policies and upper bounds than the Affine Approximation. Furthermore, H-2PIAlg achieves higher average revenues in policy simulations compared to network revenue management benchmarks in instances with limited capacity.

math.OC↗

arXiv2024

Thompson Sampling for Infinite-Horizon Discounted Decision Processes

This paper develops a viable notion of learning for sampling-based algorithms that applies in broader settings than previously considered. More specifically, we model a discounted infinite-horizon MDPs with Borel state and action spaces, whose rewards and transitions depend on an unknown parameter. To analyze adaptive learning algorithms based on sampling we introduce a general canonical probability space in this setting. Since standard definitions of regret are inadequate for policy evaluation in this setting, we propose new metrics that arise from decomposing the standard expected regret in discounted infinite-horizon MDPs into three terms: (i) the expected finite-time regret, (ii) the expected state regret, and (iii) the expected residual regret. Component (i) translates into the traditional concept of expected regret over a finite horizon. Term (ii) reflects how much future performance is compromised at a given time because earlier decisions have led the system to a less favorable state than under an optimal policy. Finally, metric (iii) measures regret with respect to the optimal reward from the current period onward, disregarding the irreversible consequences of past decisions. We further disaggregate this term by introducing the probabilistic residual regret, a finer, sample-path version of (iii) that captures the remaining loss in future performance from the current period onward, conditional on the observed history. Its expectation coincides with (iii). We then focus on Thompson sampling (TS); under assumptions that extend those used in prior work on finite state and action spaces to the Borel setting, we show that component (iii) for TS converges to zero exponentially fast. We further show that, under mild conditions ensuring the existence of the relevant limits, its probabilistic counterpart converges to zero almost surely and TS achieves complete learning.

stat.ML↗

arXiv2024

Measurized Markov Decision Processes

In this paper, we explore lifting Markov Decision Processes (MDPs) to the space of probability measures and consider the so-called measurized MDPs: deterministic processes where states are probability measures on the original state space, and actions are stochastic kernels on the original action space. We show that measurized MDPs are a generalization of stochastic MDPs, thus the measurized framework can be deployed without loss of fidelity. Bertsekas and Shreve studied similar deterministic MDPs under the discounted infinite-horizon criterion in the context of universally measurable policies. Here, we also consider the long-run average reward case, but we cast lifted MDPs within the semicontinuous-semicompact framework of Hern\'andez-Lerma and Lasserre. This makes the lifted framework more accessible as it entails (i) optimal Borel-measurable value functions and policies, (ii) reasonably mild assumptions that are easier to verify than those in the universally-measurable framework, and (iii) simpler proofs. In addition, we showcase the untapped potential of lifted MDPs by demonstrating how the measurized framework enables the incorporation of constraints and value function approximations that are not available from the standard MDP setting. Furthermore, we introduce a novel algebraic lifting procedure for any MDP, showing that non-deterministic measure-valued MDPs can emerge from lifting MDPs impacted by external random shocks.

math.OC↗