arXiv subjects

William B. Haskell

Publications and source records attributed to William B. Haskell.

At least 19 recordsLinked to original sources

arXiv2026

History-Dependent Recursive Preferences in Markov Decision Processes

In finite horizon dynamic programming with history-dependent preferences, the relevant state may be the entire realized history, even when the physical state is Markov. This paper develops a behavioral state-reduction theory for such Markov decision processes. Under behavioral axioms and a certainty-equivalent richness condition, the full-history problem admits a recursive representation composed of time and risk aggregators. We then derive a canonical preference-augmented (PA) state by quotienting histories that have the same current physical Markov state, are indifferent under every common continuation plan, and remain equivalent after every common one-step extension. This canonical PA state is minimal among reachable recursive factorizations of the underlying preferences. Under Markov feasibility and standard dynamic-programming regularity, a PA Bellman selector induces an optimal full-history policy. With additional rectangularity and exhaustiveness conditions, we reparameterize the preference memory into distinct belief and taste coordinates, and obtain a separated representation and Bellman recursion. We give a taxonomy of examples to illustrate the scope of our framework.

math.OC↗

arXiv2025

Robust Data-Driven Quasiconcave Optimization

We investigate a data-driven quasiconcave maximization problem where information about the objective function is limited to a finite sample of data points. We begin by defining an ambiguity set for admissible objective functions based on available partial information about the objective. This ambiguity set consists of those quasiconcave functions that majorize a given data sample, and that satisfy additional functional properties (monotonicity, Lipschitz continuity, and permutation invariance). We then formulate a robust optimization (RO) problem which maximizes the worst-case objective function over this ambiguity set. Based on the quasiconcave structure in this problem, we explicitly construct the upper level sets of the worst-case objective at all levels. We can then solve the resulting RO problem efficiently by doing binary search over the upper level sets and solving a logarithmic number of convex feasibility problems. This numerical approach differs from traditional subgradient descent and support function based methods for this problem class. While these methods can be applied in our setting, the binary search method displays superb finite convergence to the global optimum, whereas the others do not. This is primarily because binary search fully exploits the specific structure of the worst-case quasiconcave objective, which leads to an explicit and general convergence rate in terms of the number of convex optimization problems to be solved. Our numerical experiments on a Cobb-Douglas production efficiency problem and a fair resource allocation problem demonstrate the tractability of our approach.

math.OC↗

arXiv2024

Dynamic Capital Requirements for Markov Decision Processes

We build on the theory of capital requirements (CRs) to create a new framework for modeling dynamic risk preferences. The key question is how to evaluate the risk of a payoff stream sequentially as new information is revealed. In our model, we associate each payoff stream with a disbursement strategy and a premium schedule to form a triple of stochastic processes. We characterize risk preferences in terms of a single set that we call the risk frontier which characterizes acceptable triples. We then propose the generalized capital requirement (GCR) which evaluates the risk of a payoff stream by minimizing the premium schedule over acceptable triples. We apply this model to a risk-aware decision maker (DM) who controls a Markov decision process (MDP) and wants to find a policy to minimize the GCR of its payoff stream. The resulting GCR-MDP recovers many well-known risk-aware MDPs as special cases. To make this approach computationally viable, we obtain the temporal decomposition of the GCR in terms of the risk frontier. Then, we connect the temporal decomposition with the notion of an information state to compactly capture the dependence of DM's risk preferences on the problem history, where augmented dynamic programming can be used to compute an optimal policy. We report numerical experiments for the GCR-minimizing newsvendor.

math.OC↗

arXiv2023

An Inexact Primal-Dual Smoothing Framework for Large-Scale Non-Bilinear Saddle Point Problems

We develop an inexact primal-dual first-order smoothing framework to solve a class of non-bilinear saddle point problems with primal strong convexity. Compared with existing methods, our framework yields a significant improvement over the primal oracle complexity, while it has competitive dual oracle complexity. In addition, we consider the situation where the primal-dual coupling term has a large number of component functions. To efficiently handle this situation, we develop a randomized version of our smoothing framework, which allows the primal and dual sub-problems in each iteration to be inexactly solved by randomized algorithms in expectation. The convergence of this framework is analyzed both in expectation and with high probability. In terms of the primal and dual oracle complexities, this framework significantly improves over its deterministic counterpart. As an important application, we adapt both frameworks for solving convex optimization problems with many functional constraints. To obtain an $\varepsilon$-optimal and $\varepsilon$-feasible solution, both frameworks achieve the best-known oracle complexities.

math.OC↗

arXiv2022

Learning to Price Supply Chain Contracts against a Learning Retailer

The rise of big data analytics has automated the decision-making of companies and increased supply chain agility. In this paper, we study the supply chain contract design problem faced by a data-driven supplier who needs to respond to the inventory decisions of the downstream retailer. Both the supplier and the retailer are uncertain about the market demand and need to learn about it sequentially. The goal for the supplier is to develop data-driven pricing policies with sublinear regret bounds under a wide range of possible retailer inventory policies for a fixed time horizon. To capture the dynamics induced by the retailer's learning policy, we first make a connection to non-stationary online learning by following the notion of variation budget. The variation budget quantifies the impact of the retailer's learning strategy on the supplier's decision-making. We then propose dynamic pricing policies for the supplier for both discrete and continuous demand. We also note that our proposed pricing policy only requires access to the support of the demand distribution, but critically, does not require the supplier to have any prior knowledge about the retailer's learning policy or the demand realizations. We examine several well-known data-driven policies for the retailer, including sample average approximation, distributionally robust optimization, and parametric approaches, and show that our pricing policies lead to sublinear regret bounds in all these cases. At the managerial level, we answer affirmatively that there is a pricing policy with a sublinear regret bound under a wide range of retailer's learning policies, even though she faces a learning retailer and an unknown demand distribution. Our work also provides a novel perspective in data-driven operations management where the principal has to learn to react to the learning policies employed by other agents in the system.

cs.LG↗

arXiv2022

Robustness to Modeling Errors in Risk-Sensitive Markov Decision Problems with Markov Risk Measures

We consider risk-sensitive Markov decision processes (MDPs), where the MDP model is influenced by a parameter which takes values in a compact metric space. We identify sufficient conditions under which small perturbations in the model parameters lead to small changes in the optimal value function and optimal policy. We further establish the robustness of the risk-sensitive optimal policies to modeling errors. Implications of the results for data-driven decision-making, decision-making with preference uncertainty, and systems with changing noise distributions are discussed.

math.OC↗

arXiv2021

Asymptotic Analysis for Data-Driven Inventory Policies

We study periodic review stochastic inventory control in the data-driven setting where the retailer makes ordering decisions based only on historical demand observations without any knowledge of the probability distribution of the demand. Since an (s, S)-policy is optimal when the demand distribution is known, we investigate the statistical properties of the data-driven (s, S)-policy obtained by recursively computing the empirical cost-to-go functions. This policy is inherently challenging to analyze because the recursion induces propagation of the estimation error backwards in time. In this work, we establish the asymptotic properties of this data-driven policy by fully accounting for the error propagation. First, we rigorously show the consistency of the estimated parameters by filling in some gaps (due to unaccounted error propagation) in the existing studies. In this setting, empirical process theory (EPT) cannot be directly applied to show asymptotic normality. To explain, the empirical cost-to-go functions for the estimated parameters are not i.i.d. sums due to the error propagation. Our main methodological innovation comes from an asymptotic representation for multi-sample U-processes in terms of i.i.d. sums. This representation enables us to apply EPT to derive the influence functions of the estimated parameters and to establish joint asymptotic normality. Based on these results, we also propose an entirely data-driven estimator of the optimal expected cost and we derive its asymptotic distribution. We demonstrate some useful applications of our asymptotic results, including sample size determination and interval estimation. The results from our numerical simulations conform to our theoretical analysis.lations conform to our theoretical analysis.

math.ST↗

arXiv2021

A Unifying Framework for Variance Reduction Algorithms for Finding Zeroes of Monotone Operators

It is common to encounter large-scale monotone inclusion problems where the objective has a finite sum structure. We develop a general framework for variance-reduced forward-backward splitting algorithms for this problem. This framework includes a number of existing deterministic and variance-reduced algorithms for function minimization as special cases, and it is also applicable to more general problems such as saddle-point problems and variational inequalities. With a carefully constructed Lyapunov function, we show that the algorithms covered by our framework enjoy a linear convergence rate in expectation under mild assumptions. We further consider Catalyst acceleration and asynchronous implementation to reduce the algorithmic complexity and computation time. We apply our proposed framework to a policy evaluation problem and a strongly monotone two-player game, both of which fall outside of function minimization.

stat.ML↗

arXiv2021

Convergence of Recursive Stochastic Algorithms using Wasserstein Divergence

This paper develops a unified framework, based on iterated random operator theory, to analyze the convergence of constant stepsize recursive stochastic algorithms (RSAs). RSAs use randomization to efficiently compute expectations, and so their iterates form a stochastic process. The key idea of our analysis is to lift the RSA into an appropriate higher-dimensional space and then express it as an equivalent Markov chain. Instead of determining the convergence of this Markov chain (which may not converge under constant stepsize), we study the convergence of the distribution of this Markov chain. To study this, we define a new notion of Wasserstein divergence. We show that if the distribution of the iterates in the Markov chain satisfy a contraction property with respect to the Wasserstein divergence, then the Markov chain admits an invariant distribution. We show that convergence of a large family of constant stepsize RSAs can be understood using this framework, and we provide several detailed examples.

cs.LG↗

arXiv2020

An Accelerated Fitted Value Iteration Algorithm for MDPs with Finite and Vector-Valued Action Space

This paper studies an accelerated fitted value iteration (FVI) algorithm to solve high-dimensional Markov decision processes (MDPs). FVI is an approximate dynamic programming algorithm that has desirable theoretical properties. However, it can be intractable when the action space is finite but vector-valued. To solve such MDPs via FVI, we first approximate the value functions by a two-layer neural network (NN) with rectified linear units (ReLU) being activation functions. We then verify that such approximators are strong enough for the MDP. To speed up the FVI, we recast the action selection problem as a two-stage stochastic programming problem, where the resulting recourse function comes from the two-layer NN. Then, the action selection problem is solved with a specialized multi-cut decomposition algorithm. More specifically, we design valid cuts by exploiting the structure of the approximated value functions to update the actions. We prove that the decomposition can find the global optimal solution in a finite number of iterations and the overall accelerated FVI is consistent. Finally, we verify the performance of the FVI algorithm via a multi-facility capacity investment problem (MCIP). A comprehensive numerical study is implemented, where the results show that the FVI is significantly accelerated without sacrificing too much in precision.

math.OC↗

arXiv2020

Efficiently Computing the Quasiconcave Envelope with Incomplete Information

In this paper, we study the approximation of an unknown quasiconcave function based on limited partial information. Available information includes lower bounds on the values of the target function at a specified set of points, as well as some functional properties including monotonicity, Lipschitz continuity, ranking, and permutation invariance. We consider the class of admissible quasiconcave functions that dominate these lower bounds and satisfy these functional properties. We then compute the smallest quasiconcave function among the class of admissible quasiconcave functions. Specifically, we show how to efficiently compute the quasiconcave envelope (QCoE) of a data sample of points, subject to the additional functional properties. The solution procedure takes two steps. First, a value problem is solved to determine the values of the QCoE on the given data sample. Second, an interpolation problem is solved to compute the values of the QCoE on other points. Both the value problem and the interpolation problem introduce some theoretical and computational challenges, as they are non-convex and large-scale. The MILP reformulations of both problems require an exponential number of linear programs (LPs) to be solved in the worst-case. As our main contribution, we solve the value problem with only a polynomial number of LPs, and then solve the interpolation problem for any candidate point with only a logarithmic number of LPs. Some preliminary numerical tests show that the proposed approach is efficient and proper.

q-fin.RM↗

arXiv2020

A dynamic analytic method for risk-aware controlled martingale problems

We present a new, tractable method for solving and analyzing risk-aware control problems over finite and infinite, discounted time-horizons where the dynamics of the controlled process are described as a martingale problem. Supposing general Polish state and action spaces, and using generalized, relaxed controls, we state a risk-aware dynamic optimal control problem of minimizing risk of costs described by a generic risk function. We then construct an alternative formulation that takes the form of a nonlinear programming problem, constrained by the dynamic, {i.e.} time-dependent, and linear Kolmogorov forward equation describing the distribution of the state and accumulated costs. We show that the formulations are equivalent, and that the optimal control process can be taken to be Markov in the controlled process state, running costs, and time. We further prove that under additional conditions, the optimal value is attained. An example numeric problem is presented and solved.

math.OC↗

arXiv2020

A Randomized Nonlinear Rescaling Method in Large-Scale Constrained Convex Optimization

We propose a new randomized algorithm for solving convex optimization problems that have a large number of constraints (with high probability). Existing methods like interior-point or Newton-type algorithms are hard to apply to such problems because they have expensive computation and storage requirements for Hessians and matrix inversions. Our algorithm is based on nonlinear rescaling (NLR), which is a primal-dual-type algorithm by Griva and Polyak {[{Math. Program., 106(2):237-259, 2006}]}. NLR introduces an equivalent problem through a transformation of the constraint functions, minimizes the corresponding augmented Lagrangian for given dual variables, and then uses this minimizer to update the dual variables for the next iteration. The primal update at each iteration is the solution of an unconstrained finite sum minimization problem where the terms are weighted by the current dual variables. We use randomized first-order algorithms to do these primal updates, for which they are especially well suited. In particular, we use the scaled dual variables as the sampling distribution for each primal update, and we show that this distribution is the optimal one among all probability distributions. We conclude by demonstrating the favorable numerical performance of our algorithm.

math.OC↗

arXiv2019

Stochastic Approximation for Risk-aware Markov Decision Processes

We develop a stochastic approximation-type algorithm to solve finite state/action, infinite-horizon, risk-aware Markov decision processes. Our algorithm has two loops. The inner loop computes the risk by solving a stochastic saddle-point problem. The outer loop performs $Q$-learning to compute an optimal risk-aware policy. Several widely investigated risk measures (e.g. conditional value-at-risk, optimized certainty equivalent, and absolute semi-deviation) are covered by our algorithm. Almost sure convergence and the convergence rate of the algorithm are established. For an error tolerance $ε>0$ for the optimal $Q$-value estimation gap and learning rate $k\in(1/2,\,1]$, the overall convergence rate of our algorithm is $Ω((\ln(1/δε)/ε^{2})^{1/k}+(\ln(1/ε))^{1/(1-k)})$ with probability at least $1-δ$.

math.OC↗

arXiv2019

Model and Reinforcement Learning for Markov Games with Risk Preferences

We motivate and propose a new model for non-cooperative Markov game which considers the interactions of risk-aware players. This model characterizes the time-consistent dynamic "risk" from both stochastic state transitions (inherent to the game) and randomized mixed strategies (due to all other players). An appropriate risk-aware equilibrium concept is proposed and the existence of such equilibria is demonstrated in stationary strategies by an application of Kakutani's fixed point theorem. We further propose a simulation-based Q-learning type algorithm for risk-aware equilibrium computation. This algorithm works with a special form of minimax risk measures which can naturally be written as saddle-point stochastic optimization problems, and covers many widely investigated risk measures. Finally, the almost sure convergence of this simulation-based algorithm to an equilibrium is demonstrated under some mild conditions. Our numerical experiments on a two player queuing game validate the properties of our model and algorithm, and demonstrate their worth and applicability in real life competitive decision-making.

cs.GT↗

arXiv2019

Risk aware minimum principle for optimal control of stochastic differential equations

We present a probabilistic formulation of risk aware optimal control problems for stochastic differential equations. Risk awareness is in our framework captured by objective functions in which the risk neutral expectation is replaced by a risk function, a nonlinear functional of random variables that account for the controller's risk preferences. We state and prove a risk aware minimum principle that is a parsimonious generalization of the well-known risk neutral, stochastic Pontryagin's minimum principle. As our main results we give necessary and also sufficient conditions for optimality of control processes taking values on probability measures defined on a given action space. We show that remarkably, going from the risk neutral to the risk aware case, the minimum principle is simply modified by the introduction of one additional real-valued stochastic process that acts as a risk adjustment factor for given cost rate and terminal cost functions. This adjustment process is explicitly given as the expectation, conditional on the filtration at the given time, of an appropriately defined functional derivative of the risk function evaluated at the random total cost. For our results we rely on the Fréchet differentiability of the risk function, and for completeness, we prove under mild assumptions the existence of Fréchet derivatives of some common risk functions. We give a simple application of the results for a portfolio allocation problem and show that the risk awareness of the objective function gives rise to a risk premium term that is characterized by the risk adjustment process described above. This suggests uses of our results in e.g. pricing of risk modeled by generic risk functions in financial applications.

math.OC↗

arXiv2019

A Flexible Multi-Facility Capacity Expansion Problem with Risk Aversion

This paper studies flexible multi-facility capacity expansion with risk aversion. In this setting, the decision maker can periodically expand the capacity of facilities given observations of uncertain demand. We model this situation as a multi-stage stochastic programming problem. We express risk aversion in this problem through conditional value-at-risk (CVaR), and we formulate a mean-CVaR objective. To solve the multi-stage problem, we optimize over decision rules. In particular, we approximate the full policy space of the problem with a tractable family of if-then policies. Subsequently, a decomposition algorithm is proposed to optimize the decision rule. This algorithm decomposes the model over scenarios and it updates solutions via the subgradients of the recourse function. We demonstrate that this algorithm can quickly converge to high-performance policies. To illustrate the practical effectiveness of this method, a case study on the waste-to-energy system in Singapore is presented. These simulation results show that by adjusting the weight factor of the objective function, decision makers are able to trade off between a risk-averse policy that has a higher expected cost but a lower value-at-risk, and a risk-neutral policy that has a lower expected cost but a higher value-at-risk risk.

math.OC↗

arXiv2019

An Empirical Dynamic Programming Algorithm for Continuous MDPs

We propose universal randomized function approximation-based empirical value iteration (EVI) algorithms for Markov decision processes. The `empirical' nature comes from each iteration being done empirically from samples available from simulations of the next state. This makes the Bellman operator a random operator. A parametric and a non-parametric method for function approximation using a parametric function space and the Reproducing Kernel Hilbert Space (RKHS) respectively are then combined with EVI. Both function spaces have the universal function approximation property. Basis functions are picked randomly. Convergence analysis is done using a random operator framework with techniques from the theory of stochastic dominance. Finite time sample complexity bounds are derived for both universal approximate dynamic programming algorithms. Numerical experiments support the versatility and effectiveness of this approach.

math.OC↗