arXiv subjects

Jefferson Huang

Publications and source records attributed to Jefferson Huang.

8 recordsLinked to original sources

arXiv2026

Markov Decision Process Approximation Methods for Water Distribution Network Inspection and Maintenance: A Case Study of the U.S. Virgin Islands

We develop a repair-oriented inspection and maintenance decision framework for water distribution networks. This work is motivated by utilities operating in data-sparse environments, such as in remote locations like the U.S. Virgin Islands, where data collection about network state and underground pipeline outages is limited to above-ground and easy to access information (e.g., water tank levels and pump operations). We formulate the problem as a discounted Markov decision process and integrate it with high-fidelity hydraulic simulation. The model captures latent system dynamics without requiring pipe-level sensing. The results reveal state-dependent optimal policies and heterogeneous failure characteristics across pipes, including rare but high-impact behaviors. We further show that certain observable system states uniquely correspond to specific pipe failures, enabling a form of virtual sensing. These findings demonstrate that system-level dynamics can support inspection planning and maintenance decisions under uncertainty in resource-constrained settings.

cs.CE↗

arXiv2025

Convergence of Fast Policy Iteration in Markov Games and Robust MDPs

Markov games and robust MDPs are closely related models that involve computing a pair of saddle point policies. As part of the long-standing effort to develop efficient algorithms for these models, the Filar-Tolwinski (FT) algorithm has shown considerable promise. As our first contribution, we demonstrate that FT may fail to converge to a saddle point and may loop indefinitely, even in small games. This observation contradicts the proof of FT's convergence to a saddle point in the original paper. As our second contribution, we propose Residual Conditioned Policy Iteration (RCPI). RCPI builds on FT, but is guaranteed to converge to a saddle point. Our numerical results show that RCPI outperforms other convergent algorithms by several orders of magnitude.

cs.GT↗

arXiv2025

A Bi-Objective Markov Decision Process Design approach to redundancy allocation with dynamic maintenance for a parallel system

The reliability of a system can be improved by the addition of redundant elements, giving rise to the well-known redundancy allocation problem (RAP). We propose a novel extension to the RAP called the Bi-Objective Integrated Design and Dynamic Maintenance Problem (BO-IDDMP) which allows for future dynamic maintenance decisions to be incorporated. This leads to a problem with first-stage redundancy design decisions and second-stage sequential maintenance decisions under uncertainty. To the best of our knowledge, this is the first use of a continuous-time Markov Decision Process Design framework to formulate a problem with non-trivial dynamics, as well as its first use alongside bi-objective optimization. A general heuristic optimization methodology for bi-objective MDP Design problems is developed, and then applied to the BO-IDDMP. The efficiency and accuracy of our methodology are demonstrated against an exact mixed-integer linear programming solver. The heuristic is shown to be orders of magnitude faster in the majority of cases, and in only 2 out of 84 cases produces a solution that is dominated by the exact method. The inclusion of dynamic maintenance policies is shown to yield stronger and better-populated Pareto fronts, allowing more flexibility for the decision-maker. The impacts of varying parameters unique to our problem are also investigated.

math.OC↗

arXiv2025

Computing optimal policies for managing inventories with noisy observations

This paper implements the Deep Deterministic Policy Gradient (DDPG) algorithm for computing optimal policies for partially observable single-product periodic review inventory control problems with setup costs and backorders. The decision maker does not know the exact inventory level, but can obtain noise-corrupted observations of them. The goal is to maximize the expected total discounted costs incurred over a finite planning horizon. We also investigate the Gaussian version of this problem with normally distributed initial inventories, demands, and observation noise. We show that expected posterior observations of inventory levels, also called mean beliefs, provide sufficient statistics for the Gaussian problem. Moreover, they can be represented in the form of a Markov Decision Processes for an inventory control system with time-dependent holding costs and demands. Thus, for a Gaussian problem, the there exist (s_t,S_t)-optimal policies based on mean beliefs, and this fact explains the structure of the approximately optimal policies computed by DDPG. For the Gaussian case, we also numerically compare the performance of policies derived from DDPG to optimal policies for discretized versions of the original continuous problem.

math.OC↗

arXiv2024

Tropical Decision Boundaries for Neural Networks Are Robust Against Adversarial Attacks

We introduce a simple, easy to implement, and computationally efficient tropical convolutional neural network architecture that is robust against adversarial attacks. We exploit the tropical nature of piece-wise linear neural networks by embedding the data in the tropical projective torus in a single hidden layer which can be added to any model. We study the geometry of its decision boundary theoretically and show its robustness against adversarial attacks on image datasets using computational experiments.

cs.LG↗

arXiv2017

Reduction of total-cost and average-cost MDPs with weakly continuous transition probabilities to discounted MDPs

This note describes sufficient conditions under which total-cost and average-cost Markov decision processes (MDPs) with general state and action spaces, and with weakly continuous transition probabilities, can be reduced to discounted MDPs. For undiscounted problems, these reductions imply the validity of optimality equations and the existence of stationary optimal policies. The reductions also provide methods for computing optimal policies. The results are applied to a capacitated inventory control problem with fixed costs and lost sales.

math.OC↗

arXiv2017

On the Reduction of Total-Cost and Average-Cost MDPs to Discounted MDPs

This paper provides conditions under which total-cost and average-cost Markov decision processes (MDPs) can be reduced to discounted ones. Results are given for transient total-cost MDPs with tran- sition rates whose values may be greater than one, as well as for average-cost MDPs with transition probabilities satisfying the condition that there is a state such that the expected time to reach it is uniformly bounded for all initial states and stationary policies. In particular, these reductions imply sufficient conditions for the validity of optimality equations and the existence of stationary optimal poli- cies for MDPs with undiscounted total cost and average-cost criteria. When the state and action sets are finite, these reductions lead to linear programming formulations and complexity estimates for MDPs under the aforementioned criteria.

math.OC↗

arXiv2013

The Value Iteration Algorithm is Not Strongly Polynomial for Discounted Dynamic Programming

This note provides a simple example demonstrating that, if exact computations are allowed, the number of iterations required for the value iteration algorithm to find an optimal policy for discounted dynamic programming problems may grow arbitrarily quickly with the size of the problem. In particular, the number of iterations can be exponential in the number of actions. Thus, unlike policy iterations, the value iteration algorithm is not strongly polynomial for discounted dynamic programming.

cs.AI↗