arXiv subjects

Xun Shen

Publications and source records attributed to Xun Shen.

16 recordsLinked to original sources

arXiv2026

MedGym:A Unified Continuous-Time Benchmark for Dynamic Medical Treatment Reinforcement Learning

Medical treatment recommendation poses several challenges to reinforcement learning (RL): patient physiology evolves in continuous time, measurements and interventions are performed at irregular intervals, and treatment effects vary substantially across individuals. Existing RL formulations and simulated environments, however, are based on discrete-time MDP or POMDP abstractions with fixed or pre-specified decision intervals. Thus, it remains difficult to evaluate whether RL methods can handle time-interval-dependent disease progression, personalized treatment response, and safety between consecutive measurement points. To address this gap, we introduce MedGym, a benchmark environment for dynamic treatment recommendation. MedGym models longitudinal patient evolution in a continuous-time framework and constructs a configurable medical RL benchmark from clinical data by using Physics-Informed Neural Networks. The resulting benchmark supports both offline and online RL, and enables direct comparison between discrete-time and continuous-time methods under irregular treatment timing and patient-specific dynamics. Besides, MedGym supports evaluation from clinically important perspectives, including personalization, trajectory-level safety, and the performance gap between model-based offline learning and online deployment. By providing a standardized and configurable benchmark for continuous-time dynamic treatment, MedGym aims to facilitate more realistic and informative evaluation of medical RL methods.

cs.LG↗

arXiv2026

Interaction-Limited Safe Continuous-Time RL for Dynamical Medical Treatment

Dynamic medical treatment requires deciding treatment intensity and intervention timing, while patient states evolve continuously and adverse events may occur between clinical interactions. Most existing treatment learning methods assume fixed schedules or enforce safety only at discrete decision points. We propose Interaction-Limited Safe Continuous-Time Reinforcement Learning, a framework that jointly optimizes treatment administration and clinical interaction timing under trajectory-level safety constraints. Our key idea is to reformulate the continuous time treatment problem as an option-based semi-Markov decision process, where each option specifies a continuous-time treatment policy and its duration. We develop a safety-tightening mechanism showing that suitably constructed constraints at interaction times guarantee safety over the full continuous-time trajectory with high probability. We further establish finite-sample guarantees for policy learning from logged treatment trajectories and introduce a practical data-driven conservative surrogate. Experiments show that the proposed adaptive interaction-timing mechanism improves both safety and treatment effectiveness over equidistant interaction schemes across different safe policy optimization methods.

cs.LG↗

arXiv2026

A Canonical Structure for Constructing Projected First-Order Algorithms With Delayed Feedback

This work introduces a canonical structure for a broad class of unconstrained first-order algorithms that admit a Lur'e representation, including systems with relative degree greater than one, e.g., systems with delayed gradient feedback. The proposed canonical structure is obtained through a simple linear transformation. It enables a direct extension from unconstrained optimization algorithms to set-constrained ones through projection in a Lyapunov-induced norm. The resulting projected algorithms attain the optimal solution while preserving the convergence rates of their unconstrained counterparts.

math.OC↗

arXiv2025

Offline Guarded Safe Reinforcement Learning for Medical Treatment Optimization Strategies

When applying offline reinforcement learning (RL) in healthcare scenarios, the out-of-distribution (OOD) issues pose significant risks, as inappropriate generalization beyond clinical expertise can result in potentially harmful recommendations. While existing methods like conservative Q-learning (CQL) attempt to address the OOD issue, their effectiveness is limited by only constraining action selection by suppressing uncertain actions. This action-only regularization imitates clinician actions that prioritize short-term rewards, but it fails to regulate downstream state trajectories, thereby limiting the discovery of improved long-term treatment strategies. To safely improve policy beyond clinician recommendations while ensuring that state-action trajectories remain in-distribution, we propose \textit{Offline Guarded Safe Reinforcement Learning} ($\mathsf{OGSRL}$), a theoretically grounded model-based offline RL framework. $\mathsf{OGSRL}$ introduces a novel dual constraint mechanism for improving policy with reliability and safety. First, the OOD guardian is established to specify clinically validated regions for safe policy exploration. By constraining optimization within these regions, it enables the reliable exploration of treatment strategies that outperform clinician behavior by leveraging the full patient state history, without drifting into unsupported state-action trajectories. Second, we introduce a safety cost constraint that encodes medical knowledge about physiological safety boundaries, providing domain-specific safeguards even in areas where training data might contain potentially unsafe interventions. Notably, we provide theoretical guarantees on safety and near-optimality: policies that satisfy these constraints remain in safe and reliable regions and achieve performance close to the best possible policy supported by the data.

cs.LG↗

arXiv2024

Flipping-based Policy for Chance-Constrained Markov Decision Processes

Safe reinforcement learning (RL) is a promising approach for many real-world decision-making problems where ensuring safety is a critical necessity. In safe RL research, while expected cumulative safety constraints (ECSCs) are typically the first choices, chance constraints are often more pragmatic for incorporating safety under uncertainties. This paper proposes a \textit{flipping-based policy} for Chance-Constrained Markov Decision Processes (CCMDPs). The flipping-based policy selects the next action by tossing a potentially distorted coin between two action candidates. The probability of the flip and the two action candidates vary depending on the state. We establish a Bellman equation for CCMDPs and further prove the existence of a flipping-based policy within the optimal solution sets. Since solving the problem with joint chance constraints is challenging in practice, we then prove that joint chance constraints can be approximated into Expected Cumulative Safety Constraints (ECSCs) and that there exists a flipping-based policy in the optimal solution sets for constrained MDPs with ECSCs. As a specific instance of practical implementations, we present a framework for adapting constrained policy optimization to train a flipping-based policy. This framework can be applied to other safe RL algorithms. We demonstrate that the flipping-based policy can improve the performance of the existing safe RL algorithms under the same limits of safety constraints on Safety Gym benchmarks.

cs.LG↗

arXiv2024

A Survey of Constraint Formulations in Safe Reinforcement Learning

Safety is critical when applying reinforcement learning (RL) to real-world problems. As a result, safe RL has emerged as a fundamental and powerful paradigm for optimizing an agent's policy while incorporating notions of safety. A prevalent safe RL approach is based on a constrained criterion, which seeks to maximize the expected cumulative reward subject to specific safety constraints. Despite recent effort to enhance safety in RL, a systematic understanding of the field remains difficult. This challenge stems from the diversity of constraint representations and little exploration of their interrelations. To bridge this knowledge gap, we present a comprehensive review of representative constraint formulations, along with a curated selection of algorithms designed specifically for each formulation. In addition, we elucidate the theoretical underpinnings that reveal the mathematical mutual relations among common problem formulations. We conclude with a discussion of the current state and future directions of safe reinforcement learning research.

cs.LG↗

arXiv2024

Probabilistic reachable sets of stochastic nonlinear systems with contextual uncertainties

Validating and controlling safety-critical systems in uncertain environments necessitates probabilistic reachable sets of future state evolutions. The existing methods of computing probabilistic reachable sets normally assume that stochastic uncertainties are independent of system states, inputs, and other environment variables. However, this assumption falls short in many real-world applications, where the probability distribution governing uncertainties depends on these variables, referred to as contextual uncertainties. This paper addresses the challenge of computing probabilistic reachable sets of stochastic nonlinear states with contextual uncertainties by seeking minimum-volume polynomial sublevel sets with contextual chance constraints. The formulated problem cannot be solved by the existing sample-based approximation method since the existing methods do not consider conditional probability densities. To address this, we propose a consistent sample approximation of the original problem by leveraging conditional density estimation and resampling. The obtained approximate problem is a tractable optimization problem. Additionally, we prove the proposed sample-based approximation's almost uniform convergence, showing that it gives the optimal solution almost consistently with the original ones. Through a numerical example, we evaluate the effectiveness of the proposed method against existing approaches, highlighting its capability to significantly reduce the bias inherent in sample-based approximation without considering a conditional probability density.

eess.SY↗

arXiv2024

Beyond Voice Assistants: Exploring Advantages and Risks of an In-Car Social Robot in Real Driving Scenarios

In-car Voice Assistants (VAs) play an increasingly critical role in automotive user interface design. However, existing VAs primarily perform simple 'query-answer' tasks, limiting their ability to sustain drivers' long-term attention. In this study, we investigate the effectiveness of an in-car Robot Assistant (RA) that offers functionalities beyond voice interaction. We aim to answer the question: How does the presence of a social robot impact user experience in real driving scenarios? Our study begins with a user survey to understand perspectives on in-car VAs and their influence on driving experiences. We then conduct non-driving and on-road experiments with selected participants to assess user experiences with an RA. Additionally, we conduct subjective ratings to evaluate user perceptions of the RA's personality, which is crucial for robot design. We also explore potential concerns regarding ethical risks. Finally, we provide a comprehensive discussion and recommendations for the future development of in-car RAs.

cs.HC↗

arXiv2023

Approximate Methods for Solving Chance Constrained Linear Programs in Probability Measure Space

A risk-aware decision-making problem can be formulated as a chance-constrained linear program in probability measure space. Chance-constrained linear program in probability measure space is intractable, and no numerical method exists to solve this problem. This paper presents numerical methods to solve chance-constrained linear programs in probability measure space for the first time. We propose two solvable optimization problems as approximate problems of the original problem. We prove the uniform convergence of each approximate problem. Moreover, numerical experiments have been implemented to validate the proposed methods.

math.OC↗

arXiv2023

Safe Exploration in Reinforcement Learning: A Generalized Formulation and Algorithms

Safe exploration is essential for the practical use of reinforcement learning (RL) in many real-world scenarios. In this paper, we present a generalized safe exploration (GSE) problem as a unified formulation of common safe exploration problems. We then propose a solution of the GSE problem in the form of a meta-algorithm for safe exploration, MASE, which combines an unconstrained RL algorithm with an uncertainty quantifier to guarantee safety in the current episode while properly penalizing unsafe explorations before actual safety violation to discourage them in future episodes. The advantage of MASE is that we can optimize a policy while guaranteeing with a high probability that no safety constraint will be violated under proper assumptions. Specifically, we present two variants of MASE with different constructions of the uncertainty quantifier: one based on generalized linear models with theoretical guarantees of safety and near-optimality, and another that combines a Gaussian process to ensure safety with a deep RL algorithm to maximize the reward. Finally, we demonstrate that our proposed algorithm achieves better performance than state-of-the-art algorithms on grid-world and Safety Gym benchmarks without violating any safety constraints, even during training.

cs.LG↗

arXiv2023

Chance-constrained probability measure optimization

Stochastic optimization with chance constraints often relies on deterministic decision-making, where optimal decisions are fixed and may be applied to a system repeatedly. A critical question arises: Can probabilistic decision-making, where probability measures/distributions are considered as decision variables, outperform deterministic decision-making in terms of the expected performance when chance constraints are present? This paper addresses this question by introducing the \textit{Chance-Constrained Probability Measure Optimization} (CCPMO) framework, which formulates the problem of optimizing probabilistic decisions under chance constraints. We first establish the existence of the optimal solution to CCPMO. Crucially, we prove that the optimal probabilistic decisions can always be represented by a probability measure concentrated on only two points, thereby reducing the CCPMO problem to an equivalent, simplified form. To solve this reduced problem, we propose a sample-based smooth approximation method. This approach leverages samples of model uncertainties to construct an approximate problem with uniform convergence and probabilistic feasibility guarantees. The approximate problem can be efficiently solved using standard nonlinear programming techniques. Finally, we validate the proposed framework through a numerical example of a quadrotor control problem under turbulent conditions. The results demonstrate that probabilistic decision-making outperforms deterministic approaches in expected performance while satisfying safety-critical chance constraints.

math.OC↗

arXiv2023

Bayesian Meta-Learning on Control Barrier Functions with Data from On-Board Sensors

In this paper, we consider a way to safely navigate the robots in unknown environments using measurement data from sensory devices. The control barrier function (CBF) is one of the promising approaches to encode safety requirements of the system and the recent progress on learning-based approaches for CBF realizes online synthesis of CBF-based safe controllers with sensor measurements. However, the existing methods are inefficient in the sense that the trained CBF cannot be generalized to different environments and the re-synthesis of the controller is necessary when changes in the environment occur. Thus, this paper considers a way to learn CBF that can quickly adapt to a new environment with few amount of data by utilizing the currently developed Bayesian meta-learning framework. The proposed scheme realizes efficient online synthesis of the controller as shown in the simulation study and provides probabilistic safety guarantees on the resulting controller.

eess.SY↗

arXiv2020

Receding Horizon Optimization for Disturbance-aware Predictive Control of Power Electronic Inverters

A disturbance-aware predictive control policy is proposed for DC-AC power inverters with the receding horizon optimization approach. First, a discrete event-driven hybrid automaton model has been constructed for the nonlinear inverter system dynamics. A control problem of infinite discrete state-space transition sequence optimization is formulated. A receding horizon optimization approach is applied to solve the discrete optimization problem piece-wisely on-line. Accordingly, disturbance-aware adaptive control is proposed, the external disturbance is sampled and estimated by an on-line Recursive Least Square (RLS) algorithm. Then it is elaborated that the conventional PWM control solution is a subset of solutions of the proposed control strategy and the code-transition between them is provided. By adding extra PWM constraints to the proposed control strategy, an Optimal PWM Control Mode (OPCM) is introduced as an example. The proposed controller can freely operate under the original Optimal Discrete Control Mode (ODCM) and the OPCM. The numerical simulation results have verified that the proposed discrete control strategy has realized disturbance-aware adaptive control of DC-AC inversion against load-shift, and ODCM has better control performance than OPCM. In addition, the proposed modeling and control frame has the potential to support other forms of control modes.

eess.SY↗

arXiv2020

Statistical Simulator for the Engine Knock

This paper proposes a statistical simulator for the engine knock based on the Mixture Density Network (MDN) and the accept-reject method. The proposed simulator can generate the random knock intensity signal corresponding to the input signal. The generated knock intensity has a consistent probability distribution with the real engine. Firstly, the statistical analysis is conducted with the experimental data. From the analysis results, some important assumptions on the statistical properties of the knock intensity are made. Regarding the knock intensity as a random variable on the discrete-time index, it is independent and identically distributed if the input of the engine is identical. The probability distribution of the knock intensity under identical input can be approximated by the Gaussian Mixture Model(GMM). The parameter of the GMM is a function of the input. Based on these assumptions, two sub-problems for establishing the statistical simulator are formulated: One is to approximate the function from input to the parameters of the knock intensity distribution with an absolutely continuous function; The other one is to design a random number generator that outputs the random data consistent with the given distribution. The MDN is applied to approximate the probability density of the knock intensity and the accept-reject algorithm is used for the random number generator design. The proposed method is evaluated in experimental data-based validation.

stat.AP↗

arXiv2019

Approximate Uncertain Program

Chance constrained program where one seeks to minimize an objective over decisions which satisfy randomly disturbed constraints with a given probability is computationally intractable. This paper proposes an approximate approach to address chance constrained program. Firstly, a single layer neural-network is used to approximate the function from decision domain to violation probability domain. The algorithm for updating parameters in single layer neural-network adopts sequential extreme learning machine. Based on the neural violation probability approximate model, a randomized algorithm is then proposed to approach the optimizer in the probabilistic feasible domain of decision. In the randomized algorithm, samples are extracted from decision domain uniformly at first. Then, violation probabilities of all samples are calculated according to neural violation probability approximate model. The ones with violation probability higher than the required level are discarded. The minimizer in the remained feasible decision samples is used to update sampling policy. The policy converges to the optimal feasible decision. Numerical simulations are implemented to validate the proposed method for non-convex problems comparing with scenario approach and parallel randomized algorithm. The result shows that proposed method have improved performance.

stat.CO↗

arXiv2019

Parallel Randomized Algorithm for Chance Constrained Program

Chance constrained program is computationally intractable due to the existence of chance constraints, which are randomly disturbed and should be satisfied with a probability. This paper proposes a two-layer randomized algorithm to address chance constrained program. Randomized optimization is applied to search the optimizer which satisfies chance constraints in a framework of parallel algorithm. Firstly, multiple decision samples are extracted uniformly in the decision domain without considering the chance constraints. Then, in the second sampling layer, violation probabilities of all the extracted decision samples are checked by extracting the disturbance samples and calculating the corresponding violation probabilities. The decision samples with violation probabilities higher than the required level are discarded. The minimizer of the cost function among the remained feasible decision samples are used to update optimizer iteratively. Numerical simulations are implemented to validate the proposed method for non-convex problems comparing with scenario approach. The proposed method exhibits better robustness in finding probabilistic feasible optimizer.

math.OC↗