arXiv subjects

Li Xia

Publications and source records attributed to Li Xia.

At least 19 recordsLinked to original sources

arXiv2026

Discovery under Hypothesis Redundancy: A Geometric Theory of Discovery Bottlenecks

Scientific discovery saturates when new hypotheses cease to provide independent information, even if the nominal hypothesis space remains large. We study hybrid discovery systems that combine structured local search with LLM-generated non-local proposals and pose the Search Compression Hypothesis: non-local exploration helps only when three geometric conditions co-occur: spectral compression, orthogonal escape from the explored span, and residual signal alignment with the target. We formalize these conditions, derive necessary conditions for hybrid advantage, and test the mechanism in controlled synthetic environments, large-scale A-share factor discovery, and symbolic-regression benchmarks; a public tabular operational sanity check tests the associated budget-allocation implication. Signal-planting and directed-versus-random experiments show that novelty alone is insufficient: random orthogonal jumps expand coverage but do not improve yield without predictive alignment. Across compression sweeps, real factor archives, and LLM-SRBench tasks, hybrid gains concentrate in weakly represented but target-bearing directions and vanish as the hypothesis space approaches full rank. The framework turns LLM-guided discovery from generic novelty search into a diagnostic procedure for deciding when directed non-local exploration is warranted.

cs.LG↗

arXiv2026

Long-Run Conditional Value-at-Risk Reinforcement Learning

Conditional value-at-risk (CVaR) is a prominent risk measure in financial engineering, energy systems, and supply chain management. In these domains, Markov decision processes (MDPs) with a long-run CVaR criterion effectively mitigate cost variability over a specified horizon. However, implementing MDPs relies on known transition models, which are typically unavailable in practice. This necessitates a model-free approach to risk-sensitive dynamic optimization. To tackle this challenge, we propose a reinforcement learning algorithm that simultaneously conducts policy evaluation and improvement based on a CVaR-specific Bellman local optimality equation. This algorithm employs a nonparametric incremental learning approach for policy improvement, relying on a single sample trajectory to identify the optimal policy. Under appropriate technical conditions, we prove almost sure convergence of the algorithm and derive its convergence rate. Our analysis reveals that the optimal convergence rate, measured by the mean absolute error of policy estimators, is of order O(1/n). Our main algorithm and results are further extended to solving the mean-CVaR optimization problem. Numerical experiments corroborate these results.

math.OC↗

arXiv2025

Communication-Efficient Neural Tangent Kernels for Heterogeneous Decentralized Federated Learning

Decentralized federated learning (DFL) enables collaborative model training without a central server, but converges slowly under statistical heterogeneity. Recent work has shown that neural tangent kernel (NTK) methods achieve faster convergence than gradient-based updates in DFL, while momentum has proven effective for accelerating gradient-based FL. However, applying momentum to NTK updates can destabilize training under heterogeneous data. We propose SPARK, which addresses this instability with a stage-wise annealed soft-label regularizer evaluated on neighborhood-aggregated data, so that momentum can accelerate NTK updates stably. Under high heterogeneity, SPARK converges about 3$\times$ faster than baselines and lowers the total communication to a target accuracy by up to about 70\%, and it attains higher accuracy across heterogeneity levels. We further study random projection as an optional Jacobian-compression strategy for bandwidth-constrained settings. We validate the approach across multiple datasets, network topologies, and heterogeneity levels.

cs.LG↗

arXiv2025

Independent policy gradient-based reinforcement learning for economic and reliable energy management of multi-microgrid systems

Efficiency and reliability are both crucial for energy management, especially in multi-microgrid systems (MMSs) integrating intermittent and distributed renewable energy sources. This study investigates an economic and reliable energy management problem in MMSs under a distributed scheme, where each microgrid independently updates its energy management policy in a decentralized manner to optimize the long-term system performance collaboratively. We introduce the mean and variance of the exchange power between the MMS and the main grid as indicators for the economic performance and reliability of the system. Accordingly, we formulate the energy management problem as a mean-variance team stochastic game (MV-TSG), where conventional methods based on the maximization of expected cumulative rewards are unsuitable for variance metrics. To solve MV-TSGs, we propose a fully distributed independent policy gradient algorithm, with rigorous convergence analysis, for scenarios with known model parameters. For large-scale scenarios with unknown model parameters, we further develop a deep reinforcement learning algorithm based on independent policy gradients, enabling data-driven policy optimization. Numerical experiments in two scenarios validate the effectiveness of the proposed methods. Our approaches fully leverage the distributed computational capabilities of MMSs and achieve a well-balanced trade-off between economic performance and operational reliability.

eess.SY↗

arXiv2025

Trajectory-Aware Information Matching for Multi-Step Gradient Inversion in Federated Learning

Federated learning enables distributed information sharing and collaborative model training without exposing raw client data. However, shared gradients or model updates may still contain sensitive information, making federated learning vulnerable to gradient inversion attacks. Most existing gradient inversion attacks rely on simplified update observations, such as single-step gradients or endpoint-based matching. In practical FL, however, FedAvg produces an accumulated trajectory-dependent update after multiple local steps, rather than a gradient computed at a single model state.To address this issue, we propose NL-SME, a trajectory-aware information matching method for multi-step gradient inversion. NL-SME constructs a learnable nonlinear surrogate trajectory to approximate hidden local states and integrates trajectory-level information with calibrated gradient matching. For perturbed updates, NL-SME can further use an observed-update reliability-aware strategy to reduce the influence of unreliable components. Extensive experiments under diverse multi-step FedAvg settings show that NL-SME outperforms state-of-the-art gradient inversion baselines in reconstruction quality and update-matching accuracy. Additional evaluations on natural and medical images, as well as under fused-update observations and representative defense strategies, further suggest that observable multi-step updates may still retain reconstruction signals. These results reveal potential privacy leakage risks in federated information sharing. Code is available at https://anonymous.4open.science/r/NL-SME-main/README.md.

cs.LG↗

arXiv2025

Sharpe Ratio Optimization in Markov Decision Processes

Sharpe ratio (also known as reward-to-variability ratio) is a widely-used metric in finance, which measures the additional return at the cost of per unit of increased risk (standard deviation of return). However, the optimization of Sharpe ratio in Markov decision processes (MDPs) is challenging, because there exist two difficulties hindering the application of dynamic programming. One is that dynamic programming does not work for fractional objectives, and the other is that dynamic programming is invalid for risk metrics. In this paper, we study the Sharpe ratio optimization in infinite-horizon MDPs, considering both the long-run average and discounted settings. We address the first challenge with the Dinkelbachs transform, which converts the Sharpe ratio objective to a mean-squared-variance (M2V) objective. It is shown that the M2V optimization and the original Sharpe ratio optimization share the same optimal policy when the risk-sensitive parameter is equal to the optimal Sharpe ratio. For the second challenge, we develop an iterative algorithm to solve the M2V optimization which is similar to a mean-variance optimization in MDPs. We iteratively solve the M2V problem and obtain the associated Sharpe ratio that is used to update the risk-sensitive parameter in the next iteration of M2V problems. We show that such a sequence of Sharpe ratios derived is monotonically increasing and converges to the optimal Sharpe ratio. For both average and discounted MDP settings, we develop a policy iteration procedure and prove its convergence to the optimum. Numerical experiments are conducted for validation. To the best of our knowledge, our approach is the first that solves the Sharpe ratio optimization in MDPs with dynamic programming type algorithms. We believe that the proposed algorithm can shed light on solving MDPs with other fractional objectives.

cs.AI↗

arXiv2025

Finite-Horizon Partially Observable Semi-Markov Games with Risk Probability Criteria

This paper studies partially observable two-person zero-sum semi-Markov games under a probability criterion, in which the system state may not be completely observed. It focuses on the probability that the accumulated rewards of player 1 (i.e., the incurred costs of player 2) fall short of a specified target at the terminal stage, which represents the risk of player 1 and the capacity of player 2. We study the game model via the technology of augmenting state space with the joint conditional distribution of the current unobserved state and the remaining goal. Under a mild condition, we establish a comparison theorem and derive the Shapley equation for the probability criterion. As a consequence, we prove the existence and the uniqueness of the value function and the existence of a Nash equilibrium.

math.OC↗

arXiv2025

Mean-Variance Optimization and Algorithm for Finite-Horizon Markov Decision Processes

Multi-period mean-variance optimization is a long-standing problem, caused by the failure of dynamic programming principle. This paper studies the mean-variance optimization in a setting of finite-horizon discrete-time Markov decision processes (MDPs), where the objective is to maximize the combined metrics of mean and variance of the accumulated rewards at terminal stage. By introducing the concepts of pseudo mean and pseudo variance, we convert the original mean-variance MDP to a bilevel MDP, where the outer is a single parameter optimization of the pseudo mean and the inner is a standard finite-horizon MDP with an augmented state space by adding an auxiliary state of accumulated rewards. We further study the properties of this bilevel MDP, including the optimality of history-dependent deterministic policies and the piecewise quadratic concavity of the inner MDPs' optimal values with respect to the pseudo mean. To efficiently solve this bilevel MDP, we propose an iterative algorithm that alternatingly updates the inner optimal policy and the outer pseudo mean. We prove that this algorithm converges to a local optimum. We also derive a sufficient condition under which our algorithm converges to the global optimum. Furthermore, we apply this approach to study the mean-variance optimization of multi-period portfolio selection problem, which shows that our approach exactly coincides with the classical result by Li and Ng (2000) in financial engineering. Our approach builds a new avenue to solve mean-variance optimization problems and has wide applicability to any problem modeled by MDPs, which is further demonstrated by examples of mean-variance optimization for queueing control and inventory management.

math.OC↗

arXiv2025

Team Variance Optimization of n-Player Stochastic Games with Separately Controlled Chains

In this paper, we study a subclass of n-player stochastic games, in which each player has their own internal state controlled only by their own action and their objective is a common goal called team variance which measures the total variation of the random rewards of all players. It is assumed that players cannot observe each others' state/action. Thus, players' internal chains are controlled separately by their own action and they are coupled through the objective of team variance. Since the variance metric is not additive or Markovian, the dynamic programming principle fails in this problem. We study this problem from the viewpoint of sensitivity-based optimization. A difference formula and a derivative formula for team variance with respect to policy perturbations are derived, which provide sensitivity information to guide decentralized optimization. The existence of a stationary pure Nash equilibrium policy is derived. We further develop a bilevel optimization algorithm that iteratively updates the team mean at the outer level and minimizes the team variance at the inner level, where the team mean serves as a signal to coordinate the optimization of n players in a decentralized manner. We prove that the algorithm can converge to a strictly local minimum or a first-order stationary point in the space of mixed policies. Finally, we demonstrate the effectiveness of our approach using a numerical experiment of energy management in smart grid, where the assumption of separately controlled chains holds naturally.

math.OC↗

arXiv2025

Markov Decision Processes with Value-at-Risk Criterion

Value-at-risk (VaR), also known as quantile, is a crucial risk measure in finance and other fields. However, optimizing VaR metrics in Markov decision processes (MDPs) is challenging because VaR is non-additive and the traditional dynamic programming is inapplicable. This paper conducts a comprehensive study on VaR optimization in discrete-time finite MDPs. We consider VaR in two key scenarios: the VaR of steady-state rewards over an infinite horizon and the VaR of accumulated rewards over a finite horizon. By establishing the equivalence between the VaR maximization MDP and a series of probabilistic minimization MDPs, we transform the VaR maximization MDP into a constrained bilevel optimization problem. The inner-level is a policy optimization of minimizing the probability that MDP rewards fall below a target $\lambda$, while the outer-level is a single parameter optimization of $\lambda$, representing the target VaR. For the steady-state scenario, the probabilistic minimization MDP can be resolved using the expectation criterion of a standard MDP. In contrast, for the finite-horizon case, it can be addressed via an augmented-state MDP. We prove the optimality of deterministic stationary policies for steady-state VaR MDPs and deterministic history-dependent policies for finite-horizon VaR MDPs. We derive both a policy improvement rule and a necessary-sufficient condition for optimal policies. Furthermore, we develop policy iteration type algorithms to maximize the VaR in MDPs and prove their convergence. Our results are also extended to the counterpart of VaR minimization MDPs after appropriate modifications. Finally, we conduct numerical experiments to demonstrate the computational efficiency and practical applicability of our approach. Our study paves a novel way to explore the optimization of quantile-related metrics in MDPs through the duality between quantiles and probabilities.

math.OC↗

arXiv2025

Policy Optimization and Multi-agent Reinforcement Learning for Mean-variance Team Stochastic Games

We study a long-run mean-variance team stochastic game (MV-TSG), where each agent shares a common mean-variance objective for the system and takes actions independently to maximize it. MV-TSG has two main challenges. First, the variance metric is neither additive nor Markovian in a dynamic setting. Second, simultaneous policy updates of all agents lead to a non-stationary environment for each individual agent. Both challenges make dynamic programming inapplicable. In this paper, we study MV-TSGs from the perspective of sensitivity-based optimization. The performance difference and performance derivative formulas for joint policies are derived, which provide optimization information for MV-TSGs. We prove the existence of a deterministic Nash policy for this problem. Subsequently, we propose a Mean-Variance Multi-Agent Policy Iteration (MV-MAPI) algorithm with a sequential update scheme, where individual agent policies are updated one by one in a given order. We prove that the MV-MAPI algorithm converges to a first-order stationary point of the objective function. By analyzing the local geometry of stationary points, we derive specific conditions for stationary points to be (local) Nash equilibria, and further, strict local optima. To solve large-scale MV-TSGs in scenarios with unknown environmental parameters, we extend the idea of trust region methods to MV-MAPI and develop a multi-agent reinforcement learning algorithm named Mean-Variance Multi-Agent Trust Region Policy Optimization (MV-MATRPO). We derive a performance lower bound for each update of joint policies. Finally, numerical experiments on energy management in multiple microgrid systems are conducted.

cs.MA↗

arXiv2023

On the Maximization of Long-Run Reward CVaR for Markov Decision Processes

This paper studies the optimization of Markov decision processes (MDPs) from a risk-seeking perspective, where the risk is measured by conditional value-at-risk (CVaR). The objective is to find a policy that maximizes the long-run CVaR of instantaneous rewards over an infinite horizon across all history-dependent randomized policies. By establishing two optimality inequalities of opposing directions, we prove that the maximum of long-run CVaR of MDPs over the set of history-dependent randomized policies can be found within the class of stationary randomized policies. In contrast to classical MDPs, we find that there may not exist an optimal stationary deterministic policy for maximizing CVaR. Instead, we prove the existence of an optimal stationary randomized policy that requires randomizing over at most two actions. Via a convex optimization representation of CVaR, we convert the long-run CVaR maximization MDP into a minimax problem, where we prove the interchangeability of minimum and maximum and the related existence of saddle point solutions. Furthermore, we propose an algorithm that finds the saddle point solution by solving two linear programs. These results are then extended to objectives that involve maximizing some combination of mean and CVaR of rewards simultaneously. Finally, we conduct numerical experiments to demonstrate the main results.

math.OC↗

arXiv2023

Global Algorithms for Mean-Variance Optimization in Markov Decision Processes

Dynamic optimization of mean and variance in Markov decision processes (MDPs) is a long-standing challenge caused by the failure of dynamic programming. In this paper, we propose a new approach to find the globally optimal policy for combined metrics of steady-state mean and variance in an infinite-horizon undiscounted MDP. By introducing the concepts of pseudo mean and pseudo variance, we convert the original problem to a bilevel MDP problem, where the inner one is a standard MDP optimizing pseudo mean-variance and the outer one is a single parameter selection problem optimizing pseudo mean. We use the sensitivity analysis of MDPs to derive the properties of this bilevel problem. By solving inner standard MDPs for pseudo mean-variance optimization, we can identify worse policy spaces dominated by optimal policies of the pseudo problems. We propose an optimization algorithm which can find the globally optimal policy by repeatedly removing worse policy spaces. The convergence and complexity of the algorithm are studied. Another policy dominance property is also proposed to further improve the algorithm efficiency. Numerical experiments demonstrate the performance and efficiency of our algorithms. To the best of our knowledge, our algorithm is the first that efficiently finds the globally optimal policy of mean-variance optimization in MDPs. These results are also valid for solely minimizing the variance metrics in MDPs.

math.OC↗

arXiv2022

Optimal Systemic Risk Bailout: A PGO Approach Based on Neural Network

In the financial system, bailout strategies play a pivotal role in mitigating substantial losses resulting from systemic risk. However, the lack of a closed-form objective function to the optimal bailout problem poses significant challenges in its resolution. This paper conceptualizes the optimal bailout (capital injection) problem as a black-box optimization task, where the black box is modeled as a fixed-point system consistent with the E-N framework for measuring systemic risk in the financial system. To address this challenge, we propose a novel framework, "Prediction-Gradient-Optimization" (PGO). Within PGO, the Prediction employs a neural network to approximate and forecast the objective function implied by the black box, which can be completed offline; For the online usage, the Gradient step derives gradient information from this approximation, and the Optimization step uses a gradient projection algorithm to solve the problem effectively. Extensive numerical experiments highlight the effectiveness of the proposed approach in managing systemic risk.

q-fin.RM↗

arXiv2022

Risk-Sensitive Markov Decision Processes with Long-Run CVaR Criterion

CVaR (Conditional Value at Risk) is a risk metric widely used in finance. However, dynamically optimizing CVaR is difficult since it is not a standard Markov decision process (MDP) and the principle of dynamic programming fails. In this paper, we study the infinite-horizon discrete-time MDP with a long-run CVaR criterion, from the view of sensitivity-based optimization. By introducing a pseudo CVaR metric, we derive a CVaR difference formula which quantifies the difference of long-run CVaR under any two policies. The optimality of deterministic policies is derived. We obtain a so-called Bellman local optimality equation for CVaR, which is a necessary and sufficient condition for local optimal policies and only necessary for global optimal policies. A CVaR derivative formula is also derived for providing more sensitivity information. Then we develop a policy iteration type algorithm to efficiently optimize CVaR, which is shown to converge to local optima in the mixed policy space. We further discuss some extensions including the mean-CVaR optimization and the maximization of CVaR. Finally, we conduct numerical experiments relating to portfolio management to demonstrate the main results. Our work may shed light on dynamically optimizing CVaR from a sensitivity viewpoint.

math.OC↗

arXiv2022

Distributionally Robust Offline Reinforcement Learning with Linear Function Approximation

Among the reasons hindering reinforcement learning (RL) applications to real-world problems, two factors are critical: limited data and the mismatch between the testing environment (real environment in which the policy is deployed) and the training environment (e.g., a simulator). This paper attempts to address these issues simultaneously with distributionally robust offline RL, where we learn a distributionally robust policy using historical data obtained from the source environment by optimizing against a worst-case perturbation thereof. In particular, we move beyond tabular settings and consider linear function approximation. More specifically, we consider two settings, one where the dataset is well-explored and the other where the dataset has sufficient coverage of the optimal policy. We propose two algorithms~-- one for each of the two settings~-- that achieve error bounds $\tilde{O}(d^{1/2}/N^{1/2})$ and $\tilde{O}(d^{3/2}/N^{1/2})$ respectively, where $d$ is the dimension in the linear function approximation and $N$ is the number of trajectories in the dataset. To the best of our knowledge, they provide the first non-asymptotic results of the sample complexity in this setting. Diverse experiments are conducted to demonstrate our theoretical findings, showing the superiority of our algorithm against the non-robust one.

cs.LG↗

arXiv2022

Dominant Eigenvalue-Eigenvector Pair Estimation via Graph Infection

We present a novel method to estimate the dominant eigenvalue and eigenvector pair of any non-negative real matrix via graph infection. The key idea in our technique lies in approximating the solution to the first-order matrix ordinary differential equation (ODE) with the Euler method. Graphs, which can be weighted, directed, and with loops, are first converted to its adjacency matrix A. Then by a naive infection model for graphs, we establish the corresponding first-order matrix ODE, through which A's dominant eigenvalue is revealed by the fastest growing term. When there are multiple dominant eigenvalues of the same magnitude, the classical power iteration method can fail. In contrast, our method can converge to the dominant eigenvalue even when same-magnitude counterparts exist, be it complex or opposite in sign. We conduct several experiments comparing the convergence between our method and power iteration. Our results show clear advantages over power iteration for tree graphs, bipartite graphs, directed graphs with periods, and Markov chains with spider-traps. To our knowledge, this is the first work that estimates dominant eigenvalue and eigenvector pair from the perspective of a dynamical system and matrix ODE. We believe our method can be adopted as an alternative to power iteration, especially for graphs.

math.NA↗

arXiv2022

Mean-Semivariance Policy Optimization via Risk-Averse Reinforcement Learning

Keeping risk under control is often more crucial than maximizing expected rewards in real-world decision-making situations, such as finance, robotics, autonomous driving, etc. The most natural choice of risk measures is variance, which penalizes the upside volatility as much as the downside part. Instead, the (downside) semivariance, which captures the negative deviation of a random variable under its mean, is more suitable for risk-averse proposes. This paper aims at optimizing the mean-semivariance (MSV) criterion in reinforcement learning w.r.t. steady reward distribution. Since semivariance is time-inconsistent and does not satisfy the standard Bellman equation, the traditional dynamic programming methods are inapplicable to MSV problems directly. To tackle this challenge, we resort to Perturbation Analysis (PA) theory and establish the performance difference formula for MSV. We reveal that the MSV problem can be solved by iteratively solving a sequence of RL problems with a policy-dependent reward function. Further, we propose two on-policy algorithms based on the policy gradient theory and the trust region method. Finally, we conduct diverse experiments from simple bandit problems to continuous control tasks in MuJoCo, which demonstrate the effectiveness of our proposed methods.

cs.LG↗