arXiv subjects

Fangfei Li

Publications and source records attributed to Fangfei Li.

17 recordsLinked to original sources

arXiv2026

Submodular Policy Learning for Distributed Task Allocation in Open Multi-Agent Systems

This paper studies policy learning for distributed task allocation in open multi-agent systems, where agents may join and leave in a time-varying fashion, with submodular stage team utilities. At each time, the active agents select actions from local categorical policies such that the feasible joint agent-action pairs form a partition matroid. Standard continuous relaxations of submodular set functions are based on independent Bernoulli sampling, making them inconsistent with agents' policies.To solve this mismatch, we propose the \emph{partition multilinear extension} (PME), a policy-based relaxation whose continuous support matches feasible actions under categorical policies.We prove that the marginal gains of the stage utility provide an unbiased estimator of the gradient of the PME and that maximizing the PME over action distributions is equivalent to maximizing the stage utilities over agent actions, which are critical to devise principled policy gradient.Building on this, we design \emph{SubMAPL}, a centralized-training decentralized-execution KL-mirror policy-learning method that uses local marginal gains as stochastic PME gradients during training. KL-mirror updates preserve categorical feasibility without Euclidean projection.In the case where agents run tabular-softmax policies, we introduce open policy migration and an open-system KL tracking variation to handle agent arrivals and departures. Using dynamic regret analysis, we establish a lower bound on the cumulative utility which accounts for the openness of the environment and for the gap between optimal stage-wise and global utilities. Simulations on multi-agent coverage demonstrate that SubMAPL outperforms policy-gradient and online-learning baselines.

cs.MA↗

arXiv2026

Diffusion-Guided Uncertainty-Aware Delayed Policy Optimization

Reinforcement learning in real world environments often suffers from severe performance degradation due to delayed feedback. Existing approaches typically mitigate performance degradation caused by observation delays by constructing augmented states or predicting the true states. However, these methods often overlook the inherent discrepancy between delayed state and true states induced by stochastic MDP. We theoretically prove the existence of such a discrepancy and show that it leads to the degradation of the optimal policy. To address this challenge, we propose Diffusion Guided Uncertainty Aware Delayed Policy Optimization (DUPO). Our method explicitly models the relationship between delayed state message and the current state using a diffusion model, and leverages the resulting discrepancy estimates to weight delayed policies. Extensive experiments on continuous robotic control tasks with multiple stochastic delays demonstrate that DUPO consistently outperforms existing methods and remains effective even under long and random delay scenarios.

cs.AI↗

arXiv2026

CLAP: Closed-Loop Training, Evaluation, and Release Control for Domain Agent Post-training

Domain agents often face noisy business data, uncertain post-training gains, offline/application mismatch, and adapter-release risk. This paper presents CLAP (Closed-Loop Agent Post-training), a closed-loop method that converts business data into structured SFT samples, decision-preference samples, holdout sets, risk diagnostics, and release-gate records. CLAP combines data validation, target/evidence normalization, reward/KL diagnosis, offline gates, and application-chain replay to decide whether an adapter is suitable for the target application chain. On five anonymized manufacturing-scenario batches, QLoRA-style LoRA-SFT yields modest average gains: overall score increases by 0.0098, pass rate by 0.0240, and evidence accuracy by 0.0280, while hallucination and wrong facts decrease. Yet only 3 of 5 batches improve, some batches regress, and GRPO exposes high KL risks. Application-chain replay further shows that RAG is necessary for factual extraction; under the same 3B backbone and 100 replay cases, an application-RAG-oriented LoRA-SFT adapter improves value, core fields, and answer-evidence doc/page matching over base+RAG, but increases latency. These results support managing domain-agent post-training through an integrated data-training-evaluation-release loop rather than relying on training completion or a single offline score.

cs.AI↗

arXiv2026

Submodular Multi-Agent Policy Learning for Online Distributed Task Allocation in Open Multi-Agent Systems

This paper studies multi-agent reinforcement learning with submodular team utilities for online distributed task allocation. In this setting, each agent selects one action from a local categorical policy, so feasible joint actions form a partition matroid over agent-action pairs. Classical multilinear extensions use independent Bernoulli sampling and therefore do not match the categorical policies executed by decentralized agents. To address this mismatch, we introduce the Partition Multilinear Extension (PME), a continuous relaxation whose value equals the expected team utility under factorized categorical policies. We prove that submodular difference rewards provide unbiased PME marginal-gradient information and yield a stagewise score-function policy-gradient estimator. Based on this connection, we propose SubMAPG, a centralized-training decentralized-execution policy-gradient framework with masked categorical policies and submodular difference-reward training signals. For the associated PME marginal-space projected stochastic-gradient dynamics, we prove a stagewise 1/2-approximation guarantee and sublinear dynamic regret in slowly varying environments, measured by the path length of the optimal PME marginals. To handle open systems with time-varying agents and targets, we instantiate SubMAPG with graph neural network policies. Experiments on multi-robot coverage and multi-target tracking show that SubMAPG outperforms local greedy and shared-reward baselines and is competitive with centralized myopic greedy strategies.

eess.SY↗

arXiv2025

HCPO: Hierarchical Conductor-Based Policy Optimization in Multi-Agent Reinforcement Learning

In cooperative Multi-Agent Reinforcement Learning (MARL), efficient exploration is crucial for optimizing the performance of joint policy. However, existing methods often update joint policies via independent agent exploration, without coordination among agents, which inherently constrains the expressive capacity and exploration of joint policies. To address this issue, we propose a conductor-based joint policy framework that directly enhances the expressive capacity of joint policies and coordinates exploration. In addition, we develop a Hierarchical Conductor-based Policy Optimization (HCPO) algorithm that instructs policy updates for the conductor and agents in a direction aligned with performance improvement. A rigorous theoretical guarantee further establishes the monotonicity of the joint policy optimization process. By deploying local conductors, HCPO retains centralized training benefits while eliminating inter-agent communication during execution. Finally, we evaluate HCPO on three challenging benchmarks: StarCraftII Multi-agent Challenge, Multi-agent MuJoCo, and Multi-agent Particle Environment. The results indicate that HCPO outperforms competitive MARL baselines regarding cooperative efficiency and stability.

cs.LG↗

arXiv2025

Robust and Efficient Communication in Multi-Agent Reinforcement Learning

Multi-agent reinforcement learning (MARL) has made significant strides in enabling coordinated behaviors among autonomous agents. However, most existing approaches assume that communication is instantaneous, reliable, and has unlimited bandwidth; these conditions are rarely met in real-world deployments. This survey systematically reviews recent advances in robust and efficient communication strategies for MARL under realistic constraints, including message perturbations, transmission delays, and limited bandwidth. Furthermore, because the challenges of low-latency reliability, bandwidth-intensive data sharing, and communication-privacy trade-offs are central to practical MARL systems, we focus on three applications involving cooperative autonomous driving, distributed simultaneous localization and mapping, and federated learning. Finally, we identify key open challenges and future research directions, advocating a unified approach that co-designs communication, learning, and robustness to bridge the gap between theoretical MARL models and practical implementations.

cs.AI↗

arXiv2025

Secure Distributed Consensus Estimation under False Data Injection Attacks: A Defense Strategy Based on Partial Channel Coding

This article investigates the security issue caused by false data injection attacks in distributed estimation, wherein each sensor can construct two types of residues based on local estimates and neighbor information, respectively. The resource-constrained attacker can select partial channels from the sensor network and arbitrarily manipulate the transmitted data. We derive necessary and sufficient conditions to reveal system vulnerabilities, under which the attacker is able to diverge the estimation error while preserving the stealthiness of all residues. We propose two defense strategies with mechanisms of exploiting the Euclidean distance between local estimates to detect attacks, and adopting the coding scheme to protect the transmitted data, respectively. It is proven that the former has the capability to address the majority of security loopholes, while the latter can serve as an additional enhancement to the former. By employing the time-varying coding matrix to mitigate the risk of being cracked, we demonstrate that the latter can safeguard against adversaries injecting stealthy sequences into the encoded channels. Hence, drawing upon the security analysis, we further provide a procedure to select security-critical channels that need to be encoded, thereby achieving a trade-off between security and coding costs. Finally, some numerical simulations are conducted to demonstrate the theoretical results.

eess.SY↗

arXiv2025

Differential Game with Geometric Formulation for Differential Systems Under Bilateral Perception Constraints

This letter employs differential game theory to address the defense problem of a circular target area with perception constraints, involving a single defender and a single attacker. The defender is restricted to moving along the perimeter, while the mobile attacker aims to make itself to the edge of the circular target to win. We examine a scenario where both the attacker and defender face perception constraints, dividing the interaction into four distinct stages based on detection capabilities and deriving the corresponding optimal control strategies. Simulations are conducted to validate the proposed strategies.

math.OC↗

arXiv2025

Reinforcement Learning based Constrained Optimal Control: an Interpretable Reward Design

This paper presents an interpretable reward design framework for reinforcement learning based constrained optimal control problems with state and terminal constraints. The problem is formalized within a standard partially observable Markov decision process framework. The reward function is constructed from four weighted components: a terminal constraint reward, a guidance reward, a penalty for state constraint violations, and a cost reduction incentive reward. A theoretically justified reward design is then presented, which establishes bounds on the weights of the components. This approach ensures that constraints are satisfied and objectives are optimized while mitigating numerical instability. Acknowledging the importance of prior knowledge in reward design, we sequentially solve two subproblems, using each solution to inform the reward design for the subsequent problem. Subsequently, we integrate reinforcement learning with curriculum learning, utilizing policies derived from simpler subproblems to assist in tackling more complex challenges, thereby facilitating convergence. The framework is evaluated against original and randomly weighted reward designs in a multi-agent particle environment. Experimental results demonstrate that the proposed approach significantly enhances satisfaction of terminal and state constraints and optimization of control cost.

eess.SY↗

arXiv2024

Distributed Task Allocation for Multi-Agent Systems: A Submodular Optimization Approach

This paper addresses dynamic task allocation in resource-constrained multi-agent systems (MASs) with sequentially updated assignments. We develop a submodular maximization framework integrated with $q$-independence systems, demonstrating greater flexibility than conventional matroid-based constraints for modeling heterogeneous resource limitations. The proposed distributed greedy bundles algorithm (DGBA) addresses communication limitations in MASs while providing rigorous approximation guarantees for submodular maximization under a $q$-independence system constraint, ensuring low computational complexity. DGBA achieves feasible task allocation in polynomial time with reduced space complexity compared to existing methods. Extensive Monte Carlo simulations in a micro-satellite observation scenario demonstrate that DGBA consistently outperforms benchmark algorithms in total utility, resource efficiency, and assignment stability, while maintaining real-time computational feasibility.

cs.MA↗

arXiv2024

Computer vision tasks for intelligent aerospace missions: An overview

Computer vision tasks are crucial for aerospace missions as they help spacecraft to understand and interpret the space environment, such as estimating position and orientation, reconstructing 3D models, and recognizing objects, which have been extensively studied to successfully carry out the missions. However, traditional methods like Kalman Filtering, Structure from Motion, and Multi-View Stereo are not robust enough to handle harsh conditions, leading to unreliable results. In recent years, deep learning (DL)-based perception technologies have shown great potential and outperformed traditional methods, especially in terms of their robustness to changing environments. To further advance DL-based aerospace perception, various frameworks, datasets, and strategies have been proposed, indicating significant potential for future applications. In this survey, we aim to explore the promising techniques used in perception tasks and emphasize the importance of DL-based aerospace perception. We begin by providing an overview of aerospace perception, including classical space programs developed in recent years, commonly used sensors, and traditional perception methods. Subsequently, we delve into three fundamental perception tasks in aerospace missions: pose estimation, 3D reconstruction, and recognition, as they are basic and crucial for subsequent decision-making and control. Finally, we discuss the limitations and possibilities in current research and provide an outlook on future developments, including the challenges of working with limited datasets, the need for improved algorithms, and the potential benefits of multi-source information fusion.

cs.CV↗

arXiv2023

Enhanced Q-Learning Approach to Finite-Time Reachability with Maximum Probability for Probabilistic Boolean Control Networks

In this paper, we investigate the problem of controlling probabilistic Boolean control networks (PBCNs) to achieve reachability with maximum probability in the finite time horizon. We address three questions: 1) finding control policies that achieve reachability with maximum probability under fixed, and particularly, varied finite time horizon, 2) leveraging prior knowledge to solve question 1) with faster convergence speed in scenarios where time is a variable framework, and 3) proposing an enhanced Q-learning (QL) method to efficiently address the aforementioned questions for large-scale PBCNs. For question 1), we demonstrate the applicability of QL method on the finite-time reachability problem. For question 2), considering the possibility of varied time frames, we incorporate transfer learning (TL) technique to leverage prior knowledge and enhance convergence speed. For question 3), an enhanced model-free QL approach that improves upon the traditional QL algorithm by introducing memory-efficient modifications to address these issues in large-scale PBCNs effectively. Finally, we apply the proposed method to two examples: a small-scale PBCN and a large-scale PBCN, demonstrating the effectiveness of our approach.

eess.SY↗

arXiv2023

Q-learning Based Optimal False Data Injection Attack on Probabilistic Boolean Control Networks

In this paper, we present a reinforcement learning (RL) method for solving optimal false data injection attack problems in probabilistic Boolean control networks (PBCNs) where the attacker lacks knowledge of the system model. Specifically, we employ a Q-learning (QL) algorithm to address this problem. We then propose an improved QL algorithm that not only enhances learning efficiency but also obtains optimal attack strategies for large-scale PBCNs that the standard QL algorithm cannot handle. Finally, we verify the effectiveness of our proposed approach by considering two attacked PBCNs, including a 10-node network and a 28-node network.

eess.SY↗

arXiv2023

Reinforcement Learning Based Minimum State-flipped Control for the Reachability of Boolean Control Networks

This paper proposes model-free reinforcement learning methods for minimum-cost state-flipped control in Boolean control networks (BCNs). We tackle two questions: 1) finding the flipping kernel, namely the flip set with the smallest cardinality ensuring reachability, and 2) deriving optimal policies to minimize the number of flipping actions for reachability based on the obtained flipping kernel. For question 1), Q-learning's capability in determining reachability is demonstrated. To expedite convergence, we incorporate two improvements: i) demonstrating that previously reachable states remain reachable after adding elements to the flip set, followed by employing transfer learning, and ii) initiating each episode with special initial states whose reachability to the target state set are currently unknown. Question 2) requires optimal control with terminal constraints, while Q-learning only handles unconstrained problems. To bridge the gap, we propose a BCN-characteristics-based reward scheme and prove its optimality. Questions 1) and 2) with large-scale BCNs are addressed by employing small memory Q-learning, which reduces memory usage by only recording visited action-values. An upper bound on memory usage is provided to assess the algorithm's feasibility. To expedite convergence for question 2) in large-scale BCNs, we introduce adaptive variable rewards based on the known maximum steps needed to reach the target state set without cycles. Finally, the effectiveness of the proposed methods is validated on both small- and large-scale BCNs.

eess.SY↗

arXiv2023

Deep Reinforcement Learning Based Optimal Infinite-Horizon Control of Probabilistic Boolean Control Networks

In this paper, a deep reinforcement learning based method is proposed to obtain optimal policies for optimal infinite-horizon control of probabilistic Boolean control networks (PBCNs). Compared with the existing literatures, the proposed method is model-free, namely, the system model and the initial states needn't to be known. Meanwhile, it is suitable for large-scale PBCNs. First, we establish the connection between deep reinforcement learning and optimal infinite-horizon control, and structure the problem into the framework of the Markov decision process. Then, PBCNs are defined as large-scale or small-scale, depending on whether the memory of the action-values exceeds the RAM of the computer. Based on the newly introduced definition, Q-learning (QL) and double deep Q-network (DDQN) are applied to the optimal infinite-horizon control of small-scale and large-scale PBCNs, respectively. Meanwhile, the optimal state feedback controllers are designed. Finally, two examples are presented, which are a small-scale PBCN with 3 nodes, and a large-scale one with 28 nodes. To verify the convergence of QL and DDQN, the optimal control policy and the optimal action-values, which are obtained from both the algorithms, are compared with the ones based on a model-based method named policy iteration. Meanwhile, the performance of QL is compared with DDQN in the small-scale PBCN.

eess.SY↗

arXiv2016

Direct Meissner Effect Observation of Superconductivity in Compressed H2S

Recently, an extremely high superconducting temperature (Tc) of ~200 K has been reported in the sulfur hydride system above 100 GPa. This result is supported by theoretical predictions and verified experimentally. The crystal structure of the superconducting phase was also identified experimentally, confirming the theoretically predicted structure as well as a decomposition mechanism from H2S to H3S+S. Even though nuclear resonant scattering has been successfully used to provide magnetic evidence for a superconducting state, a direct measurement of the important Meissner effect is still lacking. Here we report in situ alternating-current magnetic susceptibility measurements on compressed H2S under high pressures. It is shown that superconductivity suddenly appears at 117 GPa and that Tc reaches 183 K at 149 GPa before decreasing monotonically with a further increase in pressure. This evolution agrees with both theoretical calculations and earlier experimental measurements. The idea of conventional high temperature superconductivity in hydrogen-dominant compounds has thus been realized in the sulfur hydride system under hydrostatic pressure, opening further exciting perspectives for possibly realizing room temperature superconductivity in hydrogen-based compounds.

cond-mat.supr-con↗

arXiv2013

A Novel High-Pressure Monoclinic Metallic Phase of V2O3

Vanadium sesquioxide, V2O3, is a prototypical metal-to-insulator system where, in temperature-dependent studies, the transition always coincides with a corundum-to-monoclinic structural transition. As a function of pressure, V2O3 follows the expected behavior of increased metallicity due to a larger bandwidth for pressures up to 12.5 GPa. Surprisingly, for higher pressures when the structure becomes unstable, the resistance starts to increase. Around 32.5 GPa at 300 K, we observe a novel pressure-induced corundum-to-monoclinic transition between two metallic phases, showing that the structural phase transition can be decoupled from the metal-insulator transition. Using X-ray Raman scattering, we find that screening effects, which are strong in the corundum phase, become weakened at high pressures. Theoretical calculations indicate that this can be related to a decrease in coherent quasiparticle strength, suggesting that the high-pressure phase is likely a critical correlated metal, on the verge of Mott-insulating behavior.

cond-mat.str-el↗