arXiv subjects

Guanjun Liu

Publications and source records attributed to Guanjun Liu.

22 records · Page 2Linked to original sources

arXiv2023

Robustness Testing for Multi-Agent Reinforcement Learning: State Perturbations on Critical Agents

Multi-Agent Reinforcement Learning (MARL) has been widely applied in many fields such as smart traffic and unmanned aerial vehicles. However, most MARL algorithms are vulnerable to adversarial perturbations on agent states. Robustness testing for a trained model is an essential step for confirming the trustworthiness of the model against unexpected perturbations. This work proposes a novel Robustness Testing framework for MARL that attacks states of Critical Agents (RTCA). The RTCA has two innovations: 1) a Differential Evolution (DE) based method to select critical agents as victims and to advise the worst-case joint actions on them; and 2) a team cooperation policy evaluation method employed as the objective function for the optimization of DE. Then, adversarial state perturbations of the critical agents are generated based on the worst-case joint actions. This is the first robustness testing framework with varying victim agents. RTCA demonstrates outstanding performance in terms of the number of victim agents and destroying cooperation policies.

cs.LG↗

arXiv2023

RoMFAC: A robust mean-field actor-critic reinforcement learning against adversarial perturbations on states

Multi-agent deep reinforcement learning makes optimal decisions dependent on system states observed by agents, but any uncertainty on the observations may mislead agents to take wrong actions. The Mean-Field Actor-Critic reinforcement learning (MFAC) is well-known in the multi-agent field since it can effectively handle a scalability problem. However, it is sensitive to state perturbations that can significantly degrade the team rewards. This work proposes a Robust Mean-field Actor-Critic reinforcement learning (RoMFAC) that has two innovations: 1) a new objective function of training actors, composed of a \emph{policy gradient function} that is related to the expected cumulative discount reward on sampled clean states and an \emph{action loss function} that represents the difference between actions taken on clean and adversarial states; and 2) a repetitive regularization of the action loss, ensuring the trained actors to obtain excellent performance. Furthermore, this work proposes a game model named a State-Adversarial Stochastic Game (SASG). Despite the Nash equilibrium of SASG may not exist, adversarial perturbations to states in the RoMFAC are proven to be defensible based on SASG. Experimental results show that RoMFAC is robust against adversarial perturbations while maintaining its competitive performance in environments without perturbations.

cs.LG↗

arXiv2023

Multi-Agent Reinforcement Learning: Methods, Applications, Visionary Prospects, and Challenges

Multi-agent reinforcement learning (MARL) is a widely used Artificial Intelligence (AI) technique. However, current studies and applications need to address its scalability, non-stationarity, and trustworthiness. This paper aims to review methods and applications and point out research trends and visionary prospects for the next decade. First, this paper summarizes the basic methods and application scenarios of MARL. Second, this paper outlines the corresponding research methods and their limitations on safety, robustness, generalization, and ethical constraints that need to be addressed in the practical applications of MARL. In particular, we believe that trustworthy MARL will become a hot research topic in the next decade. In addition, we suggest that considering human interaction is essential for the practical application of MARL in various societies. Therefore, this paper also analyzes the challenges while MARL is applied to human-machine interaction.

cs.AI↗

arXiv2020

Petri Net Based Symbolic Model Checking for Computation Tree Logic of Knowledge

Computation Tree Logic of Knowledge (CTLK) can specify many design requirements of privacy and security of multi-agent systems (MAS). In our conference paper, we defined Knowledge-oriented Petri Nets (KPN) to model MAS and proposed Reachability Graphs with Equivalence Relations (RGER) to verify CTLK. In this paper, we use the technique of Ordered Binary Decision Diagrams (OBDD) to encode RGER in order to alleviate the state explosion problem and enhance the verification efficiency. We propose a heuristic method to order those variables in OBDD, which can well improve the time and space performance of producing, encoding and exploring a huge state space. More importantly, our method does not produce and encode any transition or equivalence relation of states when producing and encoding an RGER, and in fact it dynamically produces those transition or equivalence relations that are required in the verification process of CTLK formulas. This policy can save a lot of time and space since the number of transition or equivalence relations of states is much greater than the number of states themselves. We design symbolic model checking algorithms, develop a tool and apply them to two famous examples: Alice-Bob Protocol and Dining Cryptographers Protocol. We compare our tool with MCMAS which is the state-of-the-art model checker of verifying CTLK. The experimental results illustrate the advantages of our model and method. Our tool running in a general PC can totally spend less than 14 hours to verify Dining Cryptographers Protocol with 1200 concurrent cryptographers where there are about $10^{1080}$ states and the two verified CTLK formulas have more than 6000 atomic propositions and more than 3600 operators. These good performances are owed to a combination of the OBDD technique and the structure characteristics of KPN.

cs.SE↗