arXiv subjects

Baoxiang Wang

Publications and source records attributed to Baoxiang Wang.

At least 55 records · Page 3Linked to original sources

arXiv2023

Learning Adversarial Low-rank Markov Decision Processes with Unknown Transition and Full-information Feedback

In this work, we study the low-rank MDPs with adversarially changed losses in the full-information feedback setting. In particular, the unknown transition probability kernel admits a low-rank matrix decomposition \citep{REPUCB22}, and the loss functions may change adversarially but are revealed to the learner at the end of each episode. We propose a policy optimization-based algorithm POLO, and we prove that it attains the $\widetilde{O}(K^{\frac{5}{6}}A^{\frac{1}{2}}d\ln(1+M)/(1-γ)^2)$ regret guarantee, where $d$ is rank of the transition kernel (and hence the dimension of the unknown representations), $A$ is the cardinality of the action space, $M$ is the cardinality of the model class, and $γ$ is the discounted factor. Notably, our algorithm is oracle-efficient and has a regret guarantee with no dependence on the size of potentially arbitrarily large state space. Furthermore, we also prove an $Ω(\frac{γ^2}{1-γ} \sqrt{d A K})$ regret lower bound for this problem, showing that low-rank MDPs are statistically more difficult to learn than linear MDPs in the regret minimization setting. To the best of our knowledge, we present the first algorithm that interleaves representation learning, exploration, and exploitation to achieve the sublinear regret guarantee for RL with nonlinear function approximation and adversarial losses.

cs.LG↗

arXiv2023

Information Design in Multi-Agent Reinforcement Learning

Reinforcement learning (RL) is inspired by the way human infants and animals learn from the environment. The setting is somewhat idealized because, in actual tasks, other agents in the environment have their own goals and behave adaptively to the ego agent. To thrive in those environments, the agent needs to influence other agents so their actions become more helpful and less harmful. Research in computational economics distills two ways to influence others directly: by providing tangible goods (mechanism design) and by providing information (information design). This work investigates information design problems for a group of RL agents. The main challenges are two-fold. One is the information provided will immediately affect the transition of the agent trajectories, which introduces additional non-stationarity. The other is the information can be ignored, so the sender must provide information that the receiver is willing to respect. We formulate the Markov signaling game, and develop the notions of signaling gradient and the extended obedience constraints that address these challenges. Our algorithm is efficient on various mixed-motive tasks and provides further insights into computational economics. Our code is publicly available at https://github.com/YueLin301/InformationDesignMARL.

cs.GT↗

arXiv2023

Semantically Aligned Task Decomposition in Multi-Agent Reinforcement Learning

The difficulty of appropriately assigning credit is particularly heightened in cooperative MARL with sparse reward, due to the concurrent time and structural scales involved. Automatic subgoal generation (ASG) has recently emerged as a viable MARL approach inspired by utilizing subgoals in intrinsically motivated reinforcement learning. However, end-to-end learning of complex task planning from sparse rewards without prior knowledge, undoubtedly requires massive training samples. Moreover, the diversity-promoting nature of existing ASG methods can lead to the "over-representation" of subgoals, generating numerous spurious subgoals of limited relevance to the actual task reward and thus decreasing the sample efficiency of the algorithm. To address this problem and inspired by the disentangled representation learning, we propose a novel "disentangled" decision-making method, Semantically Aligned task decomposition in MARL (SAMA), that prompts pretrained language models with chain-of-thought that can suggest potential goals, provide suitable goal decomposition and subgoal allocation as well as self-reflection-based replanning. Additionally, SAMA incorporates language-grounded RL to train each agent's subgoal-conditioned policy. SAMA demonstrates considerable advantages in sample efficiency compared to state-of-the-art ASG methods, as evidenced by its performance on two challenging sparse-reward tasks, Overcooked and MiniRTS.

cs.LG↗

arXiv2023

On Optimal Approximations for $k$-Submodular Maximization via Multilinear Extension

We investigate a more generalized form of submodular maximization, referred to as $k$-submodular maximization, with applications across social networks and machine learning domains. In this work, we propose the multilinear extension of $k$-submodular functions and unified Frank-Wolfe-type frameworks based on that. Our frameworks accomodate 1) monotone or non-monotone functions, and 2) various constraint types including matroid constraints, knapsack constraints, and their combinations. Notably, we attain an asymptotically optimal $1/2$-approximation for monotone $k$-submodular maximization problems with knapsack constraints, surpassing the previous $1/3$-approximation. The foundation for our analysis stems from new insights into specific linear and monotone properties pertaining to the multilinear extension.

cs.DS↗

arXiv2023

DPMAC: Differentially Private Communication for Cooperative Multi-Agent Reinforcement Learning

Communication lays the foundation for cooperation in human society and in multi-agent reinforcement learning (MARL). Humans also desire to maintain their privacy when communicating with others, yet such privacy concern has not been considered in existing works in MARL. To this end, we propose the \textit{differentially private multi-agent communication} (DPMAC) algorithm, which protects the sensitive information of individual agents by equipping each agent with a local message sender with rigorous $(ε, δ)$-differential privacy (DP) guarantee. In contrast to directly perturbing the messages with predefined DP noise as commonly done in privacy-preserving scenarios, we adopt a stochastic message sender for each agent respectively and incorporate the DP requirement into the sender, which automatically adjusts the learned message distribution to alleviate the instability caused by DP noise. Further, we prove the existence of a Nash equilibrium in cooperative MARL with privacy-preserving communication, which suggests that this problem is game-theoretically learnable. Extensive experiments demonstrate a clear advantage of DPMAC over baseline methods in privacy-preserving scenarios.

cs.LG↗

arXiv2023

Taming the Exponential Action Set: Sublinear Regret and Fast Convergence to Nash Equilibrium in Online Congestion Games

The congestion game is a powerful model that encompasses a range of engineering systems such as traffic networks and resource allocation. It describes the behavior of a group of agents who share a common set of $F$ facilities and take actions as subsets with $k$ facilities. In this work, we study the online formulation of congestion games, where agents participate in the game repeatedly and observe feedback with randomness. We propose CongestEXP, a decentralized algorithm that applies the classic exponential weights method. By maintaining weights on the facility level, the regret bound of CongestEXP avoids the exponential dependence on the size of possible facility sets, i.e., $\binom{F}{k} \approx F^k$, and scales only linearly with $F$. Specifically, we show that CongestEXP attains a regret upper bound of $O(kF\sqrt{T})$ for every individual player, where $T$ is the time horizon. On the other hand, exploiting the exponential growth of weights enables CongestEXP to achieve a fast convergence rate. If a strict Nash equilibrium exists, we show that CongestEXP can converge to the strict Nash policy almost exponentially fast in $O(F\exp(-t^{1-α}))$, where $t$ is the number of iterations and $α\in (1/2, 1)$.

cs.GT↗

arXiv2023

Online Influence Maximization under Decreasing Cascade Model

We study online influence maximization (OIM) under a new model of decreasing cascade (DC). This model is a generalization of the independent cascade (IC) model by considering the common phenomenon of market saturation. In DC, the chance of an influence attempt being successful reduces with previous failures. The effect is neglected by previous OIM works under IC and linear threshold models. We propose the DC-UCB algorithm to solve this problem, which achieves a regret bound of the same order as the state-of-the-art works on the IC model. Extensive experiments on both synthetic and real datasets show the effectiveness of our algorithm.

cs.SI↗

arXiv2023

Global Cauchy problem for the NLKG in super-critical spaces

By introducing a class of new function spaces $B^{σ,s}_{p,q}$ as the resolution spaces, we study the Cauchy problem for the nonlinear Klein-Gordon equation (NLKG) in all spatial dimensions $d \geqslant 1$, $$ \partial^2_t u + u- Δu + u^{1+α} =0, \ \ (u, \partial_t u)|_{t=0} = (u_0,u_1). $$ We consider the initial data $(u_0,u_1)$ in super-critical function spaces $E^{σ,s} \times E^{σ-1,s}$ for which their norms are defined by $$ \|f\|_{E^{σ,s}} = \|\langleξ\rangle^σ2^{s|ξ|}\widehat{f}(ξ)\|_{L^2}, \ s<0, \ σ\in \mathbb{R}. $$ Any Sobolev space $H^κ$ can be embedded into $ E^{σ,s}$, i.e., $H^κ\subset E^{σ,s}$ for any $ κ,σ\in \mathbb{R}$ and $s<0$. We show the global existence and uniqueness of the solutions of NLKG if the initial data belong to some $E^{σ,s} \times E^{ σ-1,s}$ ($s<0, \ σ\geqslant \max (d/2-2/α, \, 1/2), \ α\in \mathbb{N}, \ α\geqslant 4/d$) and their Fourier transforms are supported in the first octant, the smallness conditions on the initial data in $E^{σ,s} \times E^{σ-1,s}$ are not required for the global solutions. Similar results hold for the sinh-Gordon equation $\partial^2_t u - Δu + \sinh u=0$ if the spatial dimensions $d \geqslant 2$.

math.AP↗

arXiv2023

Learning from Good Trajectories in Offline Multi-Agent Reinforcement Learning

Offline multi-agent reinforcement learning (MARL) aims to learn effective multi-agent policies from pre-collected datasets, which is an important step toward the deployment of multi-agent systems in real-world applications. However, in practice, each individual behavior policy that generates multi-agent joint trajectories usually has a different level of how well it performs. e.g., an agent is a random policy while other agents are medium policies. In the cooperative game with global reward, one agent learned by existing offline MARL often inherits this random policy, jeopardizing the performance of the entire team. In this paper, we investigate offline MARL with explicit consideration on the diversity of agent-wise trajectories and propose a novel framework called Shared Individual Trajectories (SIT) to address this problem. Specifically, an attention-based reward decomposition network assigns the credit to each agent through a differentiable key-value memory mechanism in an offline manner. These decomposed credits are then used to reconstruct the joint offline datasets into prioritized experience replay with individual trajectories, thereafter agents can share their good trajectories and conservatively train their policies with a graph attention network (GAT) based critic. We evaluate our method in both discrete control (i.e., StarCraft II and multi-agent particle environment) and continuous control (i.e, multi-agent mujoco). The results indicate that our method achieves significantly better results in complex and mixed offline multi-agent datasets, especially when the difference of data quality between individual trajectories is large.

cs.LG↗

arXiv2023

Diverse Policy Optimization for Structured Action Space

Enhancing the diversity of policies is beneficial for robustness, exploration, and transfer in reinforcement learning (RL). In this paper, we aim to seek diverse policies in an under-explored setting, namely RL tasks with structured action spaces with the two properties of composability and local dependencies. The complex action structure, non-uniform reward landscape, and subtle hyperparameter tuning due to the properties of structured actions prevent existing approaches from scaling well. We propose a simple and effective RL method, Diverse Policy Optimization (DPO), to model the policies in structured action space as the energy-based models (EBM) by following the probabilistic RL framework. A recently proposed novel and powerful generative model, GFlowNet, is introduced as the efficient, diverse EBM-based policy sampler. DPO follows a joint optimization framework: the outer layer uses the diverse policies sampled by the GFlowNet to update the EBM-based policies, which supports the GFlowNet training in the inner layer. Experiments on ATSC and Battle benchmarks demonstrate that DPO can efficiently discover surprisingly diverse policies in challenging scenarios and substantially outperform existing state-of-the-art methods.

cs.LG↗

arXiv2023

Constraining the EdGB theory with higher harmonics and merger-ringdown contribution using GWTC-3

In this paper, we revisit the problem of using gravitational wave data to test the Einstein-dilation-Gauss-Bonnet theory, by using nine selected gravitational wave events from GWTC-3. Compared with existing work, we are taking into account the higher harmonics more properly and we also study the contribution of the merger-ringdown data. Using the inspiral data alone, we find that the best result is from GW200115, giving $\sqrt{|α|} < 1.1$ km, which is about 17\% tighter than the previous best result. We also notice the possible existence of a simple unexpected relation among the constraints from different events. Several combinations of the selected events give $\sqrt{|α|} \leq 1.0$ km. The result is further improved when the merger-ringdown data is also included, using two phenomenological schemes, giving $\sqrt{|α|} < 0.87$ km for GW200115 in the best case scenario.

gr-qc↗

arXiv2023

Improved Regret Bounds for Linear Adversarial MDPs via Linear Optimization

Learning Markov decision processes (MDP) in an adversarial environment has been a challenging problem. The problem becomes even more challenging with function approximation, since the underlying structure of the loss function and transition kernel are especially hard to estimate in a varying environment. In fact, the state-of-the-art results for linear adversarial MDP achieve a regret of $\tilde{O}(K^{6/7})$ ($K$ denotes the number of episodes), which admits a large room for improvement. In this paper, we investigate the problem with a new view, which reduces linear MDP into linear optimization by subtly setting the feature maps of the bandit arms of linear optimization. This new technique, under an exploratory assumption, yields an improved bound of $\tilde{O}(K^{4/5})$ for linear adversarial MDP without access to a transition simulator. The new view could be of independent interest for solving other MDP problems that possess a linear structure.

cs.LG↗

arXiv2022

CGIBNet: Bandwidth-constrained Communication with Graph Information Bottleneck in Multi-Agent Reinforcement Learning

Communication is one of the core components for cooperative multi-agent reinforcement learning (MARL). The communication bandwidth, in many real applications, is always subject to certain constraints. To improve communication efficiency, in this article, we propose to simultaneously optimize whom to communicate with and what to communicate for each agent in MARL. By initiating the communication between agents with a directed complete graph, we propose a novel communication model, named Communicative Graph Information Bottleneck Network (CGIBNet), to simultaneously compress the graph structure and the node information with the graph information bottleneck principle. The graph structure compression is designed to cut the redundant edges for determining whom to communicate with. The node information compression aims to address the problem of what to communicate via learning compact node representations. Moreover, CGIBNet is the first universal module for bandwidth-constrained communication, which can be applied to various training frameworks (i.e., policy-based and value-based MARL frameworks) and communication modes (i.e., single-round and multi-round communication). Extensive experiments are conducted in Traffic Control and StarCraft II environments. The results indicate that our method can achieve better performance in bandwidth-constrained settings compared with state-of-the-art algorithms.

cs.AI↗

arXiv2022

Algorithms and Theory for Supervised Gradual Domain Adaptation

The phenomenon of data distribution evolving over time has been observed in a range of applications, calling the needs of adaptive learning algorithms. We thus study the problem of supervised gradual domain adaptation, where labeled data from shifting distributions are available to the learner along the trajectory, and we aim to learn a classifier on a target data distribution of interest. Under this setting, we provide the first generalization upper bound on the learning error under mild assumptions. Our results are algorithm agnostic, general for a range of loss functions, and only depend linearly on the averaged learning error across the trajectory. This shows significant improvement compared to the previous upper bound for unsupervised gradual domain adaptation, where the learning error on the target domain depends exponentially on the initial error on the source domain. Compared with the offline setting of learning from multiple domains, our results also suggest the potential benefits of the temporal structure among different domains in adapting to the target one. Empirically, our theoretical results imply that learning proper representations across the domains will effectively mitigate the learning errors. Motivated by these theoretical insights, we propose a min-max learning objective to learn the representation and classifier simultaneously. Experimental results on both semi-synthetic and large-scale real datasets corroborate our findings and demonstrate the effectiveness of our objectives.

cs.LG↗

arXiv2022

Online Policy Optimization for Robust MDP

Reinforcement learning (RL) has exceeded human performance in many synthetic settings such as video games and Go. However, real-world deployment of end-to-end RL models is less common, as RL models can be very sensitive to slight perturbation of the environment. The robust Markov decision process (MDP) framework -- in which the transition probabilities belong to an uncertainty set around a nominal model -- provides one way to develop robust models. While previous analysis shows RL algorithms are effective assuming access to a generative model, it remains unclear whether RL can be efficient under a more realistic online setting, which requires a careful balance between exploration and exploitation. In this work, we consider online robust MDP by interacting with an unknown nominal system. We propose a robust optimistic policy optimization algorithm that is provably efficient. To address the additional uncertainty caused by an adversarial environment, our model features a new optimistic update rule derived via Fenchel conjugates. Our analysis establishes the first regret bound for online robust MDPs.

cs.LG↗

arXiv2022

Global Cauchy problems for the nonlocal (derivative) NLS in $E^s_σ$

We consider the Cauchy problem for the (derivative) nonlocal NLS in super-critical function spaces $E^s_σ$ for which the norms are defined by $$ \|f\|_{E^s_σ} = \|\langleξ\rangle^σ2^{s|ξ|}\hat{f}(ξ)\|_{L^2}, \ s<0, \ σ\in \mathbb{R}. $$ Any Sobolev space $H^{r}$ is a subspace of $E^s_σ$, i.e., $H^r \subset E^s_σ$ for any $ r,σ\in \mathbb{R}$ and $s<0$. Let $s<0$ and $σ>-1/2$ ($σ>0$) for the nonlocal NLS (for the nonlocal derivative NLS). We show the global existence and uniqueness of the solutions if the initial data belong to $E^s_σ$ and their Fourier transforms are supported in $(0, \infty)$, the smallness conditions on the initial data in $E^s_σ$ are not required for the global solutions.

math.AP↗

arXiv2022

Complex valued semi-linear heat equations in super-critical spaces $E^s_σ$

We consider the Cauchy problem for the complex valued semi-linear heat equation $$ \partial_t u - Δu - u^m =0, \ \ u (0,x) = u_0(x), $$ where $m\geq 2$ is an integer and the initial data belong to super-critical spaces $E^s_σ$ for which the norms are defined by $$ \|f\|_{E^s_σ} = \|\langle ξ\rangle^σ2^{s|ξ|}\hat{f}(ξ)\|_{L^2}, \ \ σ\in \mathbb{R}, \ s<0. $$ If $s<0$, then any Sobolev space $H^{r}$ is a subspace of $E^s_σ$, i.e., $\cup_{r \in \mathbb{R}} H^r \subset E^s_σ$. We obtain the global existence and uniqueness of the solutions if the initial data belong to $E^s_σ$ ($s<0, \ σ\geq d/2-2/(m-1)$) and their Fourier transforms are supported in the first octant, the smallness conditions on the initial data in $E^s_σ$ are not required for the global solutions. Moreover, we show that the error between the solution $u$ and the iteration solution $u^{(j)}$ is $C^j/(j\,!)^2$. Similar results also hold if the nonlinearity $u^m$ is replaced by an exponential function $e^u-1$.

math.AP↗

arXiv2022

Edge Rewiring Goes Neural: Boosting Network Resilience without Rich Features

Improving the resilience of a network is a fundamental problem in network science, which protects the underlying system from natural disasters and malicious attacks. This is traditionally achieved via successive degree-preserving edge rewiring operations, with the major limitation of being transductive. Inductively solving graph-related tasks with sequential actions is accomplished by adopting graph neural networks (GNNs) coupled with reinforcement learning under the scenario with rich graph features. However, such frameworks cannot be directly applied to resilience tasks where only pure topological structure is available. In this case, GNNs can barely learn useful information, resulting in prohibitive difficulty in making actions for successively rewiring edges under a reinforcement learning context. In this paper, we study in depth the reasons why typical GNNs cause such failure. Based on this investigation, we propose ResiNet, the first end-to-end trainable inductive framework to discover resilient network topologies while balancing network utility. To this end, we reformulate resilience optimization as an MDP equipped with edge rewiring action space, and propose a pure topology-oriented variant of GNN called filtration enhanced graph neural network (FireGNN), which can learn from graphs without rich features. Extensive experiments demonstrate that ResiNet achieves a near-optimal resilience gain on various graphs while balancing the utility, and outperforms existing approaches by a large margin.

cs.LG↗