arXiv subjects

Baoxiang Wang

Publications and source records attributed to Baoxiang Wang.

At least 73 records · Page 4Linked to original sources

arXiv2022

Provably Efficient Convergence of Primal-Dual Actor-Critic with Nonlinear Function Approximation

We study the convergence of the actor-critic algorithm with nonlinear function approximation under a nonconvex-nonconcave primal-dual formulation. Stochastic gradient descent ascent is applied with an adaptive proximal term for robust learning rates. We show the first efficient convergence result with primal-dual actor-critic with a convergence rate of $\mathcal{O}\left(\sqrt{\frac{\ln \left(N d G^2 \right)}{N}}\right)$ under Markovian sampling, where $G$ is the element-wise maximum of the gradient, $N$ is the number of iterations, and $d$ is the dimension of the gradient. Our result is presented with only the Polyak-Łojasiewicz condition for the dual variables, which is easy to verify and applicable to a wide range of reinforcement learning (RL) scenarios. The algorithm and analysis are general enough to be applied to other RL settings, like multi-agent RL. Empirical results on OpenAI Gym continuous control tasks corroborate our theoretical findings.

cs.LG↗

arXiv2022

Differentially Private Temporal Difference Learning with Stochastic Nonconvex-Strongly-Concave Optimization

Temporal difference (TD) learning is a widely used method to evaluate policies in reinforcement learning. While many TD learning methods have been developed in recent years, little attention has been paid to preserving privacy and most of the existing approaches might face the concerns of data privacy from users. To enable complex representative abilities of policies, in this paper, we consider preserving privacy in TD learning with nonlinear value function approximation. This is challenging because such a nonlinear problem is usually studied in the formulation of stochastic nonconvex-strongly-concave optimization to gain finite-sample analysis, which would require simultaneously preserving the privacy on primal and dual sides. To this end, we employ a momentum-based stochastic gradient descent ascent to achieve a single-timescale algorithm, and achieve a good trade-off between meaningful privacy and utility guarantees of both the primal and dual sides by perturbing the gradients on both sides using well-calibrated Gaussian noises. As a result, our DPTD algorithm could provide $(ε,δ)$-differential privacy (DP) guarantee for the sensitive information encoded in transitions and retain the original power of TD learning, with the utility upper bounded by $\widetilde{\mathcal{O}}(\frac{(d\log(1/δ))^{1/8}}{(nε)^{1/4}})$ (The tilde in this paper hides the log factor.), where $n$ is the trajectory length and $d$ is the dimension. Extensive experiments conducted in OpenAI Gym show the advantages of our proposed algorithm.

cs.LG↗

arXiv2022

Shapley Counterfactual Credits for Multi-Agent Reinforcement Learning

Centralized Training with Decentralized Execution (CTDE) has been a popular paradigm in cooperative Multi-Agent Reinforcement Learning (MARL) settings and is widely used in many real applications. One of the major challenges in the training process is credit assignment, which aims to deduce the contributions of each agent according to the global rewards. Existing credit assignment methods focus on either decomposing the joint value function into individual value functions or measuring the impact of local observations and actions on the global value function. These approaches lack a thorough consideration of the complicated interactions among multiple agents, leading to an unsuitable assignment of credit and subsequently mediocre results on MARL. We propose Shapley Counterfactual Credit Assignment, a novel method for explicit credit assignment which accounts for the coalition of agents. Specifically, Shapley Value and its desired properties are leveraged in deep MARL to credit any combinations of agents, which grants us the capability to estimate the individual credit for each agent. Despite this capability, the main technical difficulty lies in the computational complexity of Shapley Value who grows factorially as the number of agents. We instead utilize an approximation method via Monte Carlo sampling, which reduces the sample complexity while maintaining its effectiveness. We evaluate our method on StarCraft II benchmarks across different scenarios. Our method outperforms existing cooperative MARL algorithms significantly and achieves the state-of-the-art, with especially large margins on tasks with more severe difficulties.

cs.AI↗

arXiv2021

Combinatorial Bandits under Strategic Manipulations

Strategic behavior against sequential learning methods, such as "click framing" in real recommendation systems, have been widely observed. Motivated by such behavior we study the problem of combinatorial multi-armed bandits (CMAB) under strategic manipulations of rewards, where each arm can modify the emitted reward signals for its own interest. This characterization of the adversarial behavior is a relaxation of previously well-studied settings such as adversarial attacks and adversarial corruption. We propose a strategic variant of the combinatorial UCB algorithm, which has a regret of at most $O(m\log T + m B_{max})$ under strategic manipulations, where $T$ is the time horizon, $m$ is the number of arms, and $B_{max}$ is the maximum budget of an arm. We provide lower bounds on the budget for arms to incur certain regret of the bandit algorithm. Extensive experiments on online worker selection for crowdsourcing systems, online influence maximization and online recommendations with both synthetic and real datasets corroborate our theoretical findings on robustness and regret bounds, in a variety of regimes of manipulation budgets.

cs.LG↗

arXiv2021

Incentivizing an Unknown Crowd

Motivated by the common strategic activities in crowdsourcing labeling, we study the problem of sequential eliciting information without verification (EIWV) for workers with a heterogeneous and unknown crowd. We propose a reinforcement learning-based approach that is effective against a wide range of settings including potential irrationality and collusion among workers. With the aid of a costly oracle and the inference method, our approach dynamically decides the oracle calls and gains robustness even under the presence of frequent collusion activities. Extensive experiments show the advantage of our approach. Our results also present the first comprehensive experiments of EIWV on large-scale real datasets and the first thorough study of the effects of environmental variables.

cs.LG↗

arXiv2021

Ill-posedness for the Navier-Stokes equations in critical Besov spaces $\dot B^{-1}_{\infty,q}$

We study the Cauchy problem for the incompressible Navier-Stokes equation \begin{align} u_t -Δu+u\cdot \nabla u +\nabla p=0, \ \ {\rm div} u=0, \ \ u(0,x)= δu_0. \label{NS} \end{align} For arbitrarily small $δ>0$, we show that the solution map $δu_0 \to u$ in critical Besov spaces $\dot B^{-1}_{\infty,q}$ ($\forall \ q\in [1,2]$) is discontinuous at origin. It is known that the Navier-Stokes equation is globally well-posed for small data in $BMO^{-1}$. Taking notice of the embedding $\dot B^{-1}_{\infty,q} \subset BMO^{-1}$ ($q\le 2$), we see that for sufficiently small $δ>0$, $u_0\in \dot B^{-1}_{\infty,q} $ ($q\le 2$) can guarantee that the Navier-Stokes equation has a unique global solution in $BMO^{-1}$, however, this solution is instable in $ \dot B^{-1}_{\infty,q} $ and the solution can have an inflation in $\dot B^{-1}_{\infty,q}$ for certain initial data. So, our result indicates that two different topological structures in the same space may determine the well and ill posedness, respectively.

math.AP↗

arXiv2021

Cascading Bandit under Differential Privacy

This paper studies \emph{differential privacy (DP)} and \emph{local differential privacy (LDP)} in cascading bandits. Under DP, we propose an algorithm which guarantees $ε$-indistinguishability and a regret of $\mathcal{O}((\frac{\log T}ε)^{1+ξ})$ for an arbitrarily small $ξ$. This is a significant improvement from the previous work of $\mathcal{O}(\frac{\log^3 T}ε)$ regret. Under ($ε$,$δ$)-LDP, we relax the $K^2$ dependence through the tradeoff between privacy budget $ε$ and error probability $δ$, and obtain a regret of $\mathcal{O}(\frac{K\log (1/δ) \log T}{ε^2})$, where $K$ is the size of the arm subset. This result holds for both Gaussian mechanism and Laplace mechanism by analyses on the composition. Our results extend to combinatorial semi-bandit. We show respective lower bounds for DP and LDP cascading bandits. Extensive experiments corroborate our theoretic findings.

cs.LG↗

arXiv2020

Almost sure scattering for the nonlinear Klein-Gordon equations with Sobolev critical power

In this paper, we study the almost sure scattering for the Klein-Gordon equations with Sobolev critical power. We obtain the almost sure scattering with random initial data in $H^s \times H^{s-1}$; $11/12 < s < 1$ for $d = 4$, $15/16 < s < 1$ for $d = 5$. We use the induction on scales and bushes argument in [9] where the model equation is wave equation. For d = 5, we use the mass term of the Klein-Gordon equation to obtain the control of the increment of energy in the process of induction on scales.

math.AP↗

arXiv2020

The Gambler's Problem and Beyond

We analyze the Gambler's problem, a simple reinforcement learning problem where the gambler has the chance to double or lose the bets until the target is reached. This is an early example introduced in the reinforcement learning textbook by Sutton and Barto (2018), where they mention an interesting pattern of the optimal value function with high-frequency components and repeating non-smooth points. It is however without further investigation. We provide the exact formula for the optimal value function for both the discrete and the continuous cases. Though simple as it might seem, the value function is pathological: fractal, self-similar, derivative taking either zero or infinity, and not written as elementary functions. It is in fact one of the generalized Cantor functions, where it holds a complexity that has been uncharted thus far. Our analyses could provide insights into improving value function approximation, gradient-based algorithms, and Q-learning, in real applications and implementations.

stat.ML↗

arXiv2020

Scaling limit of Modulation Spaces and Their Applications

Modulation spaces $M^s_{p,q}$ were introduced by Feichtinger \cite{Fei83} in 1983. By resorting to the wavelet basis, Bényi and Oh \cite{BeOh20} defined a modified version to Feichtinger's modulation spaces for which the symmetry scalings are emphasized for its possible applications in PDE. By carefully investigating the scaling properties of modulation spaces and their connections with the wavelet basis, we will introduce a class of generalized modulation spaces, which contain both Feichtinger's and Bényi and Oh's modulation spaces. As their applications, we will give a local well-posedness and a (small data) global well-posedness results for NLS in some rougher generalized modulation spaces, which generalize the well posedness results of \cite{BeOk09} and \cite{WaHud07}, and certain super-critical initial data in $H^s$ or in $L^p$ are involved in these spaces.

math.FA↗

arXiv2020

Learning and Testing Variable Partitions

$ $Let $F$ be a multivariate function from a product set $Σ^n$ to an Abelian group $G$. A $k$-partition of $F$ with cost $δ$ is a partition of the set of variables $\mathbf{V}$ into $k$ non-empty subsets $(\mathbf{X}_1, \dots, \mathbf{X}_k)$ such that $F(\mathbf{V})$ is $δ$-close to $F_1(\mathbf{X}_1)+\dots+F_k(\mathbf{X}_k)$ for some $F_1, \dots, F_k$ with respect to a given error metric. We study algorithms for agnostically learning $k$ partitions and testing $k$-partitionability over various groups and error metrics given query access to $F$. In particular we show that $1.$ Given a function that has a $k$-partition of cost $δ$, a partition of cost $\mathcal{O}(k n^2)(δ+ ε)$ can be learned in time $\tilde{\mathcal{O}}(n^2 \mathrm{poly} (1/ε))$ for any $ε> 0$. In contrast, for $k = 2$ and $n = 3$ learning a partition of cost $δ+ ε$ is NP-hard. $2.$ When $F$ is real-valued and the error metric is the 2-norm, a 2-partition of cost $\sqrt{δ^2 + ε}$ can be learned in time $\tilde{\mathcal{O}}(n^5/ε^2)$. $3.$ When $F$ is $\mathbb{Z}_q$-valued and the error metric is Hamming weight, $k$-partitionability is testable with one-sided error and $\mathcal{O}(kn^3/ε)$ non-adaptive queries. We also show that even two-sided testers require $Ω(n)$ queries when $k = 2$. This work was motivated by reinforcement learning control tasks in which the set of control variables can be partitioned. The partitioning reduces the task into multiple lower-dimensional ones that are relatively easier to learn. Our second algorithm empirically increases the scores attained over previous heuristic partitioning methods applied in this context.

cs.LG↗

arXiv2020

Resonant Decompositions and Global Well-posedness for 2D Zakharov-Kuznetsov Equation in Sobolev spaces of Negative Indices

The Cauchy problem for Zakharov-Kuznetsov equation on $\mathbb{R}^2$ is shown to be global well-posed for the initial date in $H^{s}$ provided $s>-\frac{1}{13}$. As conservation laws are invalid in Sobolev spaces below $L^2$, we construct an almost conserved quantity using multilinear correction term following the $I$-method introduced by Colliander, Keel, Staffilani, Takaoka and Tao. In contrast to KdV equation, the main difficulty is to handle the resonant interactions which are significant due to the multidimensional and multilinear setting of the problem. The proof relies upon the bilinear Strichartz estimate and the nonlinear Loomis-Whitney inequality.

math.AP↗

arXiv2019

Privacy-preserving Q-Learning with Functional Noise in Continuous State Spaces

We consider differentially private algorithms for reinforcement learning in continuous spaces, such that neighboring reward functions are indistinguishable. This protects the reward information from being exploited by methods such as inverse reinforcement learning. Existing studies that guarantee differential privacy are not extendable to infinite state spaces, as the noise level to ensure privacy will scale accordingly to infinity. Our aim is to protect the value function approximator, without regard to the number of states queried to the function. It is achieved by adding functional noise to the value function iteratively in the training. We show rigorous privacy guarantees by a series of analyses on the kernel of the noise space, the probabilistic bound of such noise samples, and the composition over the iterations. We gain insight into the utility analysis by proving the algorithm's approximate optimality when the state space is discrete. Experiments corroborate our theoretical findings and show improvement over existing approaches.

stat.ML↗

arXiv2019

Recurrent Existence Determination Through Policy Optimization

Binary determination of the presence of objects is one of the problems where humans perform extraordinarily better than computer vision systems, in terms of both speed and preciseness. One of the possible reasons is that humans can skip most of the clutter and attend only on salient regions. Recurrent attention models (RAM) are the first computational models to imitate the way humans process images via the REINFORCE algorithm. Despite that RAM is originally designed for image recognition, we extend it and present recurrent existence determination, an attention-based mechanism to solve the existence determination. Our algorithm employs a novel $k$-maximum aggregation layer and a new reward mechanism to address the issue of delayed rewards, which would have caused the instability of the training process. The experimental analysis demonstrates significant efficiency and accuracy improvement over existing approaches, on both synthetic and real-world datasets.

cs.LG↗

arXiv2019

Policy Optimization with Second-Order Advantage Information

Policy optimization on high-dimensional continuous control tasks exhibits its difficulty caused by the large variance of the policy gradient estimators. We present the action subspace dependent gradient (ASDG) estimator which incorporates the Rao-Blackwell theorem (RB) and Control Variates (CV) into a unified framework to reduce the variance. To invoke RB, our proposed algorithm (POSA) learns the underlying factorization structure among the action space based on the second-order advantage information. POSA captures the quadratic information explicitly and efficiently by utilizing the wide & deep architecture. Empirical studies show that our proposed approach demonstrates the performance improvements on high-dimensional synthetic settings and OpenAI Gym's MuJoCo continuous control tasks.

cs.LG↗

arXiv2019

Metatrace Actor-Critic: Online Step-size Tuning by Meta-gradient Descent for Reinforcement Learning Control

Reinforcement learning (RL) has had many successes in both "deep" and "shallow" settings. In both cases, significant hyperparameter tuning is often required to achieve good performance. Furthermore, when nonlinear function approximation is used, non-stationarity in the state representation can lead to learning instability. A variety of techniques exist to combat this --- most notably large experience replay buffers or the use of multiple parallel actors. These techniques come at the cost of moving away from the online RL problem as it is traditionally formulated (i.e., a single agent learning online without maintaining a large database of training examples). Meta-learning can potentially help with both these issues by tuning hyperparameters online and allowing the algorithm to more robustly adjust to non-stationarity in a problem. This paper applies meta-gradient descent to derive a set of step-size tuning algorithms specifically for online RL control with eligibility traces. Our novel technique, Metatrace, makes use of an eligibility trace analogous to methods like $TD(λ)$. We explore tuning both a single scalar step-size and a separate step-size for each learned parameter. We evaluate Metatrace first for control with linear function approximation in the classic mountain car problem and then in a noisy, non-stationary version. Finally, we apply Metatrace for control with nonlinear function approximation in 5 games in the Arcade Learning Environment where we explore how it impacts learning speed and robustness to initial step-size choice. Results show that the meta-step-size parameter of Metatrace is easy to set, Metatrace can speed learning, and Metatrace can allow an RL algorithm to deal with non-stationarity in the learning task.

cs.LG↗

arXiv2019

Navier-Stokes Equation in Super-Critical Spaces $E^s_{p,q}$

In this paper we develop a new way to study the global existence and uniqueness for the Navier-Stokes equation (NS) and consider the initial data in a class of modulation spaces $E^s_{p,q}$ with exponentially decaying weights $(s<0, \ 1<p,q<\infty)$ for which the norms are defined by $$ \|f\|_{E^s_{p,q}} = \left(\sum_{k\in \mathbb{Z}^d} 2^{s|k|q}\|\mathscr{F}^{-1} χ_{k+[0,1]^d}\mathscr{F} f\|^q_p \right)^{1/q}. $$ The space $E^s_{p,q}$ is a rather rough function space and cannot be treated as a subspace of tempered distributions. For example, we have the embedding $H^σ\subset E^s_{2,1}$ for all $σ<0$ and $s<0$. It is known that $H^σ$ ($σ<d/2-1$) is a super-critical space of NS, it follows that $ E^s_{2,1}$ ($s<0$) is also super-critical for NS. We show that NS has a unique global mild solution if the initial data belong to $E^s_{2,1}$ ($s<0$) and their Fourier transforms are supported in $ \mathbb{R}^d_I:= \{ξ\in \mathbb{R}^d: \ ξ_i \geq 0, \, i=1,...,d\}$. Similar results hold for the initial data in $E^s_{r,1}$ with $2< r \leq d$. Our results imply that NS has a unique global solution if the initial value $u_0$ is in $L^2$ with ${\rm supp} \, \widehat{u}_0 \, \subset \mathbb{R}^d_I$.

math.AP↗

arXiv2019

Global Well-Posedness for NLS with a Class of $H^s$-Supercritical Data

We study the Cauchy problem for NLS with a class of $H^s$-super-critical data \begin{align} & {\rm i}u_t +Δu+ λ|u|^{2κ} u =0, \quad u(0)=u_0 \label{NLSabstract} \end{align} and show that \eqref{NLSabstract} is globally well-posed and scattering in $α$-modulation spaces $M^{s,α}_{2,1}$ ($α\in [0,1), \ s> dα/2-α/κ$, $κ\in \mathbb{N}$ and $κ\geq 2/d$) for the sufficiently small data. Moreover, NLS is ill-posed in $M^{s,α}_{2,1}$ if $s< dα/2-α/κ$. In particular, we obtain a class of initial data $u_0$ satisfying for any $M\gg 1$, \begin{align} \|u_0\|_2 \sim M^{1/κ-d/2 }, \ \ \|u_0\|_\infty \ =\infty , \ \ \|u_0\|_{M^{s,α}_{2,1}} \geq M^{(1-α)/κ}, \ \ \ \|u_0\|_{B^{s(κ)}_{2,\infty}} =\infty \nonumber \end{align} such that NLS is globally well-posed in $M^{s,α}_{2,1}$ if $κ>2/d, \ α\in [0,1)\ dα/2-α/κ<s < s(κ):= d/2-1/κ$. Such a kind of data are super-critical in $H^{s(κ)}$ and have infinite amplitude.

math.AP↗