arXiv subjects

Yujie Tang

Publications and source records attributed to Yujie Tang.

At least 37 records · Page 2Linked to original sources

arXiv2023

Benign Nonconvex Landscapes in Optimal and Robust Control, Part I: Global Optimality

Direct policy search has achieved great empirical success in reinforcement learning. Many recent studies have revisited its theoretical foundation for continuous control, which reveals elegant nonconvex geometry in various benchmark problems, especially in fully observable state-feedback cases. This paper considers two fundamental optimal and robust control problems with partial observability: the Linear Quadratic Gaussian (LQG) control with stochastic noises, and $\mathcal{H}_\infty$ robust control with adversarial noises. In the policy space, the former problem is smooth but nonconvex, while the latter one is nonsmooth and nonconvex. We highlight some interesting and surprising ``discontinuity'' of LQG and $\mathcal{H}_\infty$ cost functions around the boundary of their domains. Despite the lack of convexity (and possibly smoothness), our main results show that for a class of non-degenerate policies, all Clarke stationary points are globally optimal and there is no spurious local minimum for both LQG and $\mathcal{H}_\infty$ control. Our proof techniques rely on a new and unified framework of Extended Convex Lifting (ECL), which reconciles the gap between nonconvex policy optimization and convex reformulations. This ECL framework is of independent interest, and we will discuss its details in Part II of this paper.

math.OC↗

arXiv2023

Zeroth-Order Feedback-Based Optimization for Distributed Demand Response

Distributed demand response is a typical distributed optimization problem that requires coordination among multiple agents to satisfy demand response requirements. However, existing distributed algorithms for this problem still face challenges such as unknown system models, nonconvexity, privacy issues, etc. To address these challenges, we propose and analyze two distributed algorithms, in which the agents do not share their information and instead perform local updates using zeroth-order feedback information to estimate the gradient of the global objective function. One algorithm applies to problems with general convex and compact feasible sets but has higher oracle complexity bounded by $O(d/ε^2)$, while the other algorithm achieves lower complexity bound $O(d/ε)$ but is only applicable to problems with box constraints. We conduct empirical experiments to validate their performance.

eess.SY↗

arXiv2023

Unwieldy Object Delivery with Nonholonomic Mobile Base: A Stable Pushing Approach

This paper addresses the problem of pushing manipulation with nonholonomic mobile robots. Pushing is a fundamental skill that enables robots to move unwieldy objects that cannot be grasped. We propose a stable pushing method that maintains stiff contact between the robot and the object to avoid consuming repositioning actions. We prove that a line contact, rather than a single point contact, is necessary for nonholonomic robots to achieve stable pushing. We also show that the stable pushing constraint and the nonholonomic constraint of the robot can be simplified as a concise linear motion constraint. Then the pushing planning problem can be formulated as a constrained optimization problem using nonlinear model predictive control (NMPC). According to the experiments, our NMPC-based planner outperforms a reactive pushing strategy in terms of efficiency, reducing the robot's traveled distance by 23.8\% and time by 77.4\%. Furthermore, our method requires four fewer hyperparameters and decision variables than the Linear Time-Varying (LTV) MPC approach, making it easier to implement. Real-world experiments are carried out to validate the proposed method with two differential-drive robots, Husky and Boxer, under different friction conditions.

cs.RO↗

arXiv2023

Distributed Information-based Source Seeking

In this paper, we design an information-based multi-robot source seeking algorithm where a group of mobile sensors localizes and moves close to a single source using only local range-based measurements. In the algorithm, the mobile sensors perform source identification/localization to estimate the source location; meanwhile, they move to new locations to maximize the Fisher information about the source contained in the sensor measurements. In doing so, they improve the source location estimate and move closer to the source. Our algorithm is superior in convergence speed compared with traditional field climbing algorithms, is flexible in the measurement model and the choice of information metric, and is robust to measurement model errors. Moreover, we provide a fully distributed version of our algorithm, where each sensor decides its own actions and only shares information with its neighbors through a sparse communication network. We perform intensive simulation experiments to test our algorithms on large-scale systems and physical experiments on small ground vehicles with light sensors, demonstrating success in seeking a light source.

cs.RO↗

arXiv2023

Escaping saddle points in zeroth-order optimization: the power of two-point estimators

Two-point zeroth order methods are important in many applications of zeroth-order optimization, such as robotics, wind farms, power systems, online optimization, and adversarial robustness to black-box attacks in deep neural networks, where the problem may be high-dimensional and/or time-varying. Most problems in these applications are nonconvex and contain saddle points. While existing works have shown that zeroth-order methods utilizing $Ω(d)$ function valuations per iteration (with $d$ denoting the problem dimension) can escape saddle points efficiently, it remains an open question if zeroth-order methods based on two-point estimators can escape saddle points. In this paper, we show that by adding an appropriate isotropic perturbation at each iteration, a zeroth-order algorithm based on $2m$ (for any $1 \leq m \leq d$) function evaluations per iteration can not only find $ε$-second order stationary points polynomially fast, but do so using only $\tilde{O}\left(\frac{d}{mε^{2}\barψ}\right)$ function evaluations, where $\barψ \geq \tildeΩ\left(\sqrtε\right)$ is a parameter capturing the extent to which the function of interest exhibits the strict saddle property.

math.OC↗

arXiv2023

On the Global Optimality of Direct Policy Search for Nonsmooth $H_\infty$ Output-Feedback Control

Direct policy search has achieved great empirical success in reinforcement learning. Recently, there has been increasing interest in studying its theoretical properties for continuous control, and fruitful results have been established for linear quadratic regulator (LQR) and linear quadratic Gaussian (LQG) control that are smooth and nonconvex. In this paper, we consider the standard $H_\infty$ robust control for output feedback systems and investigate the global optimality of direct policy search. Unlike LQR or LQG, the $H_\infty$ cost function is nonsmooth in the policy space. Despite the lack of smoothness and convexity, our main result shows that for a class of non-degenerated stabilizing controllers, all Clarke stationary points of $H_\infty$ robust control are globally optimal and there is no spurious local minimum. Our proof technique is motivated by the idea of differentiable convex liftings (DCL), and we extend DCL to analyze the nonsmooth and nonconvex $H_\infty$ robust control via convex reformulation. Our result sheds some light on the analysis of direct policy search for solving nonsmooth and nonconvex robust control problems.

math.OC↗

arXiv2023

Beef up mmWave Dense Cellular Networks with D2D-Assisted Cooperative Edge Caching

Edge caching is emerging as the most promising solution to reduce the content retrieval delay and relieve the huge burden on the backhaul links in the ultra-dense networks by proactive caching popular contents in the small base station (SBS). However, constraint cache resource of individual SBSs significantly throttles the performance of edge caching. In this paper, we propose a device-to-device (D2D) assisted cooperative edge caching (DCEC) policy for millimeter (mmWave) dense networks, which cooperatively utilizes the cache resource of users and SBSs in proximity. In the proposed DCEC policy, a content can be cached in either users' devices or SBSs according to the content popularity, and a user can retrieve the requested content from neighboring users via D2D links or the neighboring SBSs via cellular links to efficiently exploit the cache diversity. Unlike existing cooperative caching policies in the lower frequency bands that require complex interference management techniques to suppress interference, we take advantage of directional antenna in mmWave systems to ensure high transmission rate whereas mitigating interference footprint. Taking the practical directional antenna model and the network density into consideration, we derive closed-form expressions of the backhaul offloading performance and content retrieval delay based on the stochastic information of network topology. In addition, analytical results indicate that, with the increase of the network density, the content retrieval delay via D2D links increases significantly while that via cellular links increases slightly. Comprehensive simulations validate our theoretical analysis and demonstrate that the proposed policy can achieve higher performance in offloading the backhaul traffic and reducing the content retrieval delay compared with the state-of-the-art most popular caching (MPC) policy.

eess.SY↗

arXiv2022

Improve Single-Point Zeroth-Order Optimization Using High-Pass and Low-Pass Filters

Single-point zeroth-order optimization (SZO) is useful in solving online black-box optimization and control problems in time-varying environments, as it queries the function value only once at each time step. However, the vanilla SZO method is known to suffer from a large estimation variance and slow convergence, which seriously limits its practical application. In this work, we borrow the idea of high-pass and low-pass filters from extremum seeking control (continuous-time version of SZO) and develop a novel SZO method called HLF-SZO by integrating these filters. It turns out that the high-pass filter coincides with the residual feedback method, and the low-pass filter can be interpreted as the momentum method. As a result, the proposed HLF-SZO achieves a much smaller variance and much faster convergence than the vanilla SZO method and empirically outperforms the residual-feedback SZO method, which is verified via extensive numerical experiments.

math.OC↗

arXiv2022

Reinforcement Learning for Selective Key Applications in Power Systems: Recent Advances and Future Challenges

With large-scale integration of renewable generation and distributed energy resources, modern power systems are confronted with new operational challenges, such as growing complexity, increasing uncertainty, and aggravating volatility. Meanwhile, more and more data are becoming available owing to the widespread deployment of smart meters, smart sensors, and upgraded communication networks. As a result, data-driven control techniques, especially reinforcement learning (RL), have attracted surging attention in recent years. This paper provides a comprehensive review of various RL techniques and how they can be applied to decision-making and control in power systems. In particular, we select three key applications, i.e., frequency regulation, voltage control, and energy management, as examples to illustrate RL-based models and solutions. We then present the critical issues in the application of RL, i.e., safety, robustness, scalability, and data. Several potential future directions are discussed as well.

cs.LG↗

arXiv2021

Communication-Efficient Distributed SGD with Compressed Sensing

We consider large scale distributed optimization over a set of edge devices connected to a central server, where the limited communication bandwidth between the server and edge devices imposes a significant bottleneck for the optimization procedure. Inspired by recent advances in federated learning, we propose a distributed stochastic gradient descent (SGD) type algorithm that exploits the sparsity of the gradient, when possible, to reduce communication burden. At the heart of the algorithm is to use compressed sensing techniques for the compression of the local stochastic gradients at the device side; and at the server side, a sparse approximation of the global stochastic gradient is recovered from the noisy aggregated compressed local gradients. We conduct theoretical analysis on the convergence of our algorithm in the presence of noise perturbation incurred by the communication channels, and also conduct numerical experiments to corroborate its effectiveness.

math.OC↗

arXiv2021

Reinforcement Learning Compensated Extended Kalman Filter for Attitude Estimation

Inertial measurement units are widely used in different fields to estimate the attitude. Many algorithms have been proposed to improve estimation performance. However, most of them still suffer from 1) inaccurate initial estimation, 2) inaccurate initial filter gain, and 3) non-Gaussian process and/or measurement noise. In this paper, we leverage reinforcement learning to compensate for the classical extended Kalman filter estimation, i.e., to learn the filter gain from the sensor measurements. We also analyse the convergence of the estimate error. The effectiveness of the proposed algorithm is validated on both simulated data and real data.

cs.RO↗

arXiv2021

Reinforcement Learning for Orientation Estimation Using Inertial Sensors with Performance Guarantee

This paper presents a deep reinforcement learning (DRL) algorithm for orientation estimation using inertial sensors combined with magnetometer. The Lyapunov method in control theory is employed to prove the convergence of orientation estimation errors. Based on the theoretical results, the estimator gains and a Lyapunov function are parametrized by deep neural networks and learned from samples. The DRL estimator is compared with three well-known orientation estimation methods on both numerical simulations and real datasets collected from commercially available sensors. The results show that the proposed algorithm is superior for arbitrary estimation initialization and can adapt to very large angular velocities for which other algorithms can be hardly applicable. To the best of our knowledge, this is the first DRL-based orientation estimation method with estimation error boundedness guarantee.

cs.RO↗

arXiv2021

Zeroth-Order Feedback Optimization for Cooperative Multi-Agent Systems

We study a class of cooperative multi-agent optimization problems, where each agent is associated with a local action vector and a local cost, and the goal is to cooperatively find the joint action profile that minimizes the average of the local costs. Such problems arise in many applications, such as distributed routing control, wind farm operation, etc. In many of these problems, gradient information may not be readily available, and the agents may only observe their local costs incurred by their actions as a feedback to determine their new actions. In this paper, we propose a zeroth-order feedback optimization scheme for the class of problems we consider, and provide explicit complexity bounds for both the convex and nonconvex settings with noiseless and noisy local cost observations. We also discuss briefly on the impacts of knowledge of local function dependence between agents. The algorithm's performance is justified by a numerical example of distributed routing control.

math.OC↗

arXiv2021

Analysis of the Optimization Landscape of Linear Quadratic Gaussian (LQG) Control

This paper revisits the classical Linear Quadratic Gaussian (LQG) control from a modern optimization perspective. We analyze two aspects of the optimization landscape of the LQG problem: 1) connectivity of the set of stabilizing controllers $\mathcal{C}_n$; and 2) structure of stationary points. It is known that similarity transformations do not change the input-output behavior of a dynamical controller or LQG cost. This inherent symmetry by similarity transformations makes the landscape of LQG very rich. We show that 1) the set of stabilizing controllers $\mathcal{C}_n$ has at most two path-connected components and they are diffeomorphic under a mapping defined by a similarity transformation; 2) there might exist many \emph{strictly suboptimal stationary points} of the LQG cost function over $\mathcal{C}_n$ and these stationary points are always \emph{non-minimal}; 3) all \emph{minimal} stationary points are globally optimal and they are identical up to a similarity transformation. These results shed some light on the performance analysis of direct policy gradient methods for solving the LQG problem.

math.OC↗

arXiv2020

Semi-Global Exponential Stability of Augmented Primal-Dual Gradient Dynamics for Constrained Convex Optimization

Primal-dual gradient dynamics that find saddle points of a Lagrangian have been widely employed for handling constrained optimization problems. Building on existing methods, we extend the augmented primal-dual gradient dynamics (Aug-PDGD) to incorporate general convex and nonlinear inequality constraints, and we establish its semi-global exponential stability when the objective function is strongly convex. We also provide an example of a strongly convex quadratic program of which the Aug-PDGD fails to achieve global exponential stability. Numerical simulation also suggests that the exponential convergence rate could depend on the initial distance to the KKT point.

math.OC↗

arXiv2020

Distributed Reinforcement Learning for Decentralized Linear Quadratic Control: A Derivative-Free Policy Optimization Approach

This paper considers a distributed reinforcement learning problem for decentralized linear quadratic control with partial state observations and local costs. We propose a Zero-Order Distributed Policy Optimization algorithm (ZODPO) that learns linear local controllers in a distributed fashion, leveraging the ideas of policy gradient, zero-order optimization and consensus algorithms. In ZODPO, each agent estimates the global cost by consensus, and then conducts local policy gradient in parallel based on zero-order gradient estimation. ZODPO only requires limited communication and storage even in large-scale systems. Further, we investigate the nonasymptotic performance of ZODPO and show that the sample complexity to approach a stationary point is polynomial with the error tolerance's inverse and the problem dimensions, demonstrating the scalability of ZODPO. We also show that the controllers generated throughout ZODPO are stabilizing controllers with high probability. Lastly, we numerically test ZODPO on multi-zone HVAC systems.

eess.SY↗

arXiv2020

Distributed Zero-Order Algorithms for Nonconvex Multi-Agent Optimization

Distributed multi-agent optimization finds many applications in distributed learning, control, estimation, etc. Most existing algorithms assume knowledge of first-order information of the objective and have been analyzed for convex problems. However, there are situations where the objective is nonconvex, and one can only evaluate the function values at finitely many points. In this paper we consider derivative-free distributed algorithms for nonconvex multi-agent optimization, based on recent progress in zero-order optimization. We develop two algorithms for different settings, provide detailed analysis of their convergence behavior, and compare them with existing centralized zero-order algorithms and gradient-based distributed algorithms.

math.OC↗

arXiv2018

Running Primal-Dual Gradient Method for Time-Varying Nonconvex Problems

This paper considers a nonconvex optimization problem that evolves over time, and addresses the synthesis and analysis of regularized primal-dual gradient methods to track a Karush-Kuhn-Tucker (KKT) trajectory. The proposed regularized primal-dual gradient methods are implemented in a running fashion, in the sense that the underlying optimization problem changes during the iterations of the algorithms. For a problem with twice continuously differentiable cost and constraints, and under a generalization of the Mangasarian-Fromovitz constraint qualification, sufficient conditions are derived for the running algorithm to track a KKT trajectory. Further, asymptotic bounds for the tracking error (as a function of the time-variability of a KKT trajectory) are obtained. A continuous-time version of the algorithm, framed as a system of differential inclusions, is also considered and analytical convergence results are derived. For the continuous-time setting, a set of sufficient conditions for the KKT trajectories not to bifurcate or merge is proposed. Illustrative numerical results inspired by a real-world application are provided.

math.OC↗