arXiv subjects

Yongcan Cao

Publications and source records attributed to Yongcan Cao.

At least 37 records · Page 2Linked to original sources

arXiv2023

Fairness in Preference-based Reinforcement Learning

In this paper, we address the issue of fairness in preference-based reinforcement learning (PbRL) in the presence of multiple objectives. The main objective is to design control policies that can optimize multiple objectives while treating each objective fairly. Toward this objective, we design a new fairness-induced preference-based reinforcement learning or FPbRL. The main idea of FPbRL is to learn vector reward functions associated with multiple objectives via new welfare-based preferences rather than reward-based preference in PbRL, coupled with policy learning via maximizing a generalized Gini welfare function. Finally, we provide experiment studies on three different environments to show that the proposed FPbRL approach can achieve both efficiency and equity for learning effective and fair policies.

cs.LG↗

arXiv2021

Deep Model Compression Via Two-Stage Deep Reinforcement Learning

Besides accuracy, the model size of convolutional neural networks (CNN) models is another important factor considering limited hardware resources in practical applications. For example, employing deep neural networks on mobile systems requires the design of accurate yet fast CNN for low latency in classification and object detection. To fulfill the need, we aim at obtaining CNN models with both high testing accuracy and small size to address resource constraints in many embedded devices. In particular, this paper focuses on proposing a generic reinforcement learning-based model compression approach in a two-stage compression pipeline: pruning and quantization. The first stage of compression, i.e., pruning, is achieved via exploiting deep reinforcement learning (DRL) to co-learn the accuracy and the FLOPs updated after layer-wise channel pruning and element-wise variational pruning via information dropout. The second stage, i.e., quantization, is achieved via a similar DRL approach but focuses on obtaining the optimal bits representation for individual layers. We further conduct experimental results on CIFAR-10 and ImageNet datasets. For the CIFAR-10 dataset, the proposed method can reduce the size of VGGNet by 9x from 20.04MB to 2.2MB with a slight accuracy increase. For the ImageNet dataset, the proposed method can reduce the size of VGG-16 by 33x from 138MB to 4.14MB with no accuracy loss.

cs.LG↗

arXiv2020

Human-guided Robot Behavior Learning: A GAN-assisted Preference-based Reinforcement Learning Approach

Human demonstrations can provide trustful samples to train reinforcement learning algorithms for robots to learn complex behaviors in real-world environments. However, obtaining sufficient demonstrations may be impractical because many behaviors are difficult for humans to demonstrate. A more practical approach is to replace human demonstrations by human queries, i.e., preference-based reinforcement learning. One key limitation of the existing algorithms is the need for a significant amount of human queries because a large number of labeled data is needed to train neural networks for the approximation of a continuous, high-dimensional reward function. To reduce and minimize the need for human queries, we propose a new GAN-assisted human preference-based reinforcement learning approach that uses a generative adversarial network (GAN) to actively learn human preferences and then replace the role of human in assigning preferences. The adversarial neural network is simple and only has a binary output, hence requiring much less human queries to train. Moreover, a maximum entropy based reinforcement learning algorithm is designed to shape the loss towards the desired regions or away from the undesired regions. To show the effectiveness of the proposed approach, we present some studies on complex robotic tasks without access to the environment reward in a typical MuJoCo robot locomotion environment. The obtained results show our method can achieve a reduction of about 99.8% human time without performance sacrifice.

cs.RO↗

arXiv2020

Learn to Exceed: Stereo Inverse Reinforcement Learning with Concurrent Policy Optimization

In this paper, we study the problem of obtaining a control policy that can mimic and then outperform expert demonstrations in Markov decision processes where the reward function is unknown to the learning agent. One main relevant approach is the inverse reinforcement learning (IRL), which mainly focuses on inferring a reward function from expert demonstrations. The obtained control policy by IRL and the associated algorithms, however, can hardly outperform expert demonstrations. To overcome this limitation, we propose a novel method that enables the learning agent to outperform the demonstrator via a new concurrent reward and action policy learning approach. In particular, we first propose a new stereo utility definition that aims to address the bias in the interpretation of expert demonstrations. We then propose a loss function for the learning agent to learn reward and action policies concurrently such that the learning agent can outperform expert demonstrations. The performance of the proposed method is first demonstrated in OpenAI environments. Further efforts are conducted to experimentally validate the proposed method via an indoor drone flight scenario.

cs.LG↗

arXiv2020

Graph Based Multi-layer K-means++ (G-MLKM) for Sensory Pattern Analysis in Constrained Spaces

In this paper, we focus on developing a novel unsupervised machine learning algorithm, named graph based multi-layer k-means++ (G-MLKM), to solve data-target association problem when targets move on a constrained space and minimal information of the targets can be obtained by sensors. Instead of employing the traditional data-target association methods that are based on statistical probabilities, the G-MLKM solves the problem via data clustering. We first will develop the Multi-layer K-means++ (MLKM) method for data-target association at local space given a simplified constrained space situation. Then a p-dual graph is proposed to represent the general constrained space when local spaces are interconnected. Based on the dual graph and graph theory, we then generalize MLKM to G-MLKM by first understanding local data-target association and then extracting cross-local data-target association mathematically analyze the data association at intersections of that space. To exclude potential data-target association errors that disobey physical rules, we also develop error correction mechanisms to further improve the accuracy. Numerous simulation examples are conducted to demonstrate the performance of G-MLKM.

cs.LG↗

arXiv2019

Relationship Explainable Multi-objective Optimization Via Vector Value Function Based Reinforcement Learning

Solving multi-objective optimization problems is important in various applications where users are interested in obtaining optimal policies subject to multiple, yet often conflicting objectives. A typical approach to obtain optimal policies is to first construct a loss function that is based on the scalarization of individual objectives, and then find the optimal policy that minimizes the loss. However, optimizing the scalarized (and weighted) loss does not necessarily provide a guarantee of high performance on each possibly conflicting objective. In this paper, we propose a vector value based reinforcement learning approach that seeks to explicitly learn the inter-objective relationship and optimize multiple objectives based on the learned relationship. In particular, the proposed method is to first define relationship matrix, a mathematical representation of the inter-objective relationship, and then create one actor and multiple critics that can co-learn the relationship matrix and action selection. The proposed approach can quantify the inter-objective relationship via reinforcement learning when the impact of one objective on another is unknown a prior. We also provide rigorous convergence analysis of the proposed approach and present a quantitative evaluation of the approach based on two testing scenarios.

eess.SY↗

arXiv2019

Relationship Explainable Multi-objective Reinforcement Learning with Semantic Explainability Generation

Solving multi-objective optimization problems is important in various applications where users are interested in obtaining optimal policies subject to multiple, yet often conflicting objectives. A typical approach to obtain optimal policies is to first construct a loss function that is based on the scalarization of individual objectives, and then find the optimal policy that minimizes the loss. However, optimizing the scalarized (and weighted) loss does not necessarily provide guarantee of high performance on each possibly conflicting objective because it is challenging to assign the right weights without knowing the relationship among these objectives. Moreover, the effectiveness of these gradient descent algorithms is limited by the agent's ability to explain their decisions and actions to human users. The purpose of this study is two-fold. First, we propose a vector value function based multi-objective reinforcement learning (V2f-MORL) approach that seeks to quantify the inter-objective relationship via reinforcement learning (RL) when the impact of one objective on others is unknown a prior. In particular, we construct one actor and multiple critics that can co-learn the policy and inter-objective relationship matrix (IORM), quantifying the impact of objectives on each other, in an iterative way. Second, we provide a semantic representation that can uncover the trade-off of decision policies made by users to reconcile conflicting objectives based on the proposed V2f-MORL approach for the explainability of the generated behaviors subject to given optimization objectives. We demonstrate the effectiveness of the proposed approach via a MuJoCo based robotics case study.

eess.SY↗

arXiv2017

A Multi-Layer K-means Approach for Multi-Sensor Data Pattern Recognition in Multi-Target Localization

Data-target association is an important step in multi-target localization for the intelligent operation of un- manned systems in numerous applications such as search and rescue, traffic management and surveillance. The objective of this paper is to present an innovative data association learning approach named multi-layer K-means (MLKM) based on leveraging the advantages of some existing machine learning approaches, including K-means, K-means++, and deep neural networks. To enable the accurate data association from different sensors for efficient target localization, MLKM relies on the clustering capabilities of K-means++ structured in a multi-layer framework with the error correction feature that is motivated by the backpropogation that is well-known in deep learning research. To show the effectiveness of the MLKM method, numerous simulation examples are conducted to compare its performance with K-means, K-means++, and deep neural networks.

eess.SY↗

arXiv2017

Multi-Objective Cooperative Search of Spatially Diverse Routes in Uncertain Environments

This paper focuses on developing new navigation and reconnaissance capabilities for cooperative unmanned systems in uncertain environments. The goal is to design a cooperative multi-vehicle system that can survey an unknown environment and find the most valuable route for personnel to travel. To accomplish the goal, the multi-vehicle system first explores spatially diverse routes and then selects the safest route. In particular, the proposed cooperative path planner sequentially generates a set of spatially diverse routes according to a number of factors, including travel distance, ease of travel, and uncertainty associated with the ease of travel. The planner's dependence on each of these factors is altered by a weighted score, doing so changes the criteria for determining an optimum route. To penalize the selection of same paths by different vehicles, a control gain is used to increase the cost of paths that lie near the route(s) assigned to other vehicles. By varying the control gain, the spatial diversity among routes can be accomplished. By repeatedly searching for different paths cooperatively, an optimal path can be selected that yields the most valuable route.

eess.SY↗

arXiv2017

Multi-Sensor Data Pattern Recognition for Multi-Target Localization: A Machine Learning Approach

Data-target pairing is an important step towards multi-target localization for the intelligent operation of unmanned systems. Target localization plays a crucial role in numerous applications, such as search, and rescue missions, traffic management and surveillance. The objective of this paper is to present an innovative target location learning approach, where numerous machine learning approaches, including K-means clustering and supported vector machines (SVM), are used to learn the data pattern across a list of spatially distributed sensors. To enable the accurate data association from different sensors for accurate target localization, appropriate data pre-processing is essential, which is then followed by the application of different machine learning algorithms to appropriately group data from different sensors for the accurate localization of multiple targets. Through simulation examples, the performance of these machine learning algorithms is quantified and compared.

eess.SY↗

arXiv2016

Towards Energy-Efficient Communication Management in the Distributed Control of Networked Cyber-Physical Systems

In this paper, we study the distributed control of networked cyber-physical systems when a much more energy-efficient distributed communication management strategy is proposed to solve the well-studied consensus problem. In contrast to the existing potential-based network topology control method, the proposed topology control method is based on the variation of communication ranges such that each agent can control its ad hoc communication range. The proposed network topology control technique can not only guarantee network connectivity but also reduce the communication energy. We apply the new network topology control technique, based on variable communication ranges, in a well-studied consensus problem, where the communication range for each agent is designed locally along with a new bounded control algorithm. Theoretical analysis is then provided to show that the proposed network topology control technique can guarantee consensus with bounded communication energy consumption. Finally, simulation examples are provided to show the effectiveness of the proposed energy-efficient distributed topology control technique.

eess.SY↗

arXiv2015

Decentralized Event-Triggered Consensus over Unreliable Communication Networks

This article studies distributed event-triggered consensus over unreliable communication channels. Communication is unreliable in the sense that the broadcast channel from one agent to its neighbors can drop the event-triggered packets of information, where the transmitting agent is unaware that the packet was not received and the receiving agents have no knowledge of the transmitted packet. Additionally, packets that successfully arrive at their destination may suffer from time-varying communication delays. In this paper, we consider directed graphs, and we also relax the consistency on the packet dropouts and the delays. Relaxing consistency means that the delays and dropouts for a packet broadcast by one agent can be different for each receiving node. We show that even under this challenging scenario, agents can reach consensus asymptotically while reducing transmissions of measurements based on the proposed event-triggered consensus protocol. In addition, positive inter-event times are obtained which guarantee that Zeno behavior does not occur.

math.OC↗

arXiv2015

Distributed Adaptive Fault-Tolerant Control of Uncertain Multi-Agent Systems

This paper presents an adaptive fault-tolerant control (FTC) scheme for a class of nonlinear uncertain multi-agent systems. A local FTC scheme is designed for each agent using local measurements and suitable information exchanged between neighboring agents. Each local FTC scheme consists of a fault diagnosis module and a reconfigurable controller module comprised of a baseline controller and two adaptive fault-tolerant controllers activated after fault detection and after fault isolation, respectively. Under certain assumptions, the closed-loop system's stability and leader-follower consensus properties are rigorously established under different modes of the FTC system, including the time-period before possible fault detection, between fault detection and possible isolation, and after fault isolation.

eess.SY↗

arXiv2015

Periodic Event-Triggered Synchronization of Linear Multi-agent Systems with Communication Delays

Multi-agent systems cooperation to achieve global goals is usually limited by sensing, actuation, and communication issues. At the local level, continuous measurement and actuation is only approximated by the use of digital mechanisms that measure and process information in order to compute and update new control input values at discrete time instants. Interaction with other agents or subsystems takes place, in general, through a digital communication channel with limited bandwidth where transmission of continuous-time signals is not possible. Additionally, communication channels may be subject to other imperfections such as time-varying delays. This paper considers the problem of consensus (or synchronization of state trajectories) of multi-agent systems that are described by general linear dynamics and are connected using undirected graphs. An event-triggered consensus protocol is proposed, where each agent implements discretized and decoupled models of the states of its neighbors. This approach not only avoids the need for continuous communication between agents but also provides a decentralized method for transmission of information in the presence of time-varying communication delays where each agent decides its own broadcasting time instants based only on local information. This method gives more flexibility for scheduling information broadcasting compared to periodic and sampled-data implementations. The use of discretized models by each agent allows for a periodic event-triggered strategy where continuous actuation and continuous measurement of the states are not necessary.

math.OC↗

arXiv2015

Decentralized Event-Triggered Consensus of Linear Multi-agent Systems under Directed Graphs

An event-triggered control technique for consensus of multi-agent systems with general linear dynamics is presented. This paper extends previous work to consider agents that are connected using directed graphs. Additionally, the approach shown here provides asymptotic consensus with guaranteed positive inter-event time intervals. This event-triggered control method is also used in the case where communication delays are present. For the communication delay case we also show that the agents achieve consensus asymptotically and that, for every agent, the time intervals between consecutive transmissions is lower-bounded by a positive constant.

math.OC↗

arXiv2015

Bridge Consensus: Ignoring Initial Inessentials

In this paper, the problem of bridge consensus is presented and solved. Bridge consensus consists of a network of nodes, some of whom are participating and others are non-participating. The objective is for all the agents to reach average consensus of the participating nodes initial values in a distributed and scalable manner. To do this, the nodes must use the network connections of the non-participating nodes, which act as bridges for information and ignore the initial values of the non-participating nodes. The solution to this problem is made by merging the ideas from estimation theory and consensus theory. By considering the participating nodes has having equal information and the non-participating nodes as having no information, the nodes initial values are transformed into information space. Two consensus filters are run in parallel on the information state and information matrix. Conditions ensuring that the product of the inverse information matrix and the information state of each agent reaches average consensus of the participating agents' initial values is given.

eess.SY↗

arXiv2014

UAV Circumnavigating an Unknown Target Under a GPS-denied Environment with Range-only Measurements

One typical application of unmanned aerial vehicles is the intelligence, surveillance, and reconnaissance mission, where the objective is to improve situation awareness through information acquisition. For examples, an efficient way to gather information regarding a target is to deploy UAV in such a way that it orbits around this target at a desired distance. Such a UAV motion is called circumnavigation. The objective of the paper is to design a UAV control algorithm such that this circumnavigation mission is achieved under a GPS-denied environment using range-only measurement. The control algorithm is constructed in two steps. The first step is to design a UAV control algorithm by assuming the availability of both range and range rate measurements, where the associated control input is always bounded. The second step is to further eliminate the use of range rate measurement by using an estimated range rate, obtained via a sliding-mode estimator using range measurement, to replace actual range rate measurement. Such a controller design technique is applicable in the control design of other UAV navigation and control missions under a GPS-denied environment.

eess.SY↗

arXiv2013

UAV Circumnavigation of an Unknown Target Without Location Information Using Noisy Range-based Measurements

This paper proposes a control algorithm for a UAV to circumnavigate an unknown target at a fixed radius when the location information of the UAV is unavailable. By assuming that the UAV has a constant velocity, the control algorithm makes adjustments to the heading angle of the UAV based on range and range rate measurements from the target, which may be corrupted by additive measurement noise. The control algorithm has the added benefit of being globally smooth and bounded. Exploiting the relationship between range rate and bearing angle, we transform the system dynamics from Cartesian coordinate in terms of location and heading to polar coordinate in terms of range and bearing angle. We then formulate the addition of measurement errors as a stochastic differential equation. A recurrence result is established showing that the UAV will reach a neighborhood of the desired orbit in finite time. Some statistical measures of performance are obtained to support the technical analysis.

math.OC↗