arXiv subjects

Donglin Wang

Publications and source records attributed to Donglin Wang.

162 records · Page 9Linked to original sources

arXiv2022

DARA: Dynamics-Aware Reward Augmentation in Offline Reinforcement Learning

Offline reinforcement learning algorithms promise to be applicable in settings where a fixed dataset is available and no new experience can be acquired. However, such formulation is inevitably offline-data-hungry and, in practice, collecting a large offline dataset for one specific task over one specific environment is also costly and laborious. In this paper, we thus 1) formulate the offline dynamics adaptation by using (source) offline data collected from another dynamics to relax the requirement for the extensive (target) offline data, 2) characterize the dynamics shift problem in which prior offline methods do not scale well, and 3) derive a simple dynamics-aware reward augmentation (DARA) framework from both model-free and model-based offline settings. Specifically, DARA emphasizes learning from those source transition pairs that are adaptive for the target environment and mitigates the offline dynamics shift by characterizing state-action-next-state pairs instead of the typical state-action distribution sketched by prior offline RL methods. The experimental evaluation demonstrates that DARA, by augmenting rewards in the source offline dataset, can acquire an adaptive policy for the target environment and yet significantly reduce the requirement of target offline data. With only modest amounts of target offline data, our performance consistently outperforms the prior offline RL methods in both simulated and real-world tasks.

cs.LG↗

arXiv2022

A Transferable Legged Mobile Manipulation Framework Based on Disturbance Predictive Control

Due to their ability to adapt to different terrains, quadruped robots have drawn much attention in the research field of robot learning. Legged mobile manipulation, where a quadruped robot is equipped with a robotic arm, can greatly enhance the performance of the robot in diverse manipulation tasks. Several prior works have investigated legged mobile manipulation from the viewpoint of control theory. However, modeling a unified structure for various robotic arms and quadruped robots is a challenging task. In this paper, we propose a unified framework disturbance predictive control where a reinforcement learning scheme with a latent dynamic adapter is embedded into our proposed low-level controller. Our method can adapt well to various types of robotic arms with a few random motion samples and the experimental results demonstrate the effectiveness of our method.

cs.RO↗

arXiv2022

Imitation and Adaptation Based on Consistency: A Quadruped Robot Imitates Animals from Videos Using Deep Reinforcement Learning

The essence of quadrupeds' movements is the movement of the center of gravity, which has a pattern in the action of quadrupeds. However, the gait motion planning of the quadruped robot is time-consuming. Animals in nature can provide a large amount of gait information for robots to learn and imitate. Common methods learn animal posture with a motion capture system or numerous motion data points. In this paper, we propose a video imitation adaptation network (VIAN) that can imitate the action of animals and adapt it to the robot from a few seconds of video. The deep learning model extracts key points during animal motion from videos. The VIAN eliminates noise and extracts key information of motion with a motion adaptor, and then applies the extracted movements function as the motion pattern into deep reinforcement learning (DRL). To ensure similarity between the learning result and the animal motion in the video, we introduce rewards that are based on the consistency of the motion. DRL explores and learns to maintain balance from movement patterns from videos, imitates the action of animals, and eventually, allows the model to learn the gait or skills from short motion videos of different animals and to transfer the motion pattern to the real robot.

cs.RO↗

arXiv2021

Learn Goal-Conditioned Policy with Intrinsic Motivation for Deep Reinforcement Learning

It is of significance for an agent to learn a widely applicable and general-purpose policy that can achieve diverse goals including images and text descriptions. Considering such perceptually-specific goals, the frontier of deep reinforcement learning research is to learn a goal-conditioned policy without hand-crafted rewards. To learn this kind of policy, recent works usually take as the reward the non-parametric distance to a given goal in an explicit embedding space. From a different viewpoint, we propose a novel unsupervised learning approach named goal-conditioned policy with intrinsic motivation (GPIM), which jointly learns both an abstract-level policy and a goal-conditioned policy. The abstract-level policy is conditioned on a latent variable to optimize a discriminator and discovers diverse states that are further rendered into perceptually-specific goals for the goal-conditioned policy. The learned discriminator serves as an intrinsic reward function for the goal-conditioned policy to imitate the trajectory induced by the abstract-level policy. Experiments on various robotic tasks demonstrate the effectiveness and efficiency of our proposed GPIM method which substantially outperforms prior techniques.

cs.LG↗

arXiv2021

Learning to Evolve on Dynamic Graphs

Representation learning in dynamic graphs is a challenging problem because the topology of graph and node features vary at different time. This requires the model to be able to effectively capture both graph topology information and temporal information. Most existing works are built on recurrent neural networks (RNNs), which are used to exact temporal information of dynamic graphs, and thus they inherit the same drawbacks of RNNs. In this paper, we propose Learning to Evolve on Dynamic Graphs (LEDG) - a novel algorithm that jointly learns graph information and time information. Specifically, our approach utilizes gradient-based meta-learning to learn updating strategies that have better generalization ability than RNN on snapshots. It is model-agnostic and thus can train any message passing based graph neural network (GNN) on dynamic graphs. To enhance the representation power, we disentangle the embeddings into time embeddings and graph intrinsic embeddings. We conduct experiments on various datasets and down-stream tasks, and the experimental results validate the effectiveness of our method.

cs.LG↗

arXiv2021

Mobility prediction Based on Machine Learning Algorithms

Nowadays mobile communication is growing fast in the 5G communication industry. With the increasing capacity requirements and requirements for quality of experience, mobility prediction has been widely applied to mobile communication and has becoming one of the key enablers that utilizes historical traffic information to predict future locations of traffic users, Since accurate mobility prediction can help enable efficient radio resource management, assist route planning, guide vehicle dispatching, or mitigate traffic congestion. However, mobility prediction is a challenging problem due to the complicated traffic network. In the past few years, plenty of researches have been done in this area, including Non-Machine-Learning (Non-ML)- based and Machine-Learning (ML)-based mobility prediction. In this paper, firstly we introduce the state of the art technologies for mobility prediction. Then, we selected Support Vector Machine (SVM) algorithm, the ML algorithm for practical traffic date training. Lastly, we analyse the simulation results for mobility prediction and introduce a future work plan where mobility prediction will be applied for improving mobile communication.

cs.LG↗

arXiv2021

Unsupervised Domain Adaptation with Dynamics-Aware Rewards in Reinforcement Learning

Unsupervised reinforcement learning aims to acquire skills without prior goal representations, where an agent automatically explores an open-ended environment to represent goals and learn the goal-conditioned policy. However, this procedure is often time-consuming, limiting the rollout in some potentially expensive target environments. The intuitive approach of training in another interaction-rich environment disrupts the reproducibility of trained skills in the target environment due to the dynamics shifts and thus inhibits direct transferring. Assuming free access to a source environment, we propose an unsupervised domain adaptation method to identify and acquire skills across dynamics. Particularly, we introduce a KL regularized objective to encourage emergence of skills, rewarding the agent for both discovering skills and aligning its behaviors respecting dynamics shifts. This suggests that both dynamics (source and target) shape the reward to facilitate the learning of adaptive skills. We also conduct empirical experiments to demonstrate that our method can effectively learn skills that can be smoothly deployed in target.

cs.LG↗

arXiv2021

Off-Dynamics Inverse Reinforcement Learning from Hetero-Domain

We propose an approach for inverse reinforcement learning from hetero-domain which learns a reward function in the simulator, drawing on the demonstrations from the real world. The intuition behind the method is that the reward function should not only be oriented to imitate the experts, but should encourage actions adjusted for the dynamics difference between the simulator and the real world. To achieve this, the widely used GAN-inspired IRL method is adopted, and its discriminator, recognizing policy-generating trajectories, is modified with the quantification of dynamics difference. The training process of the discriminator can yield the transferable reward function suitable for simulator dynamics, which can be guaranteed by derivation. Effectively, our method assigns higher rewards for demonstration trajectories which do not exploit discrepancies between the two domains. With extensive experiments on continuous control tasks, our method shows its effectiveness and demonstrates its scalability to high-dimensional tasks.

cs.LG↗

arXiv2021

Performance, Successes and Limitations of Deep Learning Semantic Segmentation of Multiple Defects in Transmission Electron Micrographs

In this work, we perform semantic segmentation of multiple defect types in electron microscopy images of irradiated FeCrAl alloys using a deep learning Mask Regional Convolutional Neural Network (Mask R-CNN) model. We conduct an in-depth analysis of key model performance statistics, with a focus on quantities such as predicted distributions of defect shapes, defect sizes, and defect areal densities relevant to informing modeling and understanding of irradiated Fe-based materials properties. To better understand the performance and present limitations of the model, we provide examples of useful evaluation tests which include a suite of random splits, and dataset size-dependent and domain-targeted cross validation tests. Overall, we find that the current model is a fast, effective tool for automatically characterizing and quantifying multiple defect types in microscopy images, with a level of accuracy on par with human domain expert labelers. More specifically, the model can achieve average defect identification F1 scores as high as 0.8, and, based on random cross validation, have low overall average (+/- standard deviation) defect size and density percentage errors of 7.3 (+/- 3.8)% and 12.7 (+/- 5.3)%, respectively. Further, our model predicts the expected material hardening to within 10-20 MPa (about 10% of total hardening), which is about the same error level as experiments. Our targeted evaluation tests also suggest the best path toward improving future models is not expanding existing databases with more labeled images but instead data additions that target weak points of the model domain, such as images from different microscopes, imaging conditions, irradiation environments, and alloy types. Finally, we discuss the first phase of an effort to provide an easy-to-use, open-source object detection tool to the broader community for identifying defects in new images.

cs.CV↗

arXiv2021

Deriving Autism Spectrum Disorder Functional Networks from RS-FMRI Data using Group ICA and Dictionary Learning

The objective of this study is to derive functional networks for the autism spectrum disorder (ASD) population using the group ICA and dictionary learning model together and to classify ASD and typically developing (TD) participants using the functional connectivity calculated from the derived functional networks. In our experiments, the ASD functional networks were derived from resting-state functional magnetic resonance imaging (rs-fMRI) data. We downloaded a total of 120 training samples, including 58 ASD and 62 TD participants, which were obtained from the public repository: Autism Brain Imaging Data Exchange I (ABIDE I). Our methodology and results have five main parts. First, we utilize a group ICA model to extract functional networks from the ASD group and rank the top 20 regions of interest (ROIs). Second, we utilize a dictionary learning model to extract functional networks from the ASD group and rank the top 20 ROIs. Third, we merged the 40 selected ROIs from the two models together as the ASD functional networks. Fourth, we generate three corresponding masks based on the 20 selected ROIs from group ICA, the 20 ROIs selected from dictionary learning, and the 40 combined ROIs selected from both. Finally, we extract ROIs for all training samples using the above three masks, and the calculated functional connectivity was used as features for ASD and TD classification. The classification results showed that the functional networks derived from ICA and dictionary learning together outperform those derived from a single ICA model or a single dictionary learning model.

q-bio.NC↗

arXiv2021

Adaptive Adversarial Training for Meta Reinforcement Learning

Meta Reinforcement Learning (MRL) enables an agent to learn from a limited number of past trajectories and extrapolate to a new task. In this paper, we attempt to improve the robustness of MRL. We build upon model-agnostic meta-learning (MAML) and propose a novel method to generate adversarial samples for MRL by using Generative Adversarial Network (GAN). That allows us to enhance the robustness of MRL to adversal attacks by leveraging these attacks during meta training process.

cs.LG↗

arXiv2021

Pareto Self-Supervised Training for Few-Shot Learning

While few-shot learning (FSL) aims for rapid generalization to new concepts with little supervision, self-supervised learning (SSL) constructs supervisory signals directly computed from unlabeled data. Exploiting the complementarity of these two manners, few-shot auxiliary learning has recently drawn much attention to deal with few labeled data. Previous works benefit from sharing inductive bias between the main task (FSL) and auxiliary tasks (SSL), where the shared parameters of tasks are optimized by minimizing a linear combination of task losses. However, it is challenging to select a proper weight to balance tasks and reduce task conflict. To handle the problem as a whole, we propose a novel approach named as Pareto self-supervised training (PSST) for FSL. PSST explicitly decomposes the few-shot auxiliary problem into multiple constrained multi-objective subproblems with different trade-off preferences, and here a preference region in which the main task achieves the best performance is identified. Then, an effective preferred Pareto exploration is proposed to find a set of optimal solutions in such a preference region. Extensive experiments on several public benchmark datasets validate the effectiveness of our approach by achieving state-of-the-art performance.

cs.CV↗

arXiv2021

Attributes-Guided and Pure-Visual Attention Alignment for Few-Shot Recognition

The purpose of few-shot recognition is to recognize novel categories with a limited number of labeled examples in each class. To encourage learning from a supplementary view, recent approaches have introduced auxiliary semantic modalities into effective metric-learning frameworks that aim to learn a feature similarity between training samples (support set) and test samples (query set). However, these approaches only augment the representations of samples with available semantics while ignoring the query set, which loses the potential for the improvement and may lead to a shift between the modalities combination and the pure-visual representation. In this paper, we devise an attributes-guided attention module (AGAM) to utilize human-annotated attributes and learn more discriminative features. This plug-and-play module enables visual contents and corresponding attributes to collectively focus on important channels and regions for the support set. And the feature selection is also achieved for query set with only visual information while the attributes are not available. Therefore, representations from both sets are improved in a fine-grained manner. Moreover, an attention alignment mechanism is proposed to distill knowledge from the guidance of attributes to the pure-visual branch for samples without attributes. Extensive experiments and analysis show that our proposed module can significantly improve simple metric-based approaches to achieve state-of-the-art performance on different datasets and settings.

cs.CV↗

arXiv2021

Visual Perception Generalization for Vision-and-Language Navigation via Meta-Learning

Vision-and-language navigation (VLN) is a challenging task that requires an agent to navigate in real-world environments by understanding natural language instructions and visual information received in real-time. Prior works have implemented VLN tasks on continuous environments or physical robots, all of which use a fixed camera configuration due to the limitations of datasets, such as 1.5 meters height, 90 degrees horizontal field of view (HFOV), etc. However, real-life robots with different purposes have multiple camera configurations, and the huge gap in visual information makes it difficult to directly transfer the learned navigation model between various robots. In this paper, we propose a visual perception generalization strategy based on meta-learning, which enables the agent to fast adapt to a new camera configuration with a few shots. In the training phase, we first locate the generalization problem to the visual perception module, and then compare two meta-learning algorithms for better generalization in seen and unseen environments. One of them uses the Model-Agnostic Meta-Learning (MAML) algorithm that requires a few shot adaptation, and the other refers to a metric-based meta-learning method with a feature-wise affine transformation layer. The experiment results show that our strategy successfully adapts the learned navigation model to a new camera configuration, and the two algorithms show their advantages in seen and unseen environments respectively.

cs.RO↗

arXiv2020

Effect of Retransmissions on the Performance of C-V2X Communication for 5G

In recent years, the next generation of wireless communication (5G) plays a significant role in both industry and academy societies. Cellular Vehicle-to-Everything (C-V2X) communication technology has been one of the prominent services for 5G. For C-V2X transmission mode, there is a newly defined communication channel (sidelink) that can support direct C-V2X communication. Direct C-V2X communication is a technology that allows vehicles to communicate and share safety-related information with other traffic participates directly without going through the cellular network. The C-V2X data packet will be delivered to all traffic Users (UE) in the proximity of the Transmitter (Tx). Some UEs might not successfully receive the data packets during one transmission but the sidelink Tx is not able to check whether the Receivers (Rxs) get the information or not due to the lack of feedback channel. For enabling the strict requirements in terms of reliability and latency for C-V2X communication, we propose and evaluate one retransmission scheme and retransmission with different traffic speed scheme. These schemes try to improve the reliability of the safety-related data by one blind retransmission without requiring feedback. Although this retransmission scheme is essential to C-V2X communication, the scheme has a limitation in the performance aspect because of its redundant retransmission. Since radio resources for CV2X communication are limited, we have to detect the effect of retransmission on the performance of the communication system. To the end, the simulator for evaluating the proposed schemes for the C-V2X communication has been implemented, and the performances of the different communication schemes are shown through the Packet Reception Ratio (PRR).

cs.NI↗

arXiv2020

Link Level Performance Comparison of C-V2X and ITS-G5 for Vehicular Channel Models

V2X communications plays a significant role in increasing traffic safety and efficiency by enabling vehicles to exchange their status information with other vehicles and traffic entities in their proximity. In this regard, two technologies emerged as the main contenders for enabling V2X communications which have stringent requirements in terms of latency and reliability due to their apparent safety criticality. The first one is the DSRC standard (referred to as ITS-G5 in Europe) that is well researched since 20 years and has attained enough technical maturity for current deployment. The second one is the relatively new CV2X standard that is nevertheless, based on the 3GPP standard family that have successful deployments in almost every corner of the globe. In this work, we compare the link level performance of the PHY protocols for both the technologies for different vehicular fading channel models. To this end, we construct and simulate the PHY pipelines and show the performance results by means of BLER} versus SNR graphs. Our investigations show that CV2X performs better than ITS-G5 for almost all the considered channel models due to better channel coding and estimation schemes.

eess.SP↗

arXiv2019

System-Level Simulator of LTE Sidelink C-V2X Communication for 5G

In recent years, Cellular-Vehicle-to-Everything (CV2X) has been an emerging area of interest attracting both the industry and academy societies to develop, which is also a prominent emerging service for the next generation of the cellular network (5G). In the time of the development, standardization, and further improvement of 5G, so simulations are essential to test and optimize algorithms and procedures prior to their implementation process of the equipment manufactures. And CV2X communication is used for information exchange among the traffic participants with network-assisted which can reduce traffic accidents and improve traffic efficiency. Moreover, it is also the primary enabler for cooperative driving. But CV2X communication has to meet different Quality of Service (QoS) requirements (e.g., ultra-high reliability (99.999%) and ultra-low latency). Guaranteeing high-level reliability is a big challenge. In order to assess system performance, accurate simulations of simple setups, as well as simulations of more complex systems via abstracted models are necessary for the CV2X communication. For checking the performance of the C-V2X communication on a highway scenario, a system-level simulator has been implemented. And, this simulation has been carried out on the network (system-level) context. Finally, the analysis and the simulation results for the C-V2X communication are presented, which shows that different objectives can be met via system-level simulation.

cs.NI↗

arXiv2019

Performances of C-V2X Communication on Highway under Varying Channel Propagation Models

In recent decades, both the industry and the academy society are sparing no efforts to develop and standardize the Cellular-Vehicle-to-Everything (C-V2X) communication which is one of the prominent emerging services of the next generation of wireless network (5G). C-V2X communication is used for information exchange among the traffic participants with network-assisted which can reduce traffic accidents and improve traffic efficiency. And it is also the primary enabler for cooperative driving. But C-V2X communication has to meet different Quality of Service (QoS) requirements (e.g., ultra-high reliability (99.999%) and ultra-low latency).

cs.NI↗