arXiv subjects

Tao Lin

Publications and source records attributed to Tao Lin.

At least 181 records · Page 10Linked to original sources

arXiv2021

Learning by Active Forgetting for Neural Networks

Remembering and forgetting mechanisms are two sides of the same coin in a human learning-memory system. Inspired by human brain memory mechanisms, modern machine learning systems have been working to endow machine with lifelong learning capability through better remembering while pushing the forgetting as the antagonist to overcome. Nevertheless, this idea might only see the half picture. Up until very recently, increasing researchers argue that a brain is born to forget, i.e., forgetting is a natural and active process for abstract, rich, and flexible representations. This paper presents a learning model by active forgetting mechanism with artificial neural networks. The active forgetting mechanism (AFM) is introduced to a neural network via a "plug-and-play" forgetting layer (P\&PF), consisting of groups of inhibitory neurons with Internal Regulation Strategy (IRS) to adjust the extinction rate of themselves via lateral inhibition mechanism and External Regulation Strategy (ERS) to adjust the extinction rate of excitatory neurons via inhibition mechanism. Experimental studies have shown that the P\&PF offers surprising benefits: self-adaptive structure, strong generalization, long-term learning and memory, and robustness to data and parameter perturbation. This work sheds light on the importance of forgetting in the learning process and offers new perspectives to understand the underlying mechanisms of neural networks.

cs.LG↗

arXiv2021

Representation Memorization for Fast Learning New Knowledge without Forgetting

The ability to quickly learn new knowledge (e.g. new classes or data distributions) is a big step towards human-level intelligence. In this paper, we consider scenarios that require learning new classes or data distributions quickly and incrementally over time, as it often occurs in real-world dynamic environments. We propose "Memory-based Hebbian Parameter Adaptation" (Hebb) to tackle the two major challenges (i.e., catastrophic forgetting and sample efficiency) towards this goal in a unified framework. To mitigate catastrophic forgetting, Hebb augments a regular neural classifier with a continuously updated memory module to store representations of previous data. To improve sample efficiency, we propose a parameter adaptation method based on the well-known Hebbian theory, which directly "wires" the output network's parameters with similar representations retrieved from the memory. We empirically verify the superior performance of Hebb through extensive experiments on a wide range of learning tasks (image classification, language model) and learning scenarios (continual, incremental, online). We demonstrate that Hebb effectively mitigates catastrophic forgetting, and it indeed learns new knowledge better and faster than the current state-of-the-art.

cs.LG↗

arXiv2021

The Multi-Modal Video Reasoning and Analyzing Competition

In this paper, we introduce the Multi-Modal Video Reasoning and Analyzing Competition (MMVRAC) workshop in conjunction with ICCV 2021. This competition is composed of four different tracks, namely, video question answering, skeleton-based action recognition, fisheye video-based action recognition, and person re-identification, which are based on two datasets: SUTD-TrafficQA and UAV-Human. We summarize the top-performing methods submitted by the participants in this competition and show their results achieved in the competition.

cs.CV↗

arXiv2021

From Monopoly to Competition: Optimal Contests Prevail

We study competition among contests in a general model that allows for an arbitrary and heterogeneous space of contest design, where the goal of the contest designers is to maximize the contestants' sum of efforts. Our main result shows that optimal contests in the monopolistic setting (i.e., those that maximize the sum of efforts in a model with a single contest) form an equilibrium in the model with competition among contests. Under a very natural assumption these contests are in fact dominant, and the equilibria that they form are unique. Moreover, equilibria with the optimal contests are Pareto-optimal even in cases where other equilibria emerge. In many natural cases, they also maximize the social welfare.

cs.GT↗

arXiv2021

The Optimal Size of an Epistemic Congress

We analyze the optimal size of a congress in a representative democracy. We take an epistemic view where voters decide on a binary issue with one ground truth outcome, and each voter votes correctly according to their competence levels in $[0, 1]$. Assuming that we can sample the best experts to form an epistemic congress, we find that the optimal congress size should be linear in the population size. This result is striking because it holds even when allowing the top representatives to be accurate with arbitrarily high probabilities. We then analyze real world data, finding that the actual sizes of congresses are much smaller than the optimal size our theoretical results suggest. We conclude by analyzing under what conditions congresses of sub-optimal sizes would still outperform direct democracy, in which all voters vote.

cs.AI↗

arXiv2021

Quasi-Global Momentum: Accelerating Decentralized Deep Learning on Heterogeneous Data

Decentralized training of deep learning models is a key element for enabling data privacy and on-device learning over networks. In realistic learning scenarios, the presence of heterogeneity across different clients' local datasets poses an optimization challenge and may severely deteriorate the generalization performance. In this paper, we investigate and identify the limitation of several decentralized optimization algorithms for different degrees of data heterogeneity. We propose a novel momentum-based method to mitigate this decentralized training difficulty. We show in extensive empirical experiments on various CV/NLP datasets (CIFAR-10, ImageNet, and AG News) and several network topologies (Ring and Social Network) that our method is much more robust to the heterogeneity of clients' data than other existing methods, by a significant improvement in test performance ($1\% \!-\! 20\%$). Our code is publicly available.

cs.LG↗

arXiv2021

Consensus Control for Decentralized Deep Learning

Decentralized training of deep learning models enables on-device learning over networks, as well as efficient scaling to large compute clusters. Experiments in earlier works reveal that, even in a data-center setup, decentralized training often suffers from the degradation in the quality of the model: the training and test performance of models trained in a decentralized fashion is in general worse than that of models trained in a centralized fashion, and this performance drop is impacted by parameters such as network size, communication topology and data partitioning. We identify the changing consensus distance between devices as a key parameter to explain the gap between centralized and decentralized training. We show in theory that when the training consensus distance is lower than a critical quantity, decentralized training converges as fast as the centralized counterpart. We empirically validate that the relation between generalization performance and consensus distance is consistent with this theoretical observation. Our empirical insights allow the principled design of better decentralized training schemes that mitigate the performance drop. To this end, we provide practical training guidelines and exemplify its effectiveness on the data-center setup as the important first step.

cs.LG↗

arXiv2021

Competing spreading dynamics in simplicial complex

Interactions in biology and social systems are not restricted to pairwise but can take arbitrary sizes. Extensive studies have revealed that the arbitrary-sized interactions significantly affect the spreading dynamics on networked systems. Competing spreading dynamics, i.e., several epidemics spread simultaneously and compete with each other, have been widely observed in the real world, yet the way arbitrary-sized interactions affect competing spreading dynamics still lacks systematic study. This study presents a model of two competing simplicial susceptible-infected-susceptible epidemics on a higher-order system represented by simplicial complex and analyzes the model's critical phenomena. In the proposed model, a susceptible node can only be infected by one of the two epidemics, and the transmission of infection to neighbors can occur through pairwise (i.e., an edge) and high-order (e.g., 2-simplex) interactions simultaneously. Through a mean-field (MF) theory analysis and numerical simulations, we show that the model displays rich dynamical behavior depending on the 2-simplex infection strength. When the 2-simplex infection strength is weak, the model's phase diagram is consistent with the simple graph, consisting of three regions: the absolute dominant regions for each epidemic and the epidemic-free region. With the increase of the 2-simplex infection strength, a new phase region called the alternative dominant region emerges. In this region, the survival of one epidemic depends on the initial conditions. Our theoretical analysis can reasonably predict the time evolution and steady-state outbreak size in each region. In addition, we further explore the model's phase diagram both when the 2-simplex infection strength is symmetrical and asymmetrical. The results show that the 2-simplex infection strength has a significant impact on the system phase diagram.

physics.soc-ph↗

arXiv2021

Event vertex and time reconstruction in large volume liquid scintillator detector

Large-volume liquid scintillator detectors with ultra-low background levels have been widely used to study neutrino physics and search for dark matter. Event vertex and event time are not only useful for event selection but also essential for the reconstruction of event energy. In this study, four event vertex and event time reconstruction algorithms using charge and time information collected by photomultiplier tubes were analyzed comprehensively. The effects of photomultiplier tube properties were also investigated. The results indicate that the transit time spread is the main effect degrading the vertex reconstruction, while the effect of dark noise is limited. In addition, when the event is close to the detector boundary, the charge information provides better performance for vertex reconstruction than the time information.

physics.ins-det↗

arXiv2021

Deep Learning for IoT

Deep learning and other machine learning approaches are deployed to many systems related to Internet of Things or IoT. However, it faces challenges that adversaries can take loopholes to hack these systems through tampering history data. This paper first presents overall points of adversarial machine learning. Then, we illustrate traditional methods, such as Petri Net cannot solve this new question efficiently. To help IoT data analysis more efficient, we propose a retrieval method based on deep learning (recurrent neural network). Besides, this paper presents a research on data retrieval solution to avoid hacking by adversaries in the fields of adversary machine leaning. It further directs the new approaches in terms of how to implementing this framework in IoT settings based on adversarial deep learning.

cs.LG↗

arXiv2021

Improving the energy uniformity for large liquid scintillator detectors

It is challenging to achieve high precision energy resolution for large liquid scintillator detectors. Energy non-uniformity is one of the main obstacles. To surmount it, a calibration-data driven method was developed previously to reconstruct event energy in the JUNO experiment. In this paper, we investigated the choice of calibration sources thoroughly, optimized the calibration positions and corrected the residual detector azimuthal asymmetry. All these efforts lead to a reduction of the energy non-uniformity near the detector boundary, from about 0.64% to 0.38%. And within the fiducial volume of the detector it is improved from 0.3% to 0.17%. As a result the energy resolution could be further improved.

physics.ins-det↗

arXiv2021

Ensemble Distillation for Robust Model Fusion in Federated Learning

Federated Learning (FL) is a machine learning setting where many devices collaboratively train a machine learning model while keeping the training data decentralized. In most of the current training schemes the central model is refined by averaging the parameters of the server model and the updated parameters from the client side. However, directly averaging model parameters is only possible if all models have the same structure and size, which could be a restrictive constraint in many scenarios. In this work we investigate more powerful and more flexible aggregation schemes for FL. Specifically, we propose ensemble distillation for model fusion, i.e. training the central classifier through unlabeled data on the outputs of the models from the clients. This knowledge distillation technique mitigates privacy risk and cost to the same extent as the baseline FL algorithms, but allows flexible aggregation over heterogeneous client models that can differ e.g. in size, numerical precision or structure. We show in extensive empirical experiments on various CV/NLP datasets (CIFAR-10/100, ImageNet, AG News, SST2) and settings (heterogeneous models/data) that the server model can be trained much faster, requiring fewer communication rounds than any existing FL technique so far.

cs.LG↗

arXiv2021

Muon reconstruction with a convolutional neural network in the JUNO detector

The Jiangmen Underground Neutrino Observatory (JUNO) is designed to determine the neutrino mass ordering and measure neutrino oscillation parameters. A precise muon reconstruction is crucial to reduce one of the major backgrounds induced by cosmic muons. This article proposes a novel muon reconstruction method based on convolutional neural network (CNN) models. In this method, the track information reconstructed by the top tracker is used for network training. The training dataset is augmented by applying a rotation to muon tracks to compensate for the limited angular coverage of the top tracker. The muon reconstruction with the CNN model can produce unbiased tracks with performance that spatial resolution is better than 10 cm and angular resolution is better than 0.6 degrees. By using a GPU accelerated implementation a speedup factor of 100 compared to existing CPU techniques has been demonstrated.

physics.ins-det↗

arXiv2021

Overcoming Long-term Catastrophic Forgetting through Adversarial Neural Pruning and Synaptic Consolidation

Artificial neural networks face the well-known problem of catastrophic forgetting. What's worse, the degradation of previously learned skills becomes more severe as the task sequence increases, known as the long-term catastrophic forgetting. It is due to two facts: first, as the model learns more tasks, the intersection of the low-error parameter subspace satisfying for these tasks becomes smaller or even does not exist; second, when the model learns a new task, the cumulative error keeps increasing as the model tries to protect the parameter configuration of previous tasks from interference. Inspired by the memory consolidation mechanism in mammalian brains with synaptic plasticity, we propose a confrontation mechanism in which Adversarial Neural Pruning and synaptic Consolidation (ANPyC) is used to overcome the long-term catastrophic forgetting issue. The neural pruning acts as long-term depression to prune task-irrelevant parameters, while the novel synaptic consolidation acts as long-term potentiation to strengthen task-relevant parameters. During the training, this confrontation achieves a balance in that only crucial parameters remain, and non-significant parameters are freed to learn subsequent tasks. ANPyC avoids forgetting important information and makes the model efficient to learn a large number of tasks. Specifically, the neural pruning iteratively relaxes the current task's parameter conditions to expand the common parameter subspace of the task; the synaptic consolidation strategy, which consists of a structure-aware parameter-importance measurement and an element-wise parameter updating strategy, decreases the cumulative error when learning new tasks. The full source code is available at https://github.com/GeoX-Lab/ANPyC.

cs.LG↗

arXiv2021

Current-induced magnetization switching in a chemically disordered A1 CoPt single layer

We report the first demonstration of the current-induced magnetization switching in a perpendicularly magnetized A1 CoPt single layer. We show that good perpendicular magnetic anisotropy can be obtained in a wide composition range of the A1 Co1-xPtx single layers, which allows to fabricate perpendicularly magnetized CoPt single layer with composition gradient to break the inversion symmetry of the structure. By fabricating the gradient CoPt single layer, we have evaluated the SOT efficiency and successfully realized the SOT-induced magnetization switching. Our study provides an approach to realize the current-induced magnetization in the ferromagnetic single layers without attaching SOT source materials.

cond-mat.mtrl-sci↗

arXiv2020

AST-GCN: Attribute-Augmented Spatiotemporal Graph Convolutional Network for Traffic Forecasting

Traffic forecasting is a fundamental and challenging task in the field of intelligent transportation. Accurate forecasting not only depends on the historical traffic flow information but also needs to consider the influence of a variety of external factors, such as weather conditions and surrounding POI distribution. Recently, spatiotemporal models integrating graph convolutional networks and recurrent neural networks have become traffic forecasting research hotspots and have made significant progress. However, few works integrate external factors. Therefore, based on the assumption that introducing external factors can enhance the spatiotemporal accuracy in predicting traffic and improving interpretability, we propose an attribute-augmented spatiotemporal graph convolutional network (AST-GCN). We model the external factors as dynamic attributes and static attributes and design an attribute-augmented unit to encode and integrate those factors into the spatiotemporal graph convolution model. Experiments on real datasets show the effectiveness of considering external information on traffic forecasting tasks when compared to traditional traffic prediction methods. Moreover, under different attribute-augmented schemes and prediction horizon settings, the forecasting accuracy of the AST-GCN is higher than that of the baselines.

cs.LG↗

arXiv2020

Decentralized Deep Learning with Arbitrary Communication Compression

Decentralized training of deep learning models is a key element for enabling data privacy and on-device learning over networks, as well as for efficient scaling to large compute clusters. As current approaches suffer from limited bandwidth of the network, we propose the use of communication compression in the decentralized training context. We show that Choco-SGD $-$ recently introduced and analyzed for strongly-convex objectives only $-$ converges under arbitrary high compression ratio on general non-convex functions at the rate $O\bigl(1/\sqrt{nT}\bigr)$ where $T$ denotes the number of iterations and $n$ the number of workers. The algorithm achieves linear speedup in the number of workers and supports higher compression than previous state-of-the art methods. We demonstrate the practical performance of the algorithm in two key scenarios: the training of deep learning models (i) over distributed user devices, connected by a social network and (ii) in a datacenter (outperforming all-reduce time-wise).

cs.LG↗

arXiv2020

Spin structure relation to phase contrast imaging of isolated magnetic Bloch and Neel skyrmions

Magnetic skyrmions are promising candidates for future storage devices with a large data density. A great variety of materials have been found that host skyrmions up to the room-temperature regime. Lorentz microscopy, usually performed in a transmission electron microscope (TEM), is one of the most important tools for characterizing skyrmion samples in real space. Using numerical calculations, this work relates the phase contrast in a TEM to the actual magnetization profile of an isolated Neel or Bloch skyrmion, the two most common skyrmion types. Within the framework of the used skyrmion model, the results are independent of skyrmion size and wall width and scale with sample thickness for purely magnetic specimens. Simple rules are provided to extract the actual skyrmion configuration of pure Bloch or Neel skyrmions without the need of simulations. Furthermore, first differential phase contrast (DPC) measurements on Neel skyrmions that meet experimental expectations are presented and showcase the described principles. The work is relevant for material sciences where it enables the engineering of skyrmion profiles via convenient characterization.

cond-mat.mes-hall↗