arXiv subjects

Jian Zhang

Publications and source records attributed to Jian Zhang.

At least 613 records · Page 34Linked to original sources

arXiv2020

PipeMare: Asynchronous Pipeline Parallel DNN Training

Pipeline parallelism (PP) when training neural networks enables larger models to be partitioned spatially, leading to both lower network communication and overall higher hardware utilization. Unfortunately, to preserve the statistical efficiency of sequential training, existing PP techniques sacrifice hardware efficiency by decreasing pipeline utilization or incurring extra memory costs. In this paper, we investigate to what extent these sacrifices are necessary. We devise PipeMare, a simple yet robust training method that tolerates asynchronous updates during PP execution without sacrificing utilization or memory, which allows efficient use of fine-grained pipeline parallelism. Concretely, when tested on ResNet and Transformer networks, asynchrony enables PipeMare to use up to $2.7\times$ less memory or get $4.3\times$ higher pipeline utilization, with similar model quality, when compared to state-of-the-art synchronous PP training techniques.

cs.DC↗

arXiv2020

Weyl Superconductivity in UTe2

The search for a material platform for topological quantum computation has recently focused on unconventional superconductors. Such material systems, where the superconducting order parameter breaks a symmetry of the crystal point group, are capable of hosting novel phenomena, including emergent Majorana quasiparticles. Unique among unconventional superconductors is the recently discovered UTe2, where spin-triplet superconductivity emerges from a paramagnetic normal state. Although UTe2 could be considered a relative of a family of known ferromagnetic superconductors, the unique crystal structure of this material and experimentally suggested zero Curie temperature pose a great challenge to determining the symmetries, magnetism, and topology underlying the superconducting state. These emergent properties will determine the utility of UTe2 for future spintronics and quantum information applications. Here, we report observations of a non-zero polar Kerr effect and of two transitions in the specific heat upon entering the superconducting state, which together show that the superconductivity in UTe2 is characterized by an order parameter with two components that breaks time reversal symmetry. These data allow us to place firm constraints on the symmetries of the order parameter, which strongly suggest that UTe2 is a Weyl superconductor that hosts chiral Fermi arc surface states.

cond-mat.str-el↗

arXiv2020

On the Downstream Performance of Compressed Word Embeddings

Compressing word embeddings is important for deploying NLP models in memory-constrained settings. However, understanding what makes compressed embeddings perform well on downstream tasks is challenging---existing measures of compression quality often fail to distinguish between embeddings that perform well and those that do not. We thus propose the eigenspace overlap score as a new measure. We relate the eigenspace overlap score to downstream performance by developing generalization bounds for the compressed embeddings in terms of this score, in the context of linear and logistic regression. We then show that we can lower bound the eigenspace overlap score for a simple uniform quantization compression method, helping to explain the strong empirical performance of this method. Finally, we show that by using the eigenspace overlap score as a selection criterion between embeddings drawn from a representative set we compressed, we can efficiently identify the better performing embedding with up to $2\times$ lower selection error rates than the next best measure of compression quality, and avoid the cost of training a model for each task of interest.

cs.LG↗

arXiv2019

PMC-GANs: Generating Multi-Scale High-Quality Pedestrian with Multimodal Cascaded GANs

Recently, generative adversarial networks (GANs) have shown great advantages in synthesizing images, leading to a boost of explorations of using faked images to augment data. This paper proposes a multimodal cascaded generative adversarial networks (PMC-GANs) to generate realistic and diversified pedestrian images and augment pedestrian detection data. The generator of our model applies a residual U-net structure, with multi-scale residual blocks to encode features, and attention residual blocks to help decode and rebuild pedestrian images. The model constructs in a coarse-to-fine fashion and adopts cascade structure, which is beneficial to produce high-resolution pedestrians. PMC-GANs outperforms baselines, and when used for data augmentation, it improves pedestrian detection results.

cs.CV↗

arXiv2019

Adversarial AutoAugment

Data augmentation (DA) has been widely utilized to improve generalization in training deep neural networks. Recently, human-designed data augmentation has been gradually replaced by automatically learned augmentation policy. Through finding the best policy in well-designed search space of data augmentation, AutoAugment can significantly improve validation accuracy on image classification tasks. However, this approach is not computationally practical for large-scale problems. In this paper, we develop an adversarial method to arrive at a computationally-affordable solution called Adversarial AutoAugment, which can simultaneously optimize target related object and augmentation policy search loss. The augmentation policy network attempts to increase the training loss of a target network through generating adversarial augmentation policies, while the target network can learn more robust features from harder examples to improve the generalization. In contrast to prior work, we reuse the computation in target network training for policy evaluation, and dispense with the retraining of the target network. Compared to AutoAugment, this leads to about 12x reduction in computing cost and 11x shortening in time overhead on ImageNet. We show experimental results of our approach on CIFAR-10/CIFAR-100, ImageNet, and demonstrate significant performance improvements over state-of-the-art. On CIFAR-10, we achieve a top-1 test error of 1.36%, which is the currently best performing single model. On ImageNet, we achieve a leading performance of top-1 accuracy 79.40% on ResNet-50 and 80.00% on ResNet-50-D without extra data.

cs.CV↗

arXiv2019

Relational Mimic for Visual Adversarial Imitation Learning

In this work, we introduce a new method for imitation learning from video demonstrations. Our method, Relational Mimic (RM), improves on previous visual imitation learning methods by combining generative adversarial networks and relational learning. RM is flexible and can be used in conjunction with other recent advances in generative adversarial imitation learning to better address the need for more robust and sample-efficient approaches. In addition, we introduce a new neural network architecture that improves upon the previous state-of-the-art in reinforcement learning and illustrate how increasing the relational reasoning capabilities of the agent enables the latter to achieve increasingly higher performance in a challenging locomotion task with pixel inputs. Finally, we study the effects and contributions of relational learning in policy evaluation, policy improvement and reward learning through ablation studies.

cs.LG↗

arXiv2019

Potential Passenger Flow Prediction: A Novel Study for Urban Transportation Development

Recently, practical applications for passenger flow prediction have brought many benefits to urban transportation development. With the development of urbanization, a real-world demand from transportation managers is to construct a new metro station in one city area that never planned before. Authorities are interested in the picture of the future volume of commuters before constructing a new station, and estimate how would it affect other areas. In this paper, this specific problem is termed as potential passenger flow (PPF) prediction, which is a novel and important study connected with urban computing and intelligent transportation systems. For example, an accurate PPF predictor can provide invaluable knowledge to designers, such as the advice of station scales and influences on other areas, etc. To address this problem, we propose a multi-view localized correlation learning method. The core idea of our strategy is to learn the passenger flow correlations between the target areas and their localized areas with adaptive-weight. To improve the prediction accuracy, other domain knowledge is involved via a multi-view learning process. We conduct intensive experiments to evaluate the effectiveness of our method with real-world official transportation datasets. The results demonstrate that our method can achieve excellent performance compared with other available baselines. Besides, our method can provide an effective solution to the cold-start problem in the recommender system as well, which proved by its outperformed experimental results.

cs.LG↗

arXiv2019

Adversarial Domain Adaptation with Domain Mixup

Recent works on domain adaptation reveal the effectiveness of adversarial learning on filling the discrepancy between source and target domains. However, two common limitations exist in current adversarial-learning-based methods. First, samples from two domains alone are not sufficient to ensure domain-invariance at most part of latent space. Second, the domain discriminator involved in these methods can only judge real or fake with the guidance of hard label, while it is more reasonable to use soft scores to evaluate the generated images or features, i.e., to fully utilize the inter-domain information. In this paper, we present adversarial domain adaptation with domain mixup (DM-ADA), which guarantees domain-invariance in a more continuous latent space and guides the domain discriminator in judging samples' difference relative to source and target domains. Domain mixup is jointly conducted on pixel and feature level to improve the robustness of models. Extensive experiments prove that the proposed approach can achieve superior performance on tasks with various degrees of domain shift and data complexity.

cs.CV↗

arXiv2019

Analysis of DAWNBench, a Time-to-Accuracy Machine Learning Performance Benchmark

Researchers have proposed hardware, software, and algorithmic optimizations to improve the computational performance of deep learning. While some of these optimizations perform the same operations faster (e.g., increasing GPU clock speed), many others modify the semantics of the training procedure (e.g., reduced precision), and can impact the final model's accuracy on unseen data. Due to a lack of standard evaluation criteria that considers these trade-offs, it is difficult to directly compare these optimizations. To address this problem, we recently introduced DAWNBench, a benchmark competition focused on end-to-end training time to achieve near-state-of-the-art accuracy on an unseen dataset---a combined metric called time-to-accuracy (TTA). In this work, we analyze the entries from DAWNBench, which received optimized submissions from multiple industrial groups, to investigate the behavior of TTA as a metric as well as trends in the best-performing entries. We show that TTA has a low coefficient of variation and that models optimized for TTA generalize nearly as well as those trained using standard methods. Additionally, even though DAWNBench entries were able to train ImageNet models in under 3 minutes, we find they still underutilize hardware capabilities such as Tensor Cores. Furthermore, we find that distributed entries can spend more than half of their time on communication. We show similar findings with entries to the MLPERF v0.5 benchmark.

cs.LG↗

arXiv2019

Worst Cases Policy Gradients

Recent advances in deep reinforcement learning have demonstrated the capability of learning complex control policies from many types of environments. When learning policies for safety-critical applications, it is essential to be sensitive to risks and avoid catastrophic events. Towards this goal, we propose an actor-critic framework that models the uncertainty of the future and simultaneously learns a policy based on that uncertainty model. Specifically, given a distribution of the future return for any state and action, we optimize policies for varying levels of conditional Value-at-Risk. The learned policy can map the same state to different actions depending on the propensity for risk. We demonstrate the effectiveness of our approach in the domain of driving simulations, where we learn maneuvers in two scenarios. Our learned controller can dynamically select actions along a continuous axis, where safe and conservative behaviors are found at one end while riskier behaviors are found at the other. Finally, when testing with very different simulation parameters, our risk-averse policies generalize significantly better compared to other reinforcement learning approaches.

cs.LG↗

arXiv2019

Unsupervised Euclidean Distance Attack on Network Embedding

Considering the wide application of network embedding methods in graph data mining, inspired by the adversarial attack in deep learning, this paper proposes a Genetic Algorithm (GA) based Euclidean Distance Attack strategy (EDA) to attack the network embedding, so as to prevent certain structural information from being discovered. EDA focuses on disturbing the Euclidean distance between a pair of nodes in the embedding space as much as possible through minimal modifications of the network structure. Since a large number of downstream network algorithms, such as community detection and node classification, rely on the Euclidean distance between nodes to evaluate the similarity between them in the embedding space, EDA can be considered as a universal attack on a variety of network algorithms. Different from traditional supervised attack strategies, EDA does not need labeling information, and, in other words, is an unsupervised network embedding attack method.

cs.SI↗

arXiv2019

Cooperative Rate-Splitting for MISO Broadcast Channel with User Relaying, and Performance Benefits over Cooperative NOMA

Due to its promising performance in a wide range of practical scenarios, Rate-Splitting (RS) has recently received significant attention in academia for the downlink of communication systems. In this letter, we propose and analyse a Cooperative Rate-Splitting (CRS) strategy based on the three-node relay channel where the transmitter is equipped with multiple antennas. By splitting user messages and linearly precoding common and private streams at the transmitter, and opportunistically asking the relaying user to forward its decoded common message, CRS can efficiently cope with a wide range of propagation conditions (disparity of user channel strengths and directions) and compensate for the performance degradation due to deep fading. The precoder design and the resource allocation are optimized by solving the Weighted Sum Rate (WSR) maximization problem. Numerical results demonstrate that our proposed CRS scheme can achieve an explicit rate region improvement compared to its non-cooperative counterpart and other cooperative strategies (such as cooperative NOMA).

cs.IT↗

arXiv2019

Muon spin rotation and relaxation in Pr$_{1-x}$Nd$_x$Os$_4$Sb$_{12}$: superconductivity and magnetism in Pr-rich alloys

The Pr-rich end of the alloy series Pr$_{1-x}$Nd$_x$Os$_4$Sb$_{12}$ has been studied using muon spin rotation and relaxation. The end compound PrOs$_4$Sb$_{12}$ is an unconventional heavy-fermion superconductor, which exhibits a spontaneous magnetic field in the superconducting phase associated with broken time-reversal symmetry. No spontaneous field is observed in the Nd-doped alloys for x $>$ 0.05. The superfluid density is insensitive to Nd concentration, and no Nd$^{3+}$ static magnetism is found down to the lowest temperatures of measurement. Together with the slow suppression of the superconducting transition temperature with Nd doping, these results suggest anomalously weak coupling between Nd spins and conduction-band states.

cond-mat.supr-con↗

arXiv2019

C-RPNs: Promoting Object Detection in real world via a Cascade Structure of Region Proposal Networks

Recently, significant progresses have been made in object detection on common benchmarks (i.e., Pascal VOC). However, object detection in real world is still challenging due to the serious data imbalance. Images in real world are dominated by easy samples like the wide range of background and some easily recognizable objects, for example. Although two-stage detectors like Faster R-CNN achieved big successes in object detection due to the strategy of extracting region proposals by region proposal network, they show their poor adaption in real-world object detection as a result of without considering mining hard samples during extracting region proposals. To address this issue, we propose a Cascade framework of Region Proposal Networks, referred to as C-RPNs. The essence of C-RPNs is adopting multiple stages to mine hard samples while extracting region proposals and learn stronger classifiers. Meanwhile, a feature chain and a score chain are proposed to help learning more discriminative representations for proposals. Moreover, a loss function of cascade stages is designed to train cascade classifiers through backpropagation. Our proposed method has been evaluated on Pascal VOC and several challenging datasets like BSBDV 2017, CityPersons, etc. Our method achieves competitive results compared with the current state-of-the-arts and all-sided improvements in error analysis, validating its efficacy for detection in real world.

cs.CV↗

arXiv2019

Exploiting Channel Similarity for Accelerating Deep Convolutional Neural Networks

To address the limitations of existing magnitude-based pruning algorithms in cases where model weights or activations are of large and similar magnitude, we propose a novel perspective to discover parameter redundancy among channels and accelerate deep CNNs via channel pruning. Precisely, we argue that channels revealing similar feature information have functional overlap and that most channels within each such similarity group can be removed without compromising model's representational power. After deriving an effective metric for evaluating channel similarity through probabilistic modeling, we introduce a pruning algorithm via hierarchical clustering of channels. In particular, the proposed algorithm does not rely on sparsity training techniques or complex data-driven optimization and can be directly applied to pre-trained models. Extensive experiments on benchmark datasets strongly demonstrate the superior acceleration performance of our approach over prior arts. On ImageNet, our pruned ResNet-50 with 30% FLOPs reduced outperforms the baseline model.

cs.LG↗

arXiv2019

Orbital Stability of Standing Waves for Fractional Hartree Equation with Unbounded Potentials

We prove the existence of the set of ground states in a suitable energy space $Σ^s=\{u: \int_{\mathbb{R}^N} \bar{u}(-Δ+m^2)^s u+V |u|^2<\infty\}$, $s\in (0,\frac{N}{2})$ for the mass-subcritical nonlinear fractional Hartree equation with unbounded potentials. As a consequence we obtain, as a priori result, the orbital stability of the set of standing waves. The main ingredient is the observation that $Σ^s$ is compactly embedded in $L^2$. This enables us to apply the concentration compactness argument in the works of Cazenave-Lions and Zhang, namely, relative compactness for any minimizing sequence in the energy space.

math.AP↗

arXiv2019

Charge-stripe order, antiferromagnetism, and spin dynamics in the cuprate-analog nickelate La$_4$Ni$_3$O$_8$

We report a muon spin rotation ($μ$SR) study of the cuprate-analog nickelate La$_4$Ni$_3$O$_8$, which undergoes a transition at 105 K to a low-temperature phase with charge-stripe and antiferromagnetic (AFM) order on square planar NiO$_2$ layers. Zero-field $μ$SR shows that this transition is abrupt and that the AFM configuration is commensurate. Comparison of observed muon precession frequencies with Ni dipolar field calculations yields Ni moments $\lesssim 0.5μ_B$. Dynamic muon spin relaxation above 105 K suggests critical slowing of Ni spin fluctuations, but is inconsistent with corresponding $^{139}$La NMR results. Critical slowing and an abrupt transition are also observed in the planar cuprate AFM La$_2$CuO$_{4+δ}$, where they are evidence for weakly interplanar-coupled two-dimensional AFM spin fluctuations, but our $μ$SR data do not agree quantitatively with theoretical predictions for this scenario.

cond-mat.mtrl-sci↗

arXiv2019

Sample Adaptive Multiple Kernel Learning for Failure Prediction of Railway Points

Railway points are among the key components of railway infrastructure. As a part of signal equipment, points control the routes of trains at railway junctions, having a significant impact on the reliability, capacity, and punctuality of rail transport. Traditionally, maintenance of points is based on a fixed time interval or raised after the equipment failures. Instead, it would be of great value if we could forecast points' failures and take action beforehand, minimising any negative effect. To date, most of the existing prediction methods are either lab-based or relying on specially installed sensors which makes them infeasible for large-scale implementation. Besides, they often use data from only one source. We, therefore, explore a new way that integrates multi-source data which are ready to hand to fulfil this task. We conducted our case study based on Sydney Trains rail network which is an extensive network of passenger and freight railways. Unfortunately, the real-world data are usually incomplete due to various reasons, e.g., faults in the database, operational errors or transmission faults. Besides, railway points differ in their locations, types and some other properties, which means it is hard to use a unified model to predict their failures. Aiming at this challenging task, we firstly constructed a dataset from multiple sources and selected key features with the help of domain experts. In this paper, we formulate our prediction task as a multiple kernel learning problem with missing kernels. We present a robust multiple kernel learning algorithm for predicting points failures. Our model takes into account the missing pattern of data as well as the inherent variance on different sets of railway points. Extensive experiments demonstrate the superiority of our algorithm compared with other state-of-the-art methods.

cs.LG↗