arXiv subjects

Han Zhang

Publications and source records attributed to Han Zhang.

At least 433 records · Page 24Linked to original sources

arXiv2020

Distilling Effective Supervision from Severe Label Noise

Collecting large-scale data with clean labels for supervised training of neural networks is practically challenging. Although noisy labels are usually cheap to acquire, existing methods suffer a lot from label noise. This paper targets at the challenge of robust training at high label noise regimes. The key insight to achieve this goal is to wisely leverage a small trusted set to estimate exemplar weights and pseudo labels for noisy data in order to reuse them for supervised training. We present a holistic framework to train deep neural networks in a way that is highly invulnerable to label noise. Our method sets the new state of the art on various types of label noise and achieves excellent performance on large-scale datasets with real-world label noise. For instance, on CIFAR100 with a $40\%$ uniform noise ratio and only 10 trusted labeled data per class, our method achieves $80.2{\pm}0.3\%$ classification accuracy, where the error rate is only $1.4\%$ higher than a neural network trained without label noise. Moreover, increasing the noise ratio to $80\%$, our method still maintains a high accuracy of $75.5{\pm}0.2\%$, compared to the previous best accuracy $48.2\%$. Source code available: https://github.com/google-research/google-research/tree/master/ieg

cs.LG↗

arXiv2020

ERNIE-GEN: An Enhanced Multi-Flow Pre-training and Fine-tuning Framework for Natural Language Generation

Current pre-training works in natural language generation pay little attention to the problem of exposure bias on downstream tasks. To address this issue, we propose an enhanced multi-flow sequence to sequence pre-training and fine-tuning framework named ERNIE-GEN, which bridges the discrepancy between training and inference with an infilling generation mechanism and a noise-aware generation method. To make generation closer to human writing patterns, this framework introduces a span-by-span generation flow that trains the model to predict semantically-complete spans consecutively rather than predicting word by word. Unlike existing pre-training methods, ERNIE-GEN incorporates multi-granularity target sampling to construct pre-training data, which enhances the correlation between encoder and decoder. Experimental results demonstrate that ERNIE-GEN achieves state-of-the-art results with a much smaller amount of pre-training data and parameters on a range of language generation tasks, including abstractive summarization (Gigaword and CNN/DailyMail), question generation (SQuAD), dialogue generation (Persona-Chat) and generative question answering (CoQA).

cs.CL↗

arXiv2020

Towards Personalized and Semantic Retrieval: An End-to-End Solution for E-commerce Search via Embedding Learning

Nowadays e-commerce search has become an integral part of many people's shopping routines. Two critical challenges stay in today's e-commerce search: how to retrieve items that are semantically relevant but not exact matching to query terms, and how to retrieve items that are more personalized to different users for the same search query. In this paper, we present a novel approach called DPSR, which stands for Deep Personalized and Semantic Retrieval, to tackle this problem. Explicitly, we share our design decisions on how to architect a retrieval system so as to serve industry-scale traffic efficiently and how to train a model so as to learn query and item semantics accurately. Based on offline evaluations and online A/B test with live traffics, we show that DPSR model outperforms existing models, and DPSR system can retrieve more personalized and semantically relevant items to significantly improve users' search experience by +1.29% conversion rate, especially for long tail queries by +10.03%. As a result, our DPSR system has been successfully deployed into JD.com's search production since 2019.

cs.IR↗

arXiv2020

Observation of Two-dimensional Superconductivity in an Ultrathin Iron-Arsenic Superconductor

Two-dimensional (2D) superconductors supply important platforms for exploring new quantum physics and high-$T_c$ superconductivity. The intrinsic superconducting properties in the 2D iron-arsenic superconductors are still unknown owing to the difficulties in the preparation of ultrathin samples. Here we report the fabrication and physical investigations of the high quality single-crystalline ultrathin films of the iron-arsenic superconductor KCa$_2$Fe$_4$As$_4$F$_2$. For the sample with the thickness of 2.6$\sim$5 nm (1$\sim$2 unit cells), a sharp superconducting transition at around 30 K (onset point) is observed. Compare with the bulk material, the ultrathin sample reveals a relatively lower $T_c$, wider transition width, and broader flux liquid region under the in-plane field. Moreover, the angle dependent upper critical field follows the Tinkham model, demonstrating the two-dimensional superconductivity in ultrathin KCa$_2$Fe$_4$As$_4$F$_2$.

cond-mat.supr-con↗

arXiv2020

Image Augmentations for GAN Training

Data augmentations have been widely studied to improve the accuracy and robustness of classifiers. However, the potential of image augmentation in improving GAN models for image synthesis has not been thoroughly investigated in previous studies. In this work, we systematically study the effectiveness of various existing augmentation techniques for GAN training in a variety of settings. We provide insights and guidelines on how to augment images for both vanilla GANs and GANs with regularizations, improving the fidelity of the generated images substantially. Surprisingly, we find that vanilla GANs attain generation quality on par with recent state-of-the-art results if we use augmentations on both real and generated images. When this GAN training is combined with other augmentation-based regularization techniques, such as contrastive loss and consistency regularization, the augmentations further improve the quality of generated images. We provide new state-of-the-art results for conditional generation on CIFAR-10 with both consistency loss and contrastive loss as additional regularizations.

cs.LG↗

arXiv2020

Distributed Equivalent Substitution Training for Large-Scale Recommender Systems

We present Distributed Equivalent Substitution (DES) training, a novel distributed training framework for large-scale recommender systems with dynamic sparse features. DES introduces fully synchronous training to large-scale recommendation system for the first time by reducing communication, thus making the training of commercial recommender systems converge faster and reach better CTR. DES requires much less communication by substituting the weights-rich operators with the computationally equivalent sub-operators and aggregating partial results instead of transmitting the huge sparse weights directly through the network. Due to the use of synchronous training on large-scale Deep Learning Recommendation Models (DLRMs), DES achieves higher AUC(Area Under ROC). We successfully apply DES training on multiple popular DLRMs of industrial scenarios. Experiments show that our implementation outperforms the state-of-the-art PS-based training framework, achieving up to 68.7% communication savings and higher throughput compared to other PS-based recommender systems.

cs.LG↗

arXiv2020

Anomalous Magnetoresistance in Centrosymmetric Skyrmion-Lattice Magnet Gd2PdSi3

We performed a systematic study of the temperature- and field-dependence of magnetization and resistivity of Gd2PdSi3, which is a centrosymmetric skyrmion crystal. While the magnetization behavior is consistent with the reported phase diagram based on susceptibility, we show that a phase diagram can also be constructed based on the anomalous magnetoresistance with one-to-one correspondence among all the features. In addition, the crossover boundary into the field-induced ferromagnetic state is also identified. Our results suggest that the ferromagnetic spin fluctuations above the Néel temperature play a key role in the high sensitivity of the resistivity anomalies to magnetic field, pointing to the rich interplay of different magnetic correlations at zero and finite wave vectors underlying the skyrmion lattice in this frustrated itinerant magnet.

cond-mat.str-el↗

arXiv2020

Dynamics of entanglement in the one-dimensional anisotropic XXZ model

The dynamics of entanglement in the one-dimensional spin-1/2 anisotropic XXZ model is studied using the quantum renormalization-group method. We obtain the analytical expression of the concurrence, for two different quenching methods, it is found that initial state plays a key role in the evolution of system entanglement, i.e., the system returns completely to the initial state every other period. Our computations and analysis indicate that the first derivative of the characteristic time at which the concurrence reaches its maximum or minimum with respect to the anisotropic parameter occurs nonanalytic behaviors at the quantum critical point. Interestingly, the minimum value of the first derivative of the characteristic time versus the size of the system exhibits the scaling behavior which is the same as the scaling behavior of the system ground-state entanglement in equilibrium. In particular, the scaling behavior near the critical point is independent of the initial state.

cond-mat.stat-mech↗

arXiv2020

Theoretical investigation on magnetic property of monolayer CrI3 from microscale to macroscale

Magnetic two-dimensional (2D) materials have received tremendous attention recently due to its potential application in spintronics and other magnetism related fields. To our knowledge, five kinds of 2D materials with intrinsic magnetism have been synthesized in experiment. They are CrI3, Cr2Ge2Te6, FePS3, Fe3GeTe2 and VSe2. Apart from the above intrinsic magnetic 2D materials, many strategies have also been proposed to induce magnetism in normal 2D materials such as atomic modification, spin valve and proximity effect. Various devices have also been designed to fulfill the basic functions of spintronics: inducing spin, manipulating spin and detecting spin.

physics.comp-ph↗

arXiv2020

Multicolor Graphdiyne Random Lasers

By breaking the restriction of mirrors, random lasers from a disordered medium have found unique applications spanning from displays, spectroscopy, biomedical treatments, to Li-Fi.Gain media in the form of two-dimension with distinct physical and chemical properties may lead to the next-generation of random lasers. Graphdiyne, a 2D graphene allotrope with intrigued carbon hybridization, atomic lattice, and optoelectronic properties, has attracted increasing attention recently. Herein, the photon emission characteristics and photo-carrier dynamics in graphdiyne are systematically studied, and the multicolor random lasers have been unprecedently realized using graphdiyne nanosheets as the gain. Considering the well bio-compatibility of graphdiyne, these results may look ahead a plethora of potential applications in the nanotechnology platform based on graphdiyne.

physics.optics↗

arXiv2020

Approximation Capabilities of Neural ODEs and Invertible Residual Networks

Neural ODEs and i-ResNet are recently proposed methods for enforcing invertibility of residual neural models. Having a generic technique for constructing invertible models can open new avenues for advances in learning systems, but so far the question of whether Neural ODEs and i-ResNets can model any continuous invertible function remained unresolved. Here, we show that both of these models are limited in their approximation capabilities. We then prove that any homeomorphism on a $p$-dimensional Euclidean space can be approximated by a Neural ODE operating on a $2p$-dimensional Euclidean space, and a similar result for i-ResNets. We conclude by showing that capping a Neural ODE or an i-ResNet with a single linear layer is sufficient to turn the model into a universal approximator for non-invertible continuous functions.

cs.LG↗

arXiv2020

Consistency Regularization for Generative Adversarial Networks

Generative Adversarial Networks (GANs) are known to be difficult to train, despite considerable research effort. Several regularization techniques for stabilizing training have been proposed, but they introduce non-trivial computational overheads and interact poorly with existing techniques like spectral normalization. In this work, we propose a simple, effective training stabilizer based on the notion of consistency regularization---a popular technique in the semi-supervised learning literature. In particular, we augment data passing into the GAN discriminator and penalize the sensitivity of the discriminator to these augmentations. We conduct a series of experiments to demonstrate that consistency regularization works effectively with spectral normalization and various GAN architectures, loss functions and optimizer settings. Our method achieves the best FID scores for unconditional image generation compared to other regularization methods on CIFAR-10 and CelebA. Moreover, Our consistency regularized GAN (CR-GAN) improves state-of-the-art FID scores for conditional generation from 14.73 to 11.48 on CIFAR-10 and from 8.73 to 6.66 on ImageNet-2012.

cs.LG↗

arXiv2020

ReMixMatch: Semi-Supervised Learning with Distribution Alignment and Augmentation Anchoring

We improve the recently-proposed "MixMatch" semi-supervised learning algorithm by introducing two new techniques: distribution alignment and augmentation anchoring. Distribution alignment encourages the marginal distribution of predictions on unlabeled data to be close to the marginal distribution of ground-truth labels. Augmentation anchoring feeds multiple strongly augmented versions of an input into the model and encourages each output to be close to the prediction for a weakly-augmented version of the same input. To produce strong augmentations, we propose a variant of AutoAugment which learns the augmentation policy while the model is being trained. Our new algorithm, dubbed ReMixMatch, is significantly more data-efficient than prior work, requiring between $5\times$ and $16\times$ less data to reach the same accuracy. For example, on CIFAR-10 with 250 labeled examples we reach $93.73\%$ accuracy (compared to MixMatch's accuracy of $93.58\%$ with $4{,}000$ examples) and a median accuracy of $84.92\%$ with just four labels per class. We make our code and data open-source at https://github.com/google-research/remixmatch.

cs.LG↗

arXiv2020

Experimental self-characterization of quantum measurements

The accurate and reliable description of measurement devices is a central problem in both observing uniquely non-classical behaviors and realizing quantum technologies from powerful computing to precision metrology. To date quantum tomography is the prevalent tool to characterize quantum detectors. However, such a characterization relies on accurately characterized probe states, rendering reliability of the characterization lost in circular argument. Here we report a self-characterization method of quantum measurements based on reconstructing the response range, the entirety of attainable measurement outcomes, eliminating the reliance on known states. We characterize two representative measurements implemented with photonic setups and obtain fidelities above 99.99% with the conventional tomographic reconstructions. This initiates range-based techniques in characterizing quantum systems and foreshadows novel device-independent protocols of quantum information applications.

quant-ph↗

arXiv2019

Your Local GAN: Designing Two Dimensional Local Attention Mechanisms for Generative Models

We introduce a new local sparse attention layer that preserves two-dimensional geometry and locality. We show that by just replacing the dense attention layer of SAGAN with our construction, we obtain very significant FID, Inception score and pure visual improvements. FID score is improved from $18.65$ to $15.94$ on ImageNet, keeping all other parameters the same. The sparse attention patterns that we propose for our new layer are designed using a novel information theoretic criterion that uses information flow graphs. We also present a novel way to invert Generative Adversarial Networks with attention. Our method extracts from the attention layer of the discriminator a saliency map, which we use to construct a new loss function for the inversion. This allows us to visualize the newly introduced attention heads and show that they indeed capture interesting aspects of two-dimensional geometry of real images.

cs.LG↗

arXiv2019

MANELA: A Multi-Agent Algorithm for Learning Network Embeddings

Playing an essential role in data mining, machine learning has a long history of being applied to networks on multifarious tasks and has played an essential role in data mining. However, the discrete and sparse natures of networks often render it difficult to apply machine learning directly to networks. To circumvent this difficulty, one major school of thought to approach networks using machine learning is via network embeddings. On the one hand, this network embeddings have achieved huge success on aggregated network data in recent years. On the other hand, learning network embeddings on distributively stored networks still remained understudied: To the best of our knowledge, all existing algorithms for learning network embeddings have hitherto been exclusively centralized and thus cannot be applied to these networks. To accommodate distributively stored networks, in this paper, we proposed a multi-agent model. Under this model, we developed the multi-agent network embedding learning algorithm (MANELA) for learning network embeddings. We demonstrate MANELA's advantages over other existing centralized network embedding learning algorithms both theoretically and experimentally. Finally, we further our understanding in MANELA via visualization and exploration of its relationship to DeepWalk.

cs.MA↗

arXiv2019

Possible Scale Invariant Linear Magnetoresistance in Pyrochlore Iridates Bi$_2$Ir$_2$O$_7$

We report the observation of a linear magnetoresistance in single crystals and epitaxial thin films of the pyrochlore iridate Bi$_2$Ir$_2$O$_7$. The linear magnetoresistance is positive and isotropic at low temperatures, without any sign of saturation up to 35 Tesla. As temperature increases, the linear field dependence gradually evolves to a quadratic field dependence. The temperature and field dependence of magnetoresistance of Bi$_2$Ir$_2$O$_7$ bears strikingly resemblance to the scale invariant magnetoresistance observed in the strange metal phase in high Tc cuprates. However, the residual resistivity of Bi$_2$Ir$_2$O$_7$ is more than two orders of magnitude higher than the curpates. Our results suggest that the correlation between linear magnetoresistance and quantum fluctuations may exist beyond high temperature superconductors.

cond-mat.str-el↗

arXiv2019

Small-GAN: Speeding Up GAN Training Using Core-sets

Recent work by Brock et al. (2018) suggests that Generative Adversarial Networks (GANs) benefit disproportionately from large mini-batch sizes. Unfortunately, using large batches is slow and expensive on conventional hardware. Thus, it would be nice if we could generate batches that were effectively large though actually small. In this work, we propose a method to do this, inspired by the use of Coreset-selection in active learning. When training a GAN, we draw a large batch of samples from the prior and then compress that batch using Coreset-selection. To create effectively large batches of 'real' images, we create a cached dataset of Inception activations of each training image, randomly project them down to a smaller dimension, and then use Coreset-selection on those projected activations at training time. We conduct experiments showing that this technique substantially reduces training time and memory usage for modern GAN variants, that it reduces the fraction of dropped modes in a synthetic dataset, and that it allows GANs to reach a new state of the art in anomaly detection.

stat.ML↗