arXiv subjects

Wei Han

Publications and source records attributed to Wei Han.

At least 127 records · Page 7Linked to original sources

arXiv2022

BigSSL: Exploring the Frontier of Large-Scale Semi-Supervised Learning for Automatic Speech Recognition

We summarize the results of a host of efforts using giant automatic speech recognition (ASR) models pre-trained using large, diverse unlabeled datasets containing approximately a million hours of audio. We find that the combination of pre-training, self-training and scaling up model size greatly increases data efficiency, even for extremely large tasks with tens of thousands of hours of labeled data. In particular, on an ASR task with 34k hours of labeled data, by fine-tuning an 8 billion parameter pre-trained Conformer model we can match state-of-the-art (SoTA) performance with only 3% of the training data and significantly improve SoTA with the full training set. We also report on the universal benefits gained from using big pre-trained and self-trained models for a large set of downstream tasks that cover a wide range of speech domains and span multiple orders of magnitudes of dataset sizes, including obtaining SoTA performance on many public benchmarks. In addition, we utilize the learned representation of pre-trained networks to achieve SoTA results on non-ASR tasks.

eess.AS↗

arXiv2022

Pushing the Limits of Semi-Supervised Learning for Automatic Speech Recognition

We employ a combination of recent developments in semi-supervised learning for automatic speech recognition to obtain state-of-the-art results on LibriSpeech utilizing the unlabeled audio of the Libri-Light dataset. More precisely, we carry out noisy student training with SpecAugment using giant Conformer models pre-trained using wav2vec 2.0 pre-training. By doing so, we are able to achieve word-error-rates (WERs) 1.4%/2.6% on the LibriSpeech test/test-other sets against the current state-of-the-art WERs 1.7%/3.3%.

eess.AS↗

arXiv2022

Simulation and detection of Weyl fermions in ultracold Fermi gases with Raman-assisted spin-orbit coupling

Weyl fermion, also referred to as pseudo-magnetic monopole in momentum space, is an undiscovered massless elementary particle with half-integer spin predicted according to relativistic quantum field theory. Motivated by the recent experimental observation of Weyl semimetal band in ultracold Bose gases with Raman-assisted 3D spin-orbit coupling, we investigate the properties and possible observation of Weyl fermions in the low-energy quasi-particle excitations of ultracold Fermi gases. Following a previous suggestion that the existing Raman lattice scheme can be readily generalized to fermionic systems, here we discuss the movement of the Weyl points in the Brillouin Zone, as well as the creation and annihilation of Weyl fermions by adjusting the effective Zeeman field. The relevant topological properties are also demonstrated by calculating the Chern number. Furthermore, we propose how to experimentally verify the existence of the Weyl fermions and the associated quantum phase transition via density profile measurements.

cond-mat.quant-gas↗

arXiv2022

Scaling Autoregressive Models for Content-Rich Text-to-Image Generation

We present the Pathways Autoregressive Text-to-Image (Parti) model, which generates high-fidelity photorealistic images and supports content-rich synthesis involving complex compositions and world knowledge. Parti treats text-to-image generation as a sequence-to-sequence modeling problem, akin to machine translation, with sequences of image tokens as the target outputs rather than text tokens in another language. This strategy can naturally tap into the rich body of prior work on large language models, which have seen continued advances in capabilities and performance through scaling data and model sizes. Our approach is simple: First, Parti uses a Transformer-based image tokenizer, ViT-VQGAN, to encode images as sequences of discrete tokens. Second, we achieve consistent quality improvements by scaling the encoder-decoder Transformer model up to 20B parameters, with a new state-of-the-art zero-shot FID score of 7.23 and finetuned FID score of 3.22 on MS-COCO. Our detailed analysis on Localized Narratives as well as PartiPrompts (P2), a new holistic benchmark of over 1600 English prompts, demonstrate the effectiveness of Parti across a wide variety of categories and difficulty aspects. We also explore and highlight limitations of our models in order to define and exemplify key areas of focus for further improvements. See https://parti.research.google/ for high-resolution images.

cs.CV↗

arXiv2022

Lossy Computing with Side Information via Multi-Hypergraphs

We consider a problem of coding for computing, where the decoder wishes to estimate a function of its local message and the source message at the encoder within a given distortion. We show that the rate-distortion function can be characterized through a characteristic multi-hypergraph, which simplifies the evaluation of the rate-distortion function.

cs.IT↗

arXiv2022

Accented Speech Recognition: Benchmarking, Pre-training, and Diverse Data

Building inclusive speech recognition systems is a crucial step towards developing technologies that speakers of all language varieties can use. Therefore, ASR systems must work for everybody independently of the way they speak. To accomplish this goal, there should be available data sets representing language varieties, and also an understanding of model configuration that is the most helpful in achieving robust understanding of all types of speech. However, there are not enough data sets for accented speech, and for the ones that are already available, more training approaches need to be explored to improve the quality of accented speech recognition. In this paper, we discuss recent progress towards developing more inclusive ASR systems, namely, the importance of building new data sets representing linguistic diversity, and exploring novel training approaches to improve performance for all users. We address recent directions within benchmarking ASR systems for accented speech, measure the effects of wav2vec 2.0 pre-training on accented speech recognition, and highlight corpora relevant for diverse ASR evaluations.

eess.AS↗

arXiv2022

Unsupervised Data Selection via Discrete Speech Representation for ASR

Self-supervised learning of speech representations has achieved impressive results in improving automatic speech recognition (ASR). In this paper, we show that data selection is important for self-supervised learning. We propose a simple and effective unsupervised data selection method which selects acoustically similar speech to a target domain. It takes the discrete speech representation available in common self-supervised learning frameworks as input, and applies a contrastive data selection method on the discrete tokens. Through extensive empirical studies we show that our proposed method reduces the amount of required pre-training data and improves the downstream ASR performance. Pre-training on a selected subset of 6% of the general data pool results in 11.8% relative improvements in LibriSpeech test-other compared to pre-training on the full set. On Multilingual LibriSpeech French, German, and Spanish test sets, selecting 6% data for pre-training reduces word error rate by more than 15% relatively compared to the full set, and achieves competitive results compared to current state-of-the-art performances.

eess.AS↗

arXiv2022

Spin Seebeck effect in quantum magnet Pb2V3O9

Spin Seebeck effect (SSE), the generation of spin current from heat, has been extensively studied in a large variety of magnetic materials, including ferromagnets, antiferromagnets, paramagnets, and quantum spin liquids. In this paper, we report the study of the SSE in the single crystalline Pb2V3O9, a spin-gapped quantum magnet candidate with quasi-one-dimensional spin-1/2 chain. Detailed temperature and magnetic field dependences of the SSE are investigated, and the temperature-dependent critical magnetic fields show a strong correlation to the Bose-Einstein condensation phase of the quantum magnet Pb2V3O9. This work shows the potential of using spin current as a probe to study the spin correlation and phase transition properties in quantum magnets.

cond-mat.mtrl-sci↗

arXiv2021

Co-training Transformer with Videos and Images Improves Action Recognition

In learning action recognition, models are typically pre-trained on object recognition with images, such as ImageNet, and later fine-tuned on target action recognition with videos. This approach has achieved good empirical performance especially with recent transformer-based video architectures. While recently many works aim to design more advanced transformer architectures for action recognition, less effort has been made on how to train video transformers. In this work, we explore several training paradigms and present two findings. First, video transformers benefit from joint training on diverse video datasets and label spaces (e.g., Kinetics is appearance-focused while SomethingSomething is motion-focused). Second, by further co-training with images (as single-frame videos), the video transformers learn even better video representations. We term this approach as Co-training Videos and Images for Action Recognition (CoVeR). In particular, when pretrained on ImageNet-21K based on the TimeSFormer architecture, CoVeR improves Kinetics-400 Top-1 Accuracy by 2.4%, Kinetics-600 by 2.3%, and SomethingSomething-v2 by 2.3%. When pretrained on larger-scale image datasets following previous state-of-the-art, CoVeR achieves best results on Kinetics-400 (87.2%), Kinetics-600 (87.9%), Kinetics-700 (79.8%), SomethingSomething-v2 (70.9%), and Moments-in-Time (46.1%), with a simple spatio-temporal video transformer.

cs.CV↗

arXiv2021

A Tensor-BTD-based Modulation for Massive Unsourced Random Access

In this letter, we propose a novel tensor-based modulation scheme for massive unsourced random access. The proposed modulation can be deemed as a summation of third-order tensors, of which the factors are representatives of subspaces. A constellation design based on high-dimensional Grassmann manifold is presented for information encoding. The uniqueness of tensor decomposition provides theoretical guarantee for active user separation. Simulation results show that our proposed method outperforms the state-of-the-art tensor-based modulation.

eess.SP↗

arXiv2021

Harmonic Retrieval with $L_1$-Tucker Tensor Decomposition

Harmonic retrieval (HR) has a wide range of applications in the scenes where signals are modelled as a summation of sinusoids. Past works have developed a number of approaches to recover the original signals. Most of them rely on classical singular value decomposition, which are vulnerable to unexpected outliers. In this paper, we present new decomposition algorithms of third-order complex-valued tensors with $L_1$-principle component analysis ($L_1$-PCA) of complex data and apply them to a novel random access HR model in presence of outliers. We also develop a novel subcarrier recovery method for the proposed model. Simulations are designed to compare our proposed method with some existing tensor-based algorithms for HR. The results demonstrate the outlier-insensitivity of the proposed method.

eess.SP↗

arXiv2021

Structural Entropy of the Stochastic Block Models

With the rapid expansion of graphs and networks and the growing magnitude of data from all areas of science, effective treatment and compression schemes of context-dependent data is extremely desirable. A particularly interesting direction is to compress the data while keeping the "structural information" only and ignoring the concrete labelings. Under this direction, Choi and Szpankowski introduced the structures (unlabeled graphs) which allowed them to compute the structural entropy of the Erdős--Rényi random graph model. Moreover, they also provided an asymptotically optimal compression algorithm that (asymptotically) achieves this entropy limit and runs in expectation in linear time. In this paper, we consider the Stochastic Block Models with an arbitrary number of parts. Indeed, we define a partitioned structural entropy for Stochastic Block Models, which generalizes the structural entropy for unlabeled graphs and encodes the partition information as well. We then compute the partitioned structural entropy of the Stochastic Block Models, and provide a compression scheme that asymptotically achieves this entropy limit.

cs.IT↗

arXiv2021

Evidence for anisotropic spin-triplet Andreev reflection at the 2D van der Waals ferromagnet/superconductor interface

Fundamental symmetry breaking and relativistic spin-orbit coupling give rise to fascinating phenomena in quantum materials. Of particular interest are the interfaces between ferromagnets and common s-wave superconductors, where the emergent spin-orbit fields support elusive spin-triplet superconductivity, crucial for superconducting spintronics and topologically-protected Majorana bound states. Here, we report the observation of large magnetoresistances at the interface between a quasi-two-dimensional van der Waals ferromagnet Fe0.29TaS2 and a conventional s-wave superconductor NbN, which provides the possible experimental evidence for the spin triplet Andreev reflection and induced spin-triplet superconductivity at ferromagnet/superconductor interface arising from Rashba spin-orbit coupling. The temperature, voltage, and interfacial barrier dependences of the magnetoresistance further support the induced spin-triplet superconductivity and spin-triplet Andreev reflection. This discovery, together with the impressive advances in two-dimensional van der Waals ferromagnets, opens an important opportunity to design and probe superconducting interfaces with exotic properties.

cond-mat.supr-con↗

arXiv2021

Gate-tunability of the superconducting state at the $EuO/KTaO_3 (111)$ interface

The recent discovery of superconducting interfaces in the $KTaO_3 (111)$-based heterostructures is intriguing, since a much higher superconducting critical temperature (TC ~ 2 K) is achieved compared to that in the $SrTiO_3$ heterostructures (~ 300 mK). In this paper, we report the superconducting properties of $EuO/KTaO_3 (111)$ interface as a function of the interface carrier density ($n_s$). The maximum T_C is observed to be ~ 2 K at $n_s$ ~ $1\times10^{14}$ cm-2. In addition, we show that the critical current density and the upper critical magnetic field can be effectively tuned by the back gate voltage. Interestingly, the gate dependence of the upper critical magnetic field exhibits a trend opposite to that of TC in the underdoped region, suggesting a relatively larger Cooper pairing potential.

cond-mat.supr-con↗

arXiv2021

Nanomechanical probing and strain tuning of the Curie temperature in suspended Cr$_2$Ge$_2$Te$_6$ heterostructures

Two-dimensional (2D) magnetic materials with strong magnetostriction are interesting systems for strain-tuning the magnetization, enabling potential for realizing spintronic and nanomagnetic devices. Realizing this potential requires understanding of the magneto-mechanical coupling in the 2D limit. In this work, we suspend thin Cr$_2$Ge$_2$Te$_6$ layers, creating nanomechanical membrane resonators. We probe its mechanical and magnetic properties as a function of temperature and strain. Pronounced signatures of magneto-elastic coupling are observed in the temperature-dependent resonance frequency of these membranes near $T_{\rm C}$. We further utilize Cr$_2$Ge$_2$Te$_6$ in heterostructures with thin layers of WSe$_2$ and FePS$_3$, which have positive thermal expansion coefficients, to compensate the negative thermal expansion coefficient of Cr$_2$Ge$_2$Te$_6$ and quantitatively probe the corresponding $T_{\rm C}$. Finally, we induce a strain of $0.016\%$ in a suspended heterostructure via electrostatic force and demonstrate a resulting enhancement of $T_{\rm C}$ by $2.5 \pm 0.6$ K in the absence of an external magnetic field.

cond-mat.mes-hall↗

arXiv2021

Giant oscillatory Gilbert damping in superconductor/ferromagnet/superconductor junctions

Interfaces between materials with differently ordered phases present unique opportunities for exotic physical properties, especially the interplay between ferromagnetism and superconductivity in the ferromagnet/superconductor heterostructures. The investigation of zero- and pi-junctions has been of particular interest for both fundamental physical science and emerging technologies. Here, we report the experimental observation of giant oscillatory Gilbert damping in the superconducting Nb/NiFe/Nb junctions with respect to the NiFe thickness. This observation suggests an unconventional spin pumping and relaxation via zero-energy Andreev bound states that exist only in the Nb/NiFe/Nb pi-junctions, but not in the Nb/NiFe/Nb zero-junctions. Our findings could be important for further exploring the exotic physical properties of ferromagnet/superconductor heterostructures, and potential applications of ferromagnet pi-junctions in quantum computing, such as half-quantum flux qubits.

cond-mat.supr-con↗

arXiv2021

Algorithms for Interference Minimization in Future Wireless Network Decomposition

We propose a simple and fast method for providing a high quality solution for the sum-interference minimization problem. As future networks are deployed in high density urban areas, improved clustering methods are needed to provide low interference network connectivity. The proposed algorithm applies straightforward similarity based clustering and optionally stable matchings to outperform state of the art algorithms. The running times of our algorithms are dominated by one matrix multiplication.

cs.SI↗

arXiv2021

What Should Future Wireless Network Architectures Be?

The accelerated convergence of digital and real-world lifestyles has imposed unprecedented demands on today's wireless network architectures, as it is highly desirable for such architectures to support wireless devices everywhere with high capacity and minimal signaling overhead. Conventional architectures, such as cellular architectures, are not able to satisfy these requirements simultaneously, and are thus no longer suitable for the future era. In this paper, we propose a capacity-centric (C$^2$) architecture for future wireless communication networks. It is designed based on the principles of maximizing the number of non-overlapping clusters with the average cluster capacity guaranteed to be higher than a certain threshold, and thus provides a flexible way to balance the capacity requirement against the signaling overhead. Our analytical results reveal that C$^2$ has superior generality, wherein both the cellular and the fully coordinated architectures can be viewed as its extreme cases. Simulation results show that the average capacity of C$^2$ is at least three times higher compared to that of the cellular architecture. More importantly, different from the widely adopted conventional wisdom that base-station distributions dominate architecture designs, we find that the C$^2$ architecture is independent of base-station distributions, and instead user-side information should be the focus in future wireless network architecture designs.

cs.IT↗