Searcharxiv⌕ Search

arXiv subjects

Thomas Wolf

Publications and source records attributed to Thomas Wolf.

At least 73 records · Page 4Linked to original sources
arXiv2020

HuggingFace's Transformers: State-of-the-art Natural Language Processing

Recent progress in natural language processing has been driven by advances in both model architecture and model pretraining. Transformer architectures have facilitated building higher-capacity models and pretraining has made it possible to effectively utilize this capacity for a wide variety of tasks. \textit{Transformers} is an open-source library with the goal of opening up these advances to the wider machine learning community. The library consists of carefully engineered state-of-the art Transformer architectures under a unified API. Backing this library is a curated collection of pretrained models made by and available for the community. \textit{Transformers} is designed to be extensible by researchers, simple for practitioners, and fast and robust in industrial deployments. The library is available at \url{https://github.com/huggingface/transformers}.

cs.CL↗
arXiv2020

Ultrafast magnetic dynamics in insulating YBa$_2$Cu$_3$O$_{6+x}$ revealed by time resolved two-magnon Raman Scattering

Measurement and control of magnetic order and correlations in real time is a rapidly developing scientific area relevant for magnetic memory and spintronics [1,15]. In these experiments an ultra-short laser pulse (pump) is first absorbed by excitations carrying electric dipole moment. These then give their energy to the magnetic subsystem monitored by a time-resolved probe. A lot of progress has been made in investigations of ferromagnets but antiferromagnets are more challenging. Here we introduce time-resolved two-magnon Raman scattering as a novel real time probe of magnetic correlations especially well-suited for antiferromagnets. Its application to the antiferromagnetic charge transfer insulator YBa$_2$Cu$_3$O$_{6+x}$ revealed rapid demagnetization within 90fs of photoexcitation. The relaxation back to thermal equilibrium is characterized by much slower timescales. We interpret these results in terms of slow relaxation of the charge sector and rapid equilibration of the magnetic sector to a prethermal state characterized by parameters that change slowly as the charge sector relaxes.

cond-mat.str-el↗
arXiv2020

TLDR: Token Loss Dynamic Reweighting for Reducing Repetitive Utterance Generation

Natural Language Generation (NLG) models are prone to generating repetitive utterances. In this work, we study the repetition problem for encoder-decoder models, using both recurrent neural network (RNN) and transformer architectures. To this end, we consider the chit-chat task, where the problem is more prominent than in other tasks that need encoder-decoder architectures. We first study the influence of model architectures. By using pre-attention and highway connections for RNNs, we manage to achieve lower repetition rates. However, this method does not generalize to other models such as transformers. We hypothesize that the deeper reason is that in the training corpora, there are hard tokens that are more difficult for a generative model to learn than others and, once learning has finished, hard tokens are still under-learned, so that repetitive generations are more likely to happen. Based on this hypothesis, we propose token loss dynamic reweighting (TLDR) that applies differentiable weights to individual token losses. By using higher weights for hard tokens and lower weights for easy tokens, NLG models are able to learn individual tokens at different paces. Experiments on chit-chat benchmark datasets show that TLDR is more effective in repetition reduction for both RNN and transformer architectures than baselines using different weighting functions.

cs.CL↗
arXiv2020

DistilBERT, a distilled version of BERT: smaller, faster, cheaper and lighter

As Transfer Learning from large-scale pre-trained models becomes more prevalent in Natural Language Processing (NLP), operating these large models in on-the-edge and/or under constrained computational training or inference budgets remains challenging. In this work, we propose a method to pre-train a smaller general-purpose language representation model, called DistilBERT, which can then be fine-tuned with good performances on a wide range of tasks like its larger counterparts. While most prior work investigated the use of distillation for building task-specific models, we leverage knowledge distillation during the pre-training phase and show that it is possible to reduce the size of a BERT model by 40%, while retaining 97% of its language understanding capabilities and being 60% faster. To leverage the inductive biases learned by larger models during pre-training, we introduce a triple loss combining language modeling, distillation and cosine-distance losses. Our smaller, faster and lighter model is cheaper to pre-train and we demonstrate its capabilities for on-device computations in a proof-of-concept experiment and a comparative on-device study.

cs.CL↗
arXiv2019

Reflection anisotropy experiments to study non-equilibrium dynamics of electronic nematicity

Electronic nematicity have attracted a great deal of interest in the unconventional superconductivity research. We present an experimental method to investigate reflectivity anisotropy by the electronic nematicity and its non-equilibrium dynamics. Our experimental results on Ba(Fe$_{0.955}$Co$_{0.045}$)$_{2}$As$_{2}$ single crystal clearly feature the broken four-fold symmetry along directions parallel to Fe-Fe bonding. Numerical simulations using WVASE from Woollam Co. demonstrate that our method is highly reliable under various experimental conditions. Finally, we perform time-resolved reflectivity anisotropy measurement confirming that photo-excitation by 1.55 eV photons suppresses the reflectivity anisotropy due to nematic order.

physics.optics↗
arXiv2019

Orbitally resolved superconductivity in real space: FeSe

Multi-orbital superconductors combine unconventional pairing with complex band structures, where different orbitals in the bands contribute to a multitude of superconducting gaps. We here demonstrate a fresh approach using low-temperature scanning tunneling microscopy (LT-STM) to resolve the contributions of different orbitals to superconductivity. This approach is based on STM's capability to resolve the local density of states (LDOS) with a combined high energy and sub unit-cell resolution. This technique directly determines the orbitals on defect free crystals without the need for scatters on the surface and sophisticated quasi-particle interference (QPI) measurements. Taking bulk FeSe as an example, we directly resolve the superconducting gaps within the units cell using a 30 mK STM. In combination with density functional theory calculations, we are able to identify the orbital character of each gap.

cond-mat.supr-con↗
arXiv2019

Frustrated spin order and stripe fluctuations in FeSe

The charge and spin dynamics of the structurally simplest iron-based superconductor, FeSe, may hold the key to understanding the physics of high temperature superconductors in general. Unlike the iron pnictides, FeSe lacks long range magnetic order in spite of a similar structural transition around 90\,K. Here, we report results of Raman scattering experiments as a function of temperature and polarization and simulations based on exact diagonalization of a frustrated spin model. Both experiment and theory find a persistent low energy peak close to 500cm$^{-1}$ in $B_{1g}$ symmetry, which softens slightly around 100\,K, that we assign to spin excitations. By comparing with results from neutron scattering, this study provides evidence for nearly frustrated stripe order in FeSe.

cond-mat.str-el↗
arXiv2019

TransferTransfo: A Transfer Learning Approach for Neural Network Based Conversational Agents

We introduce a new approach to generative data-driven dialogue systems (e.g. chatbots) called TransferTransfo which is a combination of a Transfer learning based training scheme and a high-capacity Transformer model. Fine-tuning is performed by using a multi-task objective which combines several unsupervised prediction tasks. The resulting fine-tuned model shows strong improvements over the current state-of-the-art end-to-end conversational models like memory augmented seq2seq and information-retrieval models. On the privately held PERSONA-CHAT dataset of the Conversational Intelligence Challenge 2, this approach obtains a new state-of-the-art, with respective perplexity, Hits@1 and F1 metrics of 16.28 (45 % absolute improvement), 80.7 (46 % absolute improvement) and 19.5 (20 % absolute improvement).

cs.CL↗
arXiv2018

Evidence of nematic order and nodal superconducting gap along [110] direction in RbFe2As2

Unconventional superconductivity often intertwines with various forms of order, such as the "nematic" order which breaks the rotational symmetry of the lattice. Investigation of these ordered phases sheds crucial light on the superconductivity itself. Here we report a low-temperature scanning tunneling microscopy (STM) study on RbFe2As2, a heavily hole-doped Fe-based superconductor (FeSC). We observe significant symmetry breaking in its electronic structure and magnetic vortex which differentiates the (π, π) and (π, -π) directions of the unfolded Brillouin zone (BZ). It is thus a novel nematic state, distinct from the nematicity of undoped/lightly-doped FeSCs which breaks the (π, 0) / (0, π) equivalence. Moreover, we observe a clear "V"-shaped superconducting gap which can be well fitted with a nodal gap function. The gap is found to be suppressed on surface Rb vacancies and at step edges, and the suppression is particularly strong at the [110]-oriented edges. This is possibly due to a dx2-y2 like pairing component with nodes along the [110] directions. We further demonstrated that such (π, π) nematic state can be suppressed via surface electron doping to RbFe2As2, and the superconductivity is subsequently enhanced. Our results thus highlight the intimate connection between nematicity and superconducting pairing in iron-based superconductors.

cond-mat.supr-con↗
arXiv2018

Magnetization and ac susceptibility study of the cubic chiral magnet Mn$_{1-x}$Fe$_x$Si

We present a comprehensive and systematic magnetization and ac susceptibility study of Mn$_{1-x}$Fe$_{x}$Si over an extensive range of ten Fe concentrations between $x$ = 0 - 0.32. With increasing Fe substitution, the critical temperature decreases but the magnetic phase diagrams remain qualitatively unaltered for $x$ $\leq$ $x^*$ $\approx$ 0.11 with clear boundaries between the helical, conical, and skyrmion lattice phase as well as an enhanced precursor phase. A notably different behavior sets in for $x$ $=$ 0.11, 0.13 and 0.14, where certain characteristics of helimagnetic correlations persist, but without clear phase boundaries. Although a qualitative change already sets in at $x^*$, the transition temperature and spontaneous magnetization vanish only at $x_C$ = 0.17 where also the average magnetic interactions change sign. Although the Curie-Weiss temperature reaches -12~K for $x$ = 0.32, no signature of long-range magnetic order is found down to the lowest temperature, indicating a possible significant role for quantum fluctuations in these systems.

cond-mat.mtrl-sci↗
arXiv2018

Evolution of helimagnetic correlations in Mn$_{1-x}$Fe$_x$Si with doping: a small-angle neutron scattering study

We present a comprehensive small angle neutron scattering study of the doping dependence of the helimagnetic correlations in Mn$_{1-x}$Fe$_{x}$Si. The long-range helimagnetic order in Mn$_{1-x}$Fe$_x$Si is suppressed with increasing Fe content and disappears for $x$ $>$ $x^*$ $\approx$ 0.11, i.e. well before $x_C$ $\approx$ 0.17 where the transition temperature vanishes. For $x$ $>$ $x^*$, only finite isotropic helimagnetic correlations persist which bear similarities with the magnetic correlations found in the precursor phase of MnSi. Magnetic fields gradually suppress and partly align these short-ranged helimagnetic correlations along their direction through a complex magnetization process.

cond-mat.mtrl-sci↗
arXiv2018

A Hierarchical Multi-task Approach for Learning Embeddings from Semantic Tasks

Much effort has been devoted to evaluate whether multi-task learning can be leveraged to learn rich representations that can be used in various Natural Language Processing (NLP) down-stream applications. However, there is still a lack of understanding of the settings in which multi-task learning has a significant effect. In this work, we introduce a hierarchical model trained in a multi-task learning setup on a set of carefully selected semantic tasks. The model is trained in a hierarchical fashion to introduce an inductive bias by supervising a set of low level tasks at the bottom layers of the model and more complex tasks at the top layers of the model. This model achieves state-of-the-art results on a number of tasks, namely Named Entity Recognition, Entity Mention Detection and Relation Extraction without hand-engineered features or external NLP tools like syntactic parsers. The hierarchical training supervision induces a set of shared semantic representations at lower layers of the model. We show that as we move from the bottom to the top layers of the model, the hidden states of the layers tend to represent more complex semantic information.

cs.CL↗
arXiv2018

Nodal gaps in the nematic superconductor FeSe from heat capacity

Superconductivity in FeSe has recently attracted a great deal of attention because it emerges out of an electronic nematic state of elusive character. Here we study both the electronic normal state and the superconducting gap structure using heat-capacity measurements on high-quality single crystals. The specific-heat curve, from 0.4 K to Tc = 9.1 K, is found to be consistent with a recent gap determination using Bogoliubov quasiparticle interference [P. O. Sprau et al., Science 357, 75 (2017)], however only if nodes are introduced on either the electron or the hole Fermi-surface sheets. Our analysis, which is consistent with quantum-oscillation measurements, relies on the presence of only two bands, and thus the fate of the theoretically predicted second electron pocket remains mysterious.

cond-mat.supr-con↗
arXiv2018

Electron-phonon coupling in undoped cuprate $YBa_2Cu_3O_6$ estimated from Raman and optical conductivity spectra

We study experimentally the Raman response of the undoped high-Tc parent compound $YBa_2Cu_3O_6$, and give a unified theory of the two-magnon Raman peak and optical conductivity based on the Hubbard-Holstein model with electron-phonon coupling (EPC). The Hubbard model without EPC can qualitatively account for the experimentally observed resonance of the Raman response, but only the Hubbard-Holstein model (i) reproduces asymmetry of the Raman spectrum, (ii) validates experimental visibility of the two-magnon peak, and (iii) predicts the correct shape and energy of the lower edge of the charge transfer gap in optical conductivity. Comparison of experiments with the theory gives the EPC strength $λ$ = 0.6. This result convincingly indicates the vital role of EPC in high-Tc cuprates providing a clue to the mechanism of high-Tc.

cond-mat.str-el↗
arXiv2018

Thermoelectric anisotropy in Ba(Fe1-xCox)2As2 iron-based superconductor

We report the in-plane anisotropy of the Seebeck and Nernst coefficients as well as of the electrical resistivity determined for the series of the strain-detwinned single crystals of Ba(Fe1-xCox)2As2. Two underdoped samples (x = 0.024, 0.045) exhibiting the transition from the tetragonal paramagnetic phase to the orthorhombic spin density wave (SDW) phase (at Ttr = 100 and 60 K, respectively) show an onset of the Nernst anisotropy at temperatures above 200 K, which is significantly higher than Ttr. In the optimally doped sample (x = 0.06) the transport properties also appear to be in-plane anisotropic below T = 120 K, despite the fact that this particular composition does not show any evidence of long-range magnetic order. However, the anisotropy observed in the optimally doped crystal is rather small and for the Seebeck and Nernst coefficients the difference between values measured along and across the uniaxial strain has opposite sign to those observed for underdoped crystals with x = 0.024 and 0.045. For these two samples, insensitivity of the Nernst anisotropy to the SDW transition suggests that the nematicity might be of other than magnetic origin.

cond-mat.supr-con↗
arXiv2018

Continuous Learning in a Hierarchical Multiscale Neural Network

We reformulate the problem of encoding a multi-scale representation of a sequence in a language model by casting it in a continuous learning framework. We propose a hierarchical multi-scale language model in which short time-scale dependencies are encoded in the hidden state of a lower-level recurrent neural network while longer time-scale dependencies are encoded in the dynamic of the lower-level network by having a meta-learner update the weights of the lower-level neural network in an online meta-learning fashion. We use elastic weights consolidation as a higher-level to prevent catastrophic forgetting in our continuous learning framework.

cs.CL↗
arXiv2018

Meta-Learning a Dynamical Language Model

We consider the task of word-level language modeling and study the possibility of combining hidden-states-based short-term representations with medium-term representations encoded in dynamical weights of a language model. Our work extends recent experiments on language models with dynamically evolving weights by casting the language modeling problem into an online learning-to-learn framework in which a meta-learner is trained by gradient-descent to continuously update a language model weights.

cs.CL↗