arXiv subjects

Thomas Wolf

Publications and source records attributed to Thomas Wolf.

At least 55 records · Page 3Linked to original sources

arXiv2022

Evaluate & Evaluation on the Hub: Better Best Practices for Data and Model Measurements

Evaluation is a key part of machine learning (ML), yet there is a lack of support and tooling to enable its informed and systematic practice. We introduce Evaluate and Evaluation on the Hub --a set of tools to facilitate the evaluation of models and datasets in ML. Evaluate is a library to support best practices for measurements, metrics, and comparisons of data and models. Its goal is to support reproducibility of evaluation, centralize and document the evaluation process, and broaden evaluation to cover more facets of model performance. It includes over 50 efficient canonical implementations for a variety of domains and scenarios, interactive documentation, and the ability to easily share implementations and outcomes. The library is available at https://github.com/huggingface/evaluate. In addition, we introduce Evaluation on the Hub, a platform that enables the large-scale evaluation of over 75,000 models and 11,000 datasets on the Hugging Face Hub, for free, at the click of a button. Evaluation on the Hub is available at https://huggingface.co/autoevaluate.

cs.LG↗

arXiv2022

Quantum critical fluctuations in an Fe-based superconductor

Quantum critical fluctuations may prove to play an instrumental role in the formation of unconventional superconductivity. Here, we show that the characteristic scaling of a marginal Fermi liquid is present in inelastic light scattering data of an Fe-based superconductor tuned through a quantum critical point (QCP) by chemical substitution or doping. From the doping dependence of the imaginary time dynamics we are able to distinguish regions dominated by quantum critical behavior from those having classical critical responses. This dichotomy reveals a connection between the marginal Fermi liquid behavior and quantum criticality. In particular, the overlap between regions of high superconducting transition temperatures and quantum critical scaling suggests a contribution from quantum fluctuations to the formation of superconductivity.

cond-mat.supr-con↗

arXiv2022

Charge-density-wave transitions, phase diagram, soft phonon and possible electronic nematicity: a thermodynamic investigation of BaNi2(As,P)2

A detailed investigation of BaNi2(As,P)2 single crystals using high-resolution thermal-expansion, heat-capacity,Young-modulus and resistivity measurements is presented. The experimental data are complemented by density-functional calculations. The phase diagram of BaNi2(As,P)2 is shown to be much richer than suggested by the original data of Kudo et al. [Phys. Rev. Lett. 109, 097002 (2012)]. The transition to the commensurate charge-density-wave (C-CDW) is always preceded by a four-fold symmetry-breaking transition associated with the long-range ordering of a strongly fluctuating unidirectional incommensurate charge-density wave (I-CDW). Significant precursors above the I-CDW and C-CDW transitions are seen in the thermal expansion and resistivity and are particularly evident in the temperature dependence of the $c/a$ ratio of the lattice parameters. Heat-capacity measurements of the crystals with a higher P content and a higher critical temperature of 3.2 K uncover a Debye-like behavior of a soft-phonon mode with a very low Debye temperature of roughly 15 K. Associated with this soft phonon are unusually large thermal-expansion anomalies, resulting in logarithmically diverging uniaxial phonon Grueneisen parameters. Young-modulus data of these higher-Tc crystals exhibit a significant softening in both B1g and B2g channels, which is argued to be incompatible with nematic criticality and is rather associated with a broad phase transition to an hitherto unknown structure. Possible origins of the increase in the superconducting critical temperature with P-substitution are discussed.

cond-mat.supr-con↗

arXiv2022

Training Transformers Together

The infrastructure necessary for training state-of-the-art models is becoming overly expensive, which makes training such models affordable only to large corporations and institutions. Recent work proposes several methods for training such models collaboratively, i.e., by pooling together hardware from many independent parties and training a shared model over the Internet. In this demonstration, we collaboratively trained a text-to-image transformer similar to OpenAI DALL-E. We invited the viewers to join the ongoing training run, showing them instructions on how to contribute using the available hardware. We explained how to address the engineering challenges associated with such a training run (slow communication, limited memory, uneven performance between devices, and security concerns) and discussed how the viewers can set up collaborative training runs themselves. Finally, we show that the resulting model generates images of reasonable quality on a number of prompts.

cs.LG↗

arXiv2022

Novel Increase of Superconducting Critical Temperature of an Iron-Superconductor due to Ion Implantation

Energetic ion irradiation usually decreases superconducting critical temperature(Tc), with the few exceptions involving increases up to a few K only. However, our recent 2.5X10^15 Ar/cm2 irradiations by 1.5 MeV Ar6+ enhanced Tc of the single crystal Fe-superconductor Ba(Fe0.943Co0.057)2As2 by 8.2 K from its initial onset Tc of ~16.9 K as measured from the real part of the magnetic susceptibility, matching measurements from the imaginary part, electrical resistivity and magnetization. Ozaki et al. (2016) explained their Tc increase of 0.5 K in FeSe0.5Te0.5 films with the thickness (t) < the irradiating proton range (R), as due to a nanoscale compressive strain developed from radiation damage of the lattice. Here, Ar irradiation with t > R results in an Ar implanted layer in our crystal. Implanted inert gas atoms often agglomerate into high-pressure bubbles to exert a large compressive strain on the lattice. We suggest that this additional compressive strain could be the reason for such a large (~49%) Tc increase.

cond-mat.supr-con↗

arXiv2022

Multitask Prompted Training Enables Zero-Shot Task Generalization

Large language models have recently been shown to attain reasonable zero-shot generalization on a diverse set of tasks (Brown et al., 2020). It has been hypothesized that this is a consequence of implicit multitask learning in language models' pretraining (Radford et al., 2019). Can zero-shot generalization instead be directly induced by explicit multitask learning? To test this question at scale, we develop a system for easily mapping any natural language tasks into a human-readable prompted form. We convert a large set of supervised datasets, each with multiple prompts with diverse wording. These prompted datasets allow for benchmarking the ability of a model to perform completely held-out tasks. We fine-tune a pretrained encoder-decoder model (Raffel et al., 2020; Lester et al., 2021) on this multitask mixture covering a wide variety of tasks. The model attains strong zero-shot performance on several standard datasets, often outperforming models up to 16x its size. Further, our approach attains strong performance on a subset of tasks from the BIG-bench benchmark, outperforming models up to 6x its size. All trained models are available at https://github.com/bigscience-workshop/t-zero and all prompts are available at https://github.com/bigscience-workshop/promptsource.

cs.LG↗

arXiv2021

Rotational symmetry breaking at the incommensurate charge-density-wave transition in Ba(Ni,Co)$_2$(As,P)$_2$: Possible nematic phase induced by charge/orbital fluctuations

Nematic phase transitions in high-temperature superconductors have a strong impact on the electronic properties of these systems. BaFe$_2$As$_2$,\@ with an established nematic transition around 137 K induced by magnetic fluctuations, and BaNi$_2$As$_2$,\@ a non-magnetic analog of BaFe$_2$As$_2$ with a structural transition in the same temperature range,\@ share a common tetragonal aristotype crystal structure with space-group type $I4/mmm$.\@ In contrast to BaFe$_2$As$_2$ where collinear stripe magnetic order is found for the orthorhombic low-$T$ phase, a unidirectional charge density wave together with (distorted) zig-zag chains are observed for the triclinic low-$T$ phase of BaNi$_2$As$_2$.\@ Here we show that between the high- and low-$T$ phases of Ba(Ni,Co)$_2$(As,P)$_2$ an additional phase with broken fourfold symmetry and $d_{xz}$ orbital order exists which is a promising candidate for charge/orbital-fluctuation-induced nematicity. Moreover, our data suggest that the enhanced $T_{\rm c}$ found for Ba(Ni,Co)$_2$(As,P)$_2$ for higher Co or P substitution levels might result from suppression of the (distorted) zig-zag chains by reducing the contribution of the $d_{xy}$ orbitals.

cond-mat.supr-con↗

arXiv2021

Distributed Deep Learning in Open Collaborations

Modern deep learning applications require increasingly more compute to train state-of-the-art models. To address this demand, large corporations and institutions use dedicated High-Performance Computing clusters, whose construction and maintenance are both environmentally costly and well beyond the budget of most organizations. As a result, some research directions become the exclusive domain of a few large industrial and even fewer academic actors. To alleviate this disparity, smaller groups may pool their computational resources and run collaborative experiments that benefit all participants. This paradigm, known as grid- or volunteer computing, has seen successful applications in numerous scientific areas. However, using this approach for machine learning is difficult due to high latency, asymmetric bandwidth, and several challenges unique to volunteer computing. In this work, we carefully analyze these constraints and propose a novel algorithmic framework designed specifically for collaborative training. We demonstrate the effectiveness of our approach for SwAV and ALBERT pretraining in realistic conditions and achieve performance comparable to traditional setups at a fraction of the cost. Finally, we provide a detailed report of successful collaborative language model pretraining with 40 participants.

cs.LG↗

arXiv2021

Nematic response revealed by coherent phonon oscillations in BaFe$_2$As$_2$

We investigate coherent phonon oscillations of BaFe$_2$As$_2$ using optical pump-probe spectroscopy. Time-resolved optical reflectivity shows periodic modulations due to $A_{1g}$ coherent phonon of $c$-axis arsenic vibrations. Optical probe beams polarized along the orthorhombic $a$- and $b$-axes reveal that the initial phase of coherent oscillations shows a systematic deviation as a function of temperature, although these oscillations arise from the same $c$-axis arsenic vibrations. The oscillation-phase remains anisotropic even in the tetragonal structure, reflecting a nematic response of BaFe$_2$As$_2$. Our study suggests that investigation on the phase of coherent phonon oscillations in optical reflectivity can offer unique evidence of a nematic order strongly coupled to a lattice instability.

cond-mat.supr-con↗

arXiv2021

Datasets: A Community Library for Natural Language Processing

The scale, variety, and quantity of publicly-available NLP datasets has grown rapidly as researchers propose new tasks, larger models, and novel benchmarks. Datasets is a community library for contemporary NLP designed to support this ecosystem. Datasets aims to standardize end-user interfaces, versioning, and documentation, while providing a lightweight front-end that behaves similarly for small datasets as for internet-scale corpora. The design of the library incorporates a distributed, community-driven approach to adding datasets and documenting usage. After a year of development, the library now includes more than 650 unique datasets, has more than 250 contributors, and has helped support a variety of novel cross-dataset research projects and shared tasks. The library is available at https://github.com/huggingface/datasets.

cs.CL↗

arXiv2021

VIMPAC: Video Pre-Training via Masked Token Prediction and Contrastive Learning

Video understanding relies on perceiving the global content and modeling its internal connections (e.g., causality, movement, and spatio-temporal correspondence). To learn these interactions, we apply a mask-then-predict pre-training task on discretized video tokens generated via VQ-VAE. Unlike language, where the text tokens are more independent, neighboring video tokens typically have strong correlations (e.g., consecutive video frames usually look very similar), and hence uniformly masking individual tokens will make the task too trivial to learn useful representations. To deal with this issue, we propose a block-wise masking strategy where we mask neighboring video tokens in both spatial and temporal domains. We also add an augmentation-free contrastive learning method to further capture the global content by predicting whether the video clips are sampled from the same video. We pre-train our model on uncurated videos and show that our pre-trained model can reach state-of-the-art results on several video understanding datasets (e.g., SSV2, Diving48). Lastly, we provide detailed analyses on model scalability and pre-training method design. Code is released at https://github.com/airsplay/vimpac.

cs.CV↗

arXiv2020

Evolution of Helimagnetic Correlations when approaching the Quantum Critical Point of Mn$_{1-x}$Fe$_x$Si

We present a comprehensive investigation of the evolution of helimagnetic correlations in Mn$_{1-x}$Fe$_x$Si with increasing doping. By combining polarised neutron scattering and high resolution Neutron Spin Echo spectroscopy we investigate three samples with $x$=0.09, 0.11 and 0.14, i.e. with compositions on both sides of the concentration $x^* \sim 0.11$ where the helimagnetic Bragg peaks disappear and between $x^*$ and the quantum critical concentration $x_C \sim 0.17$, where $T_C$ vanishes. We find that the abrupt disappearance of the long range helical periodicity at $x^*$, does not affect the precursor fluctuating correlations. These build up with decreasing temperature in a similar way as for the parent compound MnSi. Also the dynamics bears strong similarities to MnSi. The analysis of our results indicates that frustration, possibly due to achiral RKKY interactions, increases with increasing Fe doping. We argue that this effect explains both the expansion of the precursor phase with increasing $x$ and the abrupt disappearance of long range helimagnetic periodicity at $x^*$.

cond-mat.str-el↗

arXiv2020

Learning from others' mistakes: Avoiding dataset biases without modeling them

State-of-the-art natural language processing (NLP) models often learn to model dataset biases and surface form correlations instead of features that target the intended underlying task. Previous work has demonstrated effective methods to circumvent these issues when knowledge of the bias is available. We consider cases where the bias issues may not be explicitly identified, and show a method for training models that learn to ignore these problematic correlations. Our approach relies on the observation that models with limited capacity primarily learn to exploit biases in the dataset. We can leverage the errors of such limited capacity models to train a more robust model in a product of experts, thus bypassing the need to hand-craft a biased model. We show the effectiveness of this method to retain improvements in out-of-distribution settings even if no particular bias is targeted by the biased model.

cs.CL↗

arXiv2020

Short-range nematic fluctuations in Sr1-xNaxFe2As2 superconductors

Interactions between nematic fluctuations, magnetic order and superconductivity are central to the physics of iron-based superconductors. Here we report on in-plane transverse acoustic phonons in hole-doped Sr$_{1-x}$Na$_x$Fe$_2$As$_2$ measured via inelastic X-ray scattering, and extract both the nematic susceptibility and the nematic correlation length. By a self-contained method of analysis, for the underdoped ($x=0.36$) sample, which harbors a magnetically-ordered tetragonal phase, we find it hosts a short nematic correlation length $ξ$ ~ 10 $Å$ and a large nematic susceptibility $χ_{\rm nem}$. The optimal-doped ($x=0.55$) sample exhibits weaker phonon softening effects, indicative of both reduced $ξ$ and $χ_{\rm nem}$. Our results suggest short-range nematic fluctuations may favor superconductivity, placing emphasis on the nematic correlation length for understanding the iron-based superconductors.

cond-mat.supr-con↗

arXiv2020

Anisotropic optical properties of detwinned BaFe$_{2}$As$_{2}$

The optical properties of a large, detwinned single crystal of BaFe$_2$As$_2$ have been examined over a wide frequency range above and below the structural and magnetic transition at $T_{\rm N}\simeq 138$ K. Above $T_{\rm N}$ the real part of the optical conductivity and the two infrared-active lattice modes are almost completely isotropic; the lattice modes show a weak polarization dependence just above $T_{\rm N}$. For $T<T_{\rm N}$, the optical conductivity due to the free-carrier response is anisotropic, being larger along the $a$ axis than the $b$ axis below $\simeq 30$ meV; above this energy the optical conductivity is dominated by the interband contributions, which appear to be isotropic. The splitting of the low-energy infrared-active mode below $T_{\rm N}$ is clearly observed, and the polarization modulation of the new modes may be used to estimate that the crystal is $\simeq 70$% detwinned. The high-frequency mode, with a threefold increase in strength of the lower branch below $T_{\rm N}$ and nearly silent upper branch, remains enigmatic.

cond-mat.str-el↗

arXiv2020

Movement Pruning: Adaptive Sparsity by Fine-Tuning

Magnitude pruning is a widely used strategy for reducing model size in pure supervised learning; however, it is less effective in the transfer learning regime that has become standard for state-of-the-art natural language processing applications. We propose the use of movement pruning, a simple, deterministic first-order weight pruning method that is more adaptive to pretrained model fine-tuning. We give mathematical foundations to the method and compare it to existing zeroth- and first-order pruning methods. Experiments show that when pruning large pretrained language models, movement pruning shows significant improvements in high-sparsity regimes. When combined with distillation, the approach achieves minimal accuracy loss with down to only 3% of the model parameters.

cs.CL↗

arXiv2020

Unconventional Hund Metal in a Weak Itinerant Ferromagnet

The physics of weak itinerant ferromagnets is challenging due to their small magnetic moments and the ambiguous role of local interactions governing their electronic properties, many of which violate Fermi liquid theory. While magnetic fluctuations play an important role in the materials' unusual electronic states, the nature of these fluctuations and the paradigms through which they arise remain debated. Here we use inelastic neutron scattering to study magnetic fluctuations in the canonical weak itinerant ferromagnet MnSi. Data reveal that short-wavelength magnons continue to propagate until a mode crossing predicted for strongly interacting quasiparticles is reached, and the local susceptibility peaks at a coherence energy predicted for a correlated Hund metal by first-principles many-body theory. Scattering between electrons and orbital and spin fluctuations in MnSi can be understood at the local level to generate non-Fermi liquid character. These results provide crucial insight into the role of interorbital Hund's exchange within the broader class of enigmatic multiband itinerant, weak ferromagnets.

cond-mat.str-el↗

arXiv2020

Band Engineering of Dirac cones in Iron Chalcogenides

By band engineering the iron chalcogenide Fe(Se,Te) via ab-initio calculations, we search for topological surface states and realizations of Majorana bound states. Proposed topological states are expected to occur for non-stoichiometric compositions on a surface Dirac cone where issues like disorder scattering and charge transfer between relevant electronic states have to be addressed. However, this surface Dirac cone is well above the Fermi-level. Our goal is to theoretically design a substituted crystal in which the surface Dirac cone is shifted towards the Fermi-level by modifying the bulk material without disturbing the surface. Going beyond conventional density functional theory (DFT), we apply the coherent potential approximation (BEB-CPA) in a mixed basis pseudo-potential framework to scan the substitutional phase-space of co-substitutions on the Se-sites. We have identified iodine as a promising candidate for intrinsic doping. Our specific proposal is that FeSe$_{0.325}$I$_{0.175}$Te$_{0.5}$ is a very likely candidate to exhibit a Dirac cone right at the Fermi energy without inducing strong disorder scattering.

cond-mat.mtrl-sci↗