arXiv subjects

Dan Padnos

Publications and source records attributed to Dan Padnos.

8 recordsLinked to original sources

arXiv2024

Jamba-1.5: Hybrid Transformer-Mamba Models at Scale

We present Jamba-1.5, new instruction-tuned large language models based on our Jamba architecture. Jamba is a hybrid Transformer-Mamba mixture of experts architecture, providing high throughput and low memory usage across context lengths, while retaining the same or better quality as Transformer models. We release two model sizes: Jamba-1.5-Large, with 94B active parameters, and Jamba-1.5-Mini, with 12B active parameters. Both models are fine-tuned for a variety of conversational and instruction-following capabilties, and have an effective context length of 256K tokens, the largest amongst open-weight models. To support cost-effective inference, we introduce ExpertsInt8, a novel quantization technique that allows fitting Jamba-1.5-Large on a machine with 8 80GB GPUs when processing 256K-token contexts without loss of quality. When evaluated on a battery of academic and chatbot benchmarks, Jamba-1.5 models achieve excellent results while providing high throughput and outperforming other open-weight models on long-context benchmarks. The model weights for both sizes are publicly available under the Jamba Open Model License and we release ExpertsInt8 as open source.

cs.CL↗

arXiv2020

Exemplar Guided Active Learning

We consider the problem of wisely using a limited budget to label a small subset of a large unlabeled dataset. We are motivated by the NLP problem of word sense disambiguation. For any word, we have a set of candidate labels from a knowledge base, but the label set is not necessarily representative of what occurs in the data: there may exist labels in the knowledge base that very rarely occur in the corpus because the sense is rare in modern English; and conversely there may exist true labels that do not exist in our knowledge base. Our aim is to obtain a classifier that performs as well as possible on examples of each "common class" that occurs with frequency above a given threshold in the unlabeled set while annotating as few examples as possible from "rare classes" whose labels occur with less than this frequency. The challenge is that we are not informed which labels are common and which are rare, and the true label distribution may exhibit extreme skew. We describe an active learning approach that (1) explicitly searches for rare classes by leveraging the contextual embedding spaces provided by modern language models, and (2) incorporates a stopping rule that ignores classes once we prove that they occur below our target threshold with high probability. We prove that our algorithm only costs logarithmically more than a hypothetical approach that knows all true label frequencies and show experimentally that incorporating automated search can significantly reduce the number of samples needed to reach target accuracy levels.

cs.LG↗

arXiv2020

Technical Report: Auxiliary Tuning and its Application to Conditional Text Generation

We introduce a simple and efficient method, called Auxiliary Tuning, for adapting a pre-trained Language Model to a novel task; we demonstrate this approach on the task of conditional text generation. Our approach supplements the original pre-trained model with an auxiliary model that shifts the output distribution according to the target task. The auxiliary model is trained by adding its logits to the pre-trained model logits and maximizing the likelihood of the target task output. Our method imposes no constraints on the auxiliary architecture. In particular, the auxiliary model can ingest additional input relevant to the target task, independently from the pre-trained model's input. Furthermore, mixing the models at the logits level provides a natural probabilistic interpretation of the method. Our method achieved similar results to training from scratch for several different tasks, while using significantly fewer resources for training; we share a specific example of text generation conditioned on keywords.

cs.CL↗

arXiv2020

SenseBERT: Driving Some Sense into BERT

The ability to learn from large unlabeled corpora has allowed neural language models to advance the frontier in natural language understanding. However, existing self-supervision techniques operate at the word form level, which serves as a surrogate for the underlying semantic content. This paper proposes a method to employ weak-supervision directly at the word sense level. Our model, named SenseBERT, is pre-trained to predict not only the masked words but also their WordNet supersenses. Accordingly, we attain a lexical-semantic level language model, without the use of human annotation. SenseBERT achieves significantly improved lexical understanding, as we demonstrate by experimenting on SemEval Word Sense Disambiguation, and by attaining a state of the art result on the Word in Context task.

cs.CL↗

arXiv2019

Instability of Supersonic Cold Streams Feeding Galaxies III: Kelvin-Helmholtz Instability in Three Dimensions

We study the effects of Kelvin-Helmholtz instability (KHI) on the cold streams that feed high-redshift galaxies through their hot haloes, generalizing our earlier analyses of a 2D slab to a 3D cylinder, but still limiting our analysis to the adiabatic case with no gravity. We combine analytic modeling and numerical simulations in the linear and non-linear regimes. For subsonic or transonic streams with respect to the halo sound speed, the instability in 3D is qualitatively similar to 2D, but progresses at a faster pace. For supersonic streams, the instability grows much faster in 3D and can be qualitatively different due to azimuthal modes, which introduce a strong dependence on the initial width of the stream-background interface. Using analytic toy models and approximations supported by high-resolution simulations, we apply our idealized hydrodynamical analysis to the astrophysical scenario. The upper limit for the radius of a stream that disintegrates prior to reaching the central galaxy is ~70% larger than the 2D estimate; it is in the range (0.5-5)% of the halo virial radius, decreasing with increasing stream density and velocity. Stream disruption generates a turbulent mixing zone around the stream with velocities at the level of ~20% of the initial stream velocity. KHI can cause significant stream deceleration and energy dissipation in 3d, contrary to 2D estimates. For typical streams, up to (10-50)% of the gravitational energy gained by inflow down the dark-matter halo potential can be dissipated, capable of powering Lyman-alpha blobs if most of it is dissipated into radiation.

astro-ph.GA↗

arXiv2018

Instability of Supersonic Cold Streams Feeding Galaxies II. Nonlinear Evolution of Surface and Body Modes of Kelvin-Helmholtz Instability

As part of our long-term campaign to understand how cold streams feed massive galaxies at high redshift, we study the Kelvin-Helmholtz instability (KHI) of a supersonic, cold, dense gas stream as it penetrates through a hot, dilute circumgalactic medium (CGM). A linear analysis (Paper I) showed that, for realistic conditions, KHI may produce nonlinear perturbations to the stream during infall. Therefore, we proceed here to study the nonlinear stage of KHI, still limited to a two-dimensional slab with no radiative cooling or gravity. Using analytic models and numerical simulations, we examine stream breakup, deceleration and heating via surface modes and body modes. The relevant parameters are the density contrast between stream and CGM ($δ$), the Mach number of the stream velocity with respect to the CGM ($M_{\rm b}$) and the stream radius relative to the halo virial radius ($R_{\rm s}/R_{\rm v}$). We find that sufficiently thin streams disintegrate prior to reaching the central galaxy. The condition for breakup ranges from $R_{\rm s} < 0.03 R_{\rm v}$ for $(M_{\rm b} \sim 0.75, δ\sim 10)$ to $R_{\rm s} < 0.003 R_{\rm v}$ for $(M_{\rm b} \sim 2.25, δ\sim 100)$. However, due to the large stream inertia, KHI has only a small effect on the stream inflow rate and a small contribution to heating and subsequent Lyman-$α$ cooling emission.

astro-ph.GA↗

arXiv2016

The hydrodynamic stability of gaseous cosmic filaments

Virial shocks at edges of cosmic-web structures are a clear prediction of standard structure formation theories. We derive a criterion for the stability of the post-shock gas and of the virial shock itself in spherical, filamentary and planar infall geometries. When gas cooling is important, we find that shocks become unstable, and gas flows uninterrupted towards the center of the respective halo, filament or sheet. For filaments, we impose this criterion on self-similar infall solutions. We find that instability is expected for filament masses between $10^{11}-10^{13}M_\odot Mpc^{-1}.$ Using a simplified toy model, we then show that these filaments will likely feed halos with $10^{10}M_{\odot}\lesssim M_{halo}\lesssim 10^{13}M_{\odot}$ at redshift $z=3$, as well as $10^{12}M_{\odot}\lesssim M_{halo}\lesssim 10^{15}M_{\odot}$ at $z=0$. The instability will affect the survivability of the filaments as they penetrate gaseous halos in a non-trivial way. Additionally, smaller halos accreting onto non-stable filaments will not be subject to ram-pressure inside the filaments. The instreaming gas will continue towards the center, and stop either once its angular momentum balances the gravitational attraction, or when its density becomes so high that it becomes self-shielded to radiation.

astro-ph.GA↗

arXiv2016

Instability of Supersonic Cold Streams Feeding Galaxies I: Linear Kelvin-Helmholtz Instability with Body Modes

Massive galaxies at high redshift are predicted to be fed from the cosmic web by narrow, dense, cold streams. These streams penetrate supersonically through the hot medium encompassed by a stable shock near the virial radius of the dark-matter halo. Our long-term goal is to explore the heating and dissipation rate of the streams and their fragmentation and possible breakup, in order to understand how galaxies are fed, and how this affects their star-formation rate and morphology. We present here the first step, where we analyze the linear Kelvin-Helmholtz instability (KHI) of a cold, dense slab or cylinder flowing through a hot, dilute medium in the transonic regime. The current analysis is limited to the adiabatic case with no gravity and assuming equal pressure in the stream and the medium. By analytically solving the linear dispersion relation, we find a transition from a dominance of the familiar rapidly growing surface modes in the subsonic regime to more slowly growing body modes in the supersonic regime. The system is parameterized by three parameters: the density contrast between the stream and the medium, the Mach number of stream velocity with respect to the medium, and the stream width with respect to the halo virial radius. We find that a realistic choice for these parameters places the streams near the mode transition, with the KHI exponential-growth time in the range 0.01-10 virial crossing times for a perturbation wavelength comparable to the stream width. We confirm our analytic predictions with idealized hydrodynamical simulations. Our linear-KHI estimates thus indicate that KHI may in principle be effective in the evolution of streams by the time they reach the galaxy. More definite conclusions await the extension of the analysis to the nonlinear regime and the inclusion of cooling, thermal conduction, the halo potential well, self-gravity and magnetic fields.

astro-ph.GA↗