arXiv subjects

Michael Hahn

Publications and source records attributed to Michael Hahn.

At least 55 records · Page 3Linked to original sources

arXiv2024

Why are Sensitive Functions Hard for Transformers?

Empirical studies have identified a range of learnability biases and limitations of transformers, such as a persistent difficulty in learning to compute simple formal languages such as PARITY, and a bias towards low-degree functions. However, theoretical understanding remains limited, with existing expressiveness theory either overpredicting or underpredicting realistic learning abilities. We prove that, under the transformer architecture, the loss landscape is constrained by the input-space sensitivity: Transformers whose output is sensitive to many parts of the input string inhabit isolated points in parameter space, leading to a low-sensitivity bias in generalization. We show theoretically and empirically that this theory unifies a broad array of empirical observations about the learning abilities and biases of transformers, such as their generalization bias towards low sensitivity and low degree, and difficulty in length generalization for PARITY. This shows that understanding transformers' inductive biases requires studying not just their in-principle expressivity, but also their loss landscape.

cs.LG↗

arXiv2024

Experimental study of Alfvén wave reflection from an Alfvén-speed gradient relevant to the solar coronal holes

We report the first experimental detection of a reflected Alfvén wave from an Alfvén-speed gradient under conditions similar to those in coronal holes. The experiments were conducted in the Large Plasma Device at the University of California, Los Angeles. We present the experimentally measured dependence of the coefficient of reflection versus the wave inhomogeneity parameter, i.e., the ratio of the wave length of the incident wave to the length scale of the gradient. Two-fluid simulations using the Gkeyll code qualitatively agree with and support the experimental findings. Our experimental results support models of wave heating that rely on wave reflection at low heights from a smooth Alfvén-speed gradient to drive turbulence.

astro-ph.SR↗

arXiv2023

A Cross-Linguistic Pressure for Uniform Information Density in Word Order

While natural languages differ widely in both canonical word order and word order flexibility, their word orders still follow shared cross-linguistic statistical patterns, often attributed to functional pressures. In the effort to identify these pressures, prior work has compared real and counterfactual word orders. Yet one functional pressure has been overlooked in such investigations: the uniform information density (UID) hypothesis, which holds that information should be spread evenly throughout an utterance. Here, we ask whether a pressure for UID may have influenced word order patterns cross-linguistically. To this end, we use computational models to test whether real orders lead to greater information uniformity than counterfactual orders. In our empirical study of 10 typologically diverse languages, we find that: (i) among SVO languages, real word orders consistently have greater uniformity than reverse word orders, and (ii) only linguistically implausible counterfactual orders consistently exceed the uniformity of real orders. These findings are compatible with a pressure for information uniformity in the development and usage of natural languages.

cs.CL↗

arXiv2023

Are Non-thermal Velocities in Active Region Coronal Loops Anisotropic?

We have measured line widths in active region coronal loops in order to determine whether the non-thermal broadening is anisotropic with respect to the magnetic field direction. These non-thermal velocities are caused by unresolved fluid motions. Our analysis method combines spectroscopic data and a magnetic field extrapolation. We analyzed spectra from the Extreme Ultraviolet Imaging Spectrometer on Hinode. A differential emission measure analysis showed that many spectral lines that are commonly considered to be formed in the active region have a substantial contribution from the background quiet Sun. From these spectra we identified lines whose emission was dominated by the active region loops rather than background sources. Using these lines, we constructed maps of the non-thermal velocity. With data from the Helioseismic Magnetic Imager on the Solar Dynamics Observatory and the Coronal Modeling System nonlinear force-free magnetic field reconstruction code, we traced several of the magnetic field lines through the active region. Comparing the spectroscopic and magnetic data, we looked for correlations of non-thermal velocity with the viewing angle between the line of sight and the magnetic field. We found that non-thermal velocities show a weak anti-correlation with the viewing angle. That is, the tendency is for the non-thermal velocity to be slightly larger in the parallel direction. This parallel broadening may be due to acoustic waves or unresolved parallel flows.

astro-ph.SR↗

arXiv2023

A Theory of Emergent In-Context Learning as Implicit Structure Induction

Scaling large language models (LLMs) leads to an emergent capacity to learn in-context from example demonstrations. Despite progress, theoretical understanding of this phenomenon remains limited. We argue that in-context learning relies on recombination of compositional operations found in natural language data. We derive an information-theoretic bound showing how in-context learning abilities arise from generic next-token prediction when the pretraining distribution has sufficient amounts of compositional structure, under linguistically motivated assumptions. A second bound provides a theoretical justification for the empirical success of prompting LLMs to output intermediate steps towards an answer. To validate theoretical predictions, we introduce a controlled setup for inducing in-context learning; unlike previous approaches, it accounts for the compositional nature of language. Trained transformers can perform in-context learning for a range of tasks, in a manner consistent with the theoretical results. Mirroring real-world LLMs in a miniature setup, in-context learning emerges when scaling parameters and data, and models perform better when prompted to output intermediate steps. Probing shows that in-context learning is supported by a representation of the input's compositional structure. Taken together, these results provide a step towards theoretical understanding of emergent behavior in large language models.

cs.CL↗

arXiv2022

Deriving instrumental point spread functions from partially occulted images

The point-spread function (PSF) of an imaging system describes the response of the system to a point source. Accurately determining the PSF enables one to correct for the combined effects of focussing and scattering within the imaging system, and thereby enhance the spatial resolution and dynamic contrast of the resulting images. We present a semi-empirical semi-blind methodology to derive a PSF from partially occulted images. We partition the two-dimensional PSF into multiple segments, set up a multi-linear system of equations, and directly fit the system of equations to determine the PSF weight in each segment. The algorithm is guaranteed to converge towards the correct instrumental PSF for a large class of occultations, does not require a predefined functional form of the PSF, and can be applied to a large variety of partially occulted images, such as within laboratory settings, regular calibrations within a production line or in the field, astronomical images of distant clusters of stars, or partial solar eclipse images. We show that the central weight of the PSF, which gives the percentage of photons that are not scattered by the instrument, is accurate to bettern than 1.2%. The mean absolute percentage error between the reconstructed and true PSF is usually between 0.5% and 5% for the entire PSF, between 0.5% and 5% for the PSF core, and between 0.5% and 3% for the PSF tail.

astro-ph.IM↗

arXiv2022

Modeling Task Effects in Human Reading with Neural Network-based Attention

Research on human reading has long documented that reading behavior shows task-specific effects, but it has been challenging to build general models predicting what reading behavior humans will show in a given task. We introduce NEAT, a computational model of the allocation of attention in human reading, based on the hypothesis that human reading optimizes a tradeoff between economy of attention and success at a task. Our model is implemented using contemporary neural network modeling techniques, and makes explicit and testable predictions about how the allocation of attention varies across different tasks. We test this in an eyetracking study comparing two versions of a reading comprehension task, finding that our model successfully accounts for reading behavior across the tasks. Our work thus provides evidence that task effects can be modeled as optimal adaptation to task demands.

cs.CL↗

arXiv2022

Crosslinguistic word order variation reflects evolutionary pressures of dependency and information locality

Languages vary considerably in syntactic structure. About 40% of the world's languages have subject-verb-object order, and about 40% have subject-object-verb order. Extensive work has sought to explain this word order variation across languages. However, the existing approaches are not able to explain coherently the frequency distribution and evolution of word order in individual languages. We propose that variation in word order reflects different ways of balancing competing pressures of dependency locality and information locality, whereby languages favor placing elements together when they are syntactically related or contextually informative about each other. Using data from 80 languages in 17 language families and phylogenetic modeling, we demonstrate that languages evolve to balance these pressures, such that word order change is accompanied by change in the frequency distribution of the syntactic structures which speakers communicate to maintain overall efficiency. Variability in word order thus reflects different ways in which languages resolve these evolutionary pressures. We identify relevant characteristics that result from this joint optimization, particularly the frequency with which subjects and objects are expressed together for the same verb. Our findings suggest that syntactic structure and usage across languages co-adapt to support efficient communication under limited cognitive resources.

cs.CL↗

arXiv2022

Evidence for Parameteric Decay Instability in the Lower Solar Atmosphere

We find evidence for the first observation of the parametric decay instability (PDI) in the lower solar atmosphere. Specifically, we find that the power spectrum of density fluctuations near the solar transition region resembles the power spectrum of the velocity fluctuations, but with the frequency axis scaled up by about a factor of two. These results are from an analysis of the Si IV lines observed by the Interface Region Imaging Spectrometer (IRIS) in the transition region of a polar coronal hole. We also find that the density fluctuations have radial velocity of about 75 km/s and that the velocity fluctuations are much faster with an estimated speed of 250 km/s, as is expected for sound waves and Alfvén waves, respectively, in the transition region. Theoretical calculations show that this frequency relationship is consistent with those expected from PDI for the plasma conditions of the observed region. These measurements suggest an interaction between sound waves and Alfvén waves in the transition region that is evidence for the parametric decay instability.

astro-ph.SR↗

arXiv2021

Sensitivity as a Complexity Measure for Sequence Classification Tasks

We introduce a theoretical framework for understanding and predicting the complexity of sequence classification tasks, using a novel extension of the theory of Boolean function sensitivity. The sensitivity of a function, given a distribution over input sequences, quantifies the number of disjoint subsets of the input sequence that can each be individually changed to change the output. We argue that standard sequence classification methods are biased towards learning low-sensitivity functions, so that tasks requiring high sensitivity are more difficult. To that end, we show analytically that simple lexical classifiers can only express functions of bounded sensitivity, and we show empirically that low-sensitivity functions are easier to learn for LSTMs. We then estimate sensitivity on 15 NLP tasks, finding that sensitivity is higher on challenging tasks collected in GLUE than on simple text classification tasks, and that sensitivity predicts the performance both of simple lexical classifiers and of vanilla BiLSTMs without pretrained contextualized embeddings. Within a task, sensitivity predicts which inputs are hard for such simple models. Our results suggest that the success of massively pretrained contextual representations stems in part because they provide representations from which information can be extracted by low-sensitivity decoders.

cs.CL↗

arXiv2020

RNNs can generate bounded hierarchical languages with optimal memory

Recurrent neural networks empirically generate natural language with high syntactic fidelity. However, their success is not well-understood theoretically. We provide theoretical insight into this success, proving in a finite-precision setting that RNNs can efficiently generate bounded hierarchical languages that reflect the scaffolding of natural language syntax. We introduce Dyck-($k$,$m$), the language of well-nested brackets (of $k$ types) and $m$-bounded nesting depth, reflecting the bounded memory needs and long-distance dependencies of natural language syntax. The best known results use $O(k^{\frac{m}{2}})$ memory (hidden units) to generate these languages. We prove that an RNN with $O(m \log k)$ hidden units suffices, an exponential reduction in memory, by an explicit construction. Finally, we show that no algorithm, even with unbounded computation, can suffice with $o(m \log k)$ hidden units.

cs.CL↗

arXiv2020

Critical Science Plan for the Daniel K. Inouye Solar Telescope (DKIST)

The Daniel K. Inouye Solar Telescope (DKIST) will revolutionize our ability to measure, understand and model the basic physical processes that control the structure and dynamics of the Sun and its atmosphere. The first-light DKIST images, released publicly on 29 January 2020, only hint at the extraordinary capabilities which will accompany full commissioning of the five facility instruments. With this Critical Science Plan (CSP) we attempt to anticipate some of what those capabilities will enable, providing a snapshot of some of the scientific pursuits that the Daniel K. Inouye Solar Telescope hopes to engage as start-of-operations nears. The work builds on the combined contributions of the DKIST Science Working Group (SWG) and CSP Community members, who generously shared their experiences, plans, knowledge and dreams. Discussion is primarily focused on those issues to which DKIST will uniquely contribute.

astro-ph.SR↗

arXiv2020

Theoretical Limitations of Self-Attention in Neural Sequence Models

Transformers are emerging as the new workhorse of NLP, showing great success across tasks. Unlike LSTMs, transformers process input sequences entirely through self-attention. Previous work has suggested that the computational capabilities of self-attention to process hierarchical structures are limited. In this work, we mathematically investigate the computational power of self-attention to model formal languages. Across both soft and hard attention, we show strong theoretical limitations of the computational abilities of self-attention, finding that it cannot model periodic finite-state languages, nor hierarchical structure, unless the number of layers or heads increases with input length. These limitations seem surprising given the practical success of self-attention and the prominent role assigned to hierarchical structure in linguistics, suggesting that natural language can be approximated well with models that are too weak for the formal languages typically assumed in theoretical linguistics.

cs.CL↗

arXiv2019

Wreath Products of Distributive Forest Algebras

It is an open problem whether definability in Propositional Dynamic Logic (PDL) on forests is decidable. Based on an algebraic characterization by Bojańczyk, et. al.,(2012) in terms of forest algebras, Straubing (2013) described an approach to PDL based on a k-fold iterated distributive law. A proof that all languages satisfying such a k-fold iterated distributive law are in PDL would settle decidability of PDL. We solve this problem in the case k=2: All languages recognized by forest algebras satisfying a 2-fold iterated distributive law are in PDL. Furthermore, we show that this class is decidable. This provides a novel nontrivial decidable subclass of PDL, and demonstrates the viability of the proposed approach to deciding PDL in general.

cs.FL↗

arXiv2019

Laboratory Calibrations of Fe XII-XIV Line-Intensity Ratios for Electron Density Diagnostics

We have used an electron beam ion trap to measure electron-density-diagnostic line-intensity ratios for extreme ultraviolet lines from F XII, XIII, and XIV at wavelengths of 185-205 255-276 Angstroms. These ratios can be used as density diagnostics for astrophysical spectra and are especially relevant to solar physics. We found that density diagnostics using the Fe XIII 196.53/202.04 and the Fe XIV 264.79/274.21 and 270.52A/274.21 line ratios are reliable using the atomic data calculated with the Flexible Atomic Code. On the other hand, we found a large discrepancy between the FAC theory and experiment for the commonly used Fe XII (186.85 + 186.88)/195.12 line ratio. These FAC theory calculations give similar results to the data tabulated in CHIANTI, which are commonly used to analyze solar observations. Our results suggest that the discrepancies seen between solar coronal density measurements using the Fe XII (186.85 + 186.88)/195.12 and Fe XIII 196.54/202.04 line ratios are likely due to issues with the atomic calculations for Fe XII.

astro-ph.SR↗

arXiv2019

Measured reduction in Alfvén wave energy propagating through longitudinal gradients scaled to match solar coronal holes

We have explored the effectiveness of a longitudinal gradient in Alfvén speed in reducing the energy of propagating Alfvén waves under conditions scaled to match solar coronal holes. The experiments were conducted in the Large Plasma Device at the University of California, Los Angeles. Our results show that the energy of the transmitted Alfvén wave decreases as the inhomogeneity parameter, $λ/L_{\rm A}$, increases. Here, $λ$ is the wavelength of the Alfvén wave and $L_{\rm A}$ is the scale length of Alfvén speed gradient. For gradients similar to those in coronal holes, the waves are observed to lose a factor of $\approx 5$ more energy than they do when propagating through a uniform plasma without a gradient. We have carried out further experiments and analyses to constrain the cause of wave energy reduction in the gradient. The loss of Alfvén wave energy from mode coupling is unlikely, as we have not detected any other modes. Contrary to theoretical expectations, the reduction in the energy of the transmitted wave is not accompanied by a detectable reflected wave. Nonlinear effects are ruled out as the amplitude of the initial wave is too small and the wave frequency well below the ion cyclotron frequency. Since the total energy must be conserved, it is possible that the lost wave energy is being deposited in the plasma. Further studies are needed to explore where the energy is going.

physics.plasm-ph↗

arXiv2019

Tabula nearly rasa: Probing the Linguistic Knowledge of Character-Level Neural Language Models Trained on Unsegmented Text

Recurrent neural networks (RNNs) have reached striking performance in many natural language processing tasks. This has renewed interest in whether these generic sequence processing devices are inducing genuine linguistic knowledge. Nearly all current analytical studies, however, initialize the RNNs with a vocabulary of known words, and feed them tokenized input during training. We present a multi-lingual study of the linguistic knowledge encoded in RNNs trained as character-level language models, on input data with word boundaries removed. These networks face a tougher and more cognitively realistic task, having to discover any useful linguistic unit from scratch based on input statistics. The results show that our "near tabula rasa" RNNs are mostly able to solve morphological, syntactic and semantic tasks that intuitively presuppose word-level knowledge, and indeed they learned, to some extent, to track word boundaries. Our study opens the door to speculations about the necessity of an explicit, rigid word lexicon in language learning and usage.

cs.CL↗

arXiv2019

Character-based Surprisal as a Model of Reading Difficulty in the Presence of Error

Intuitively, human readers cope easily with errors in text; typos, misspelling, word substitutions, etc. do not unduly disrupt natural reading. Previous work indicates that letter transpositions result in increased reading times, but it is unclear if this effect generalizes to more natural errors. In this paper, we report an eye-tracking study that compares two error types (letter transpositions and naturally occurring misspelling) and two error rates (10% or 50% of all words contain errors). We find that human readers show unimpaired comprehension in spite of these errors, but error words cause more reading difficulty than correct words. Also, transpositions are more difficult than misspellings, and a high error rate increases difficulty for all words, including correct ones. We then present a computational model that uses character-based (rather than traditional word-based) surprisal to account for these results. The model explains that transpositions are harder than misspellings because they contain unexpected letter combinations. It also explains the error rate effect: upcoming words are more difficultto predict when the context is degraded, leading to increased surprisal.

cs.CL↗