arXiv subjects

Aaron Smith

Publications and source records attributed to Aaron Smith.

At least 145 records · Page 8Linked to original sources

arXiv2018

82 Treebanks, 34 Models: Universal Dependency Parsing with Multi-Treebank Models

We present the Uppsala system for the CoNLL 2018 Shared Task on universal dependency parsing. Our system is a pipeline consisting of three components: the first performs joint word and sentence segmentation; the second predicts part-of- speech tags and morphological features; the third predicts dependency trees from words and tags. Instead of training a single parsing model for each treebank, we trained models with multiple treebanks for one language or closely related languages, greatly reducing the number of models. On the official test run, we ranked 7th of 27 teams for the LAS and MLAS metrics. Our system obtained the best scores overall for word segmentation, universal POS tagging, and morphological features.

cs.CL↗

arXiv2018

Does Hamiltonian Monte Carlo mix faster than a random walk on multimodal densities?

Hamiltonian Monte Carlo (HMC) is a very popular and generic collection of Markov chain Monte Carlo (MCMC) algorithms. One explanation for the popularity of HMC algorithms is their excellent performance as the dimension $d$ of the target becomes large: under conditions that are satisfied for many common statistical models, optimally-tuned HMC algorithms have a running time that scales like $d^{0.25}$. In stark contrast, the running time of the usual Random-Walk Metropolis (RWM) algorithm, optimally tuned, scales like $d$. This superior scaling of the HMC algorithm with dimension is attributed to the fact that it, unlike RWM, incorporates the gradient information in the proposal distribution. In this paper, we investigate a different scaling question: does HMC beat RWM for highly $\textit{multimodal}$ targets? We find that the answer is often $\textit{no}$. We compute the spectral gaps for both the algorithms for a specific class of multimodal target densities, and show that they are identical. The key reason is that, within one mode, the gradient is effectively ignorant about other modes, thus negating the advantage the HMC algorithm enjoys in unimodal targets. We also give heuristic arguments suggesting that the above observation may hold quite generally. Our main tool for answering this question is a novel simple formula for the conductance of HMC using Liouville's theorem. This result allows us to compute the spectral gap of HMC algorithms, for both the classical HMC with isotropic momentum and the recent Riemannian HMC, for multimodal targets.

math.PR↗

arXiv2018

Simple Conditions for Metastability of Continuous Markov Chains

A family $\{Q_β\}_{β\geq 0}$ of Markov chains is said to exhibit $\textit{metastable mixing}$ with $\textit{modes}$ $S_β^{(1)},\ldots,S_β^{(k)}$ if its spectral gap (or some other mixing property) is very close to the worst conductance $\min(Φ_β(S_β^{(1)}), \ldots, Φ_β(S_β^{(k)}))$ of its modes. We give simple sufficient conditions for a family of Markov chains to exhibit metastability in this sense, and verify that these conditions hold for a prototypical Metropolis-Hastings chain targeting a mixture distribution. Our work differs from existing work on metastability in that, for the class of examples we are interested in, it gives an asymptotically exact formula for the spectral gap (rather than a bound that can be very far from sharp) while at the same time giving technical conditions that are easier to verify for many statistical examples. Our bounds from this paper are used in a companion paper to compare the mixing times of the Hamiltonian Monte Carlo algorithm and a random walk algorithm for multimodal target distributions.

math.PR↗

arXiv2018

An Investigation of the Interactions Between Pre-Trained Word Embeddings, Character Models and POS Tags in Dependency Parsing

We provide a comprehensive analysis of the interactions between pre-trained word embeddings, character models and POS tags in a transition-based dependency parser. While previous studies have shown POS information to be less important in the presence of character models, we show that in fact there are complex interactions between all three techniques. In isolation each produces large improvements over a baseline system using randomly initialised word embeddings only, but combining them quickly leads to diminishing returns. We categorise words by frequency, POS tag and language in order to systematically investigate how each of the techniques affects parsing quality. For many word categories, applying any two of the three techniques is almost as good as the full combined system. Character models tend to be more important for low-frequency open-class words, especially in morphologically rich languages, while POS tags can help disambiguate high-frequency function words. We also show that large character embedding sizes help even for languages with small character sets, especially in morphologically rich languages.

cs.CL↗

arXiv2018

Fast Mixing of Metropolis-Hastings with Unimodal Targets

A well-known folklore result in the MCMC community is that the Metropolis-Hastings algorithm mixes quickly for any unimodal target, as long as the tails are not too heavy. Although we've heard this fact stated many times in conversation, we are not aware of any quantitative statement of this result in the literature, and we are not aware of any quick derivation from well-known results. The present paper patches this small gap in the literature, providing a generic bound based on the popular "drift-and-minorization" framework of Rosenthal (1995). Our main contribution is to study two sublevel sets of the Lyapunov function and use path arguments in order to obtain a sharper general bound than what can typically be obtained from multistep minorization arguments.

math.PR↗

arXiv2018

Discrete diffusion Lyman-alpha radiative transfer

Due to its accuracy and generality, Monte Carlo radiative transfer (MCRT) has emerged as the prevalent method for Ly$α$ radiative transfer in arbitrary geometries. The standard MCRT encounters a significant efficiency barrier in the high optical depth, diffusion regime. Multiple acceleration schemes have been developed to improve the efficiency of MCRT but the noise from photon packet discretization remains a challenge. The discrete diffusion Monte Carlo (DDMC) scheme has been successfully applied in state-of-the-art radiation hydrodynamics (RHD) simulations. Still, the established framework is not optimal for resonant line transfer. Inspired by the DDMC paradigm, we present a novel extension to resonant DDMC (rDDMC) in which diffusion in space and frequency are treated on equal footing. We explore the robustness of our new method and demonstrate a level of performance that justifies incorporating the method into existing Ly$α$ codes. We present computational speedups of $\sim 10^2$-$10^6$ relative to contemporary MCRT implementations with schemes that skip scattering in the core of the line profile. This is because the rDDMC runtime scales with the spatial and frequency resolution rather than the number of scatterings - the latter is typically $\propto τ_0$ for static media, or $\propto (a τ_0)^{2/3}$ with core-skipping. We anticipate new frontiers in which on-the-fly Ly$α$ radiative transfer calculations are feasible in 3D RHD. More generally, rDDMC is transferable to any computationally demanding problem amenable to a Fokker-Planck approximation of frequency redistribution.

astro-ph.CO↗

arXiv2018

Parser Training with Heterogeneous Treebanks

How to make the most of multiple heterogeneous treebanks when training a monolingual dependency parser is an open question. We start by investigating previously suggested, but little evaluated, strategies for exploiting multiple treebanks based on concatenating training sets, with or without fine-tuning. We go on to propose a new method based on treebank embeddings. We perform experiments for several languages and show that in many cases fine-tuning and treebank embeddings lead to substantial improvements over single treebanks or concatenation, with average gains of 2.0--3.5 LAS points. We argue that treebank embeddings should be preferred due to their conceptual simplicity, flexibility and extensibility.

cs.CL↗

arXiv2018

Towards an Area-Efficient Implementation of a High ILP EDGE Soft Processor

In-order scalar RISC architectures have been the dominant paradigm in FPGA soft processor design for twenty years. Prior out-of-order superscalar implementations have not exhibited competitive area or absolute performance. This paper describes a new way to build fast and area-efficient out-of-order superscalar soft processors by utilizing an Explicit Data Graph Execution (EDGE) instruction set architecture. By carefully mapping the EDGE microarchitecture, and in particular, its dataflow instruction scheduler, we demonstrate the feasibility of an out-of-order FPGA architecture. Two scheduler design alternatives are compared.

cs.AR↗

arXiv2017

Parallel local approximation MCMC for expensive models

Performing Bayesian inference via Markov chain Monte Carlo (MCMC) can be exceedingly expensive when posterior evaluations invoke the evaluation of a computationally expensive model, such as a system of partial differential equations. In recent work [Conrad et al. JASA 2016, arXiv:1402.1694], we described a framework for constructing and refining local approximations of such models during an MCMC simulation. These posterior--adapted approximations harness regularity of the model to reduce the computational cost of inference while preserving asymptotic exactness of the Markov chain. Here we describe two extensions of that work. First, we prove that samplers running in parallel can collaboratively construct a shared posterior approximation while ensuring ergodicity of each associated chain, providing a novel opportunity for exploiting parallel computation in MCMC. Second, focusing on the Metropolis--adjusted Langevin algorithm, we describe how a proposal distribution can successfully employ gradients and other relevant information extracted from the approximation. We investigate the practical performance of our strategies using two challenging inference problems, the first in subsurface hydrology and the second in glaciology. Using local approximations constructed via parallel chains, we successfully reduce the run time needed to characterize the posterior distributions in these problems from days to hours and from months to days, respectively, dramatically improving the tractability of Bayesian inference.

stat.CO↗

arXiv2017

Rapid Mixing of Geodesic Walks on Manifolds with Positive Curvature

We introduce a Markov chain for sampling from the uniform distribution on a Riemannian manifold $\mathcal{M}$, which we call the $\textit{geodesic walk}$. We prove that the mixing time of this walk on any manifold with positive sectional curvature $C_{x}(u,v)$ bounded both above and below by $0 < \mathfrak{m}_{2} \leq C_{x}(u,v) \leq \mathfrak{M}_2 < \infty$ is $\mathcal{O}^*\left(\frac{\mathfrak{M}_2}{\mathfrak{m}_2}\right)$. In particular, this bound on the mixing time does not depend explicitly on the dimension of the manifold. In the special case that $\mathcal{M}$ is the boundary of a convex body, we give an explicit and computationally tractable algorithm for approximating the exact geodesic walk. As a consequence, we obtain an algorithm for sampling uniformly from the surface of a convex body that has running time bounded solely in terms of the curvature of the body.

math.PR↗

arXiv2017

Radiative effects during the assembly of direct collapse black holes

We perform a post-processing radiative feedback analysis on a 3D ab initio cosmological simulation of an atomic cooling halo under the direct collapse black hole (DCBH) scenario. We maintain the spatial resolution of the simulation by incorporating native ray-tracing on unstructured mesh data, including Monte Carlo Lyman-alpha (Lyα) radiative transfer. DCBHs are born in gas-rich, metal-poor environments with the possibility of Compton-thick conditions, $N_H \gtrsim 10^{24} {\rm cm}^{-2}$. Therefore, the surrounding gas is capable of experiencing the full impact of the bottled-up radiation pressure. In particular, we find that multiple scattering of Lyα photons provides an important source of mechanical feedback after the gas in the sub-parsec region becomes partially ionized, avoiding the bottleneck of destruction via the two-photon emission mechanism. We provide detailed discussion of the simulation environment, expansion of the ionization front, emission and escape of Lyα radiation, and Compton scattering. A sink particle prescription allows us to extract approximate limits on the post-formation evolution of the radiative feedback. Fully coupled Lyα radiation hydrodynamics will be crucial to consider in future DCBH simulations.

astro-ph.GA↗

arXiv2017

Rapid Mixing of Hamiltonian Monte Carlo on Strongly Log-Concave Distributions

We obtain several quantitative bounds on the mixing properties of the Hamiltonian Monte Carlo (HMC) algorithm for a strongly log-concave target distribution $π$ on $\mathbb{R}^{d}$, showing that HMC mixes quickly in this setting. One of our main results is a dimension-free bound on the mixing of an "ideal" HMC chain, which is used to show that the usual leapfrog implementation of HMC can sample from $π$ using only $\mathcal{O}(d^{\frac{1}{4}})$ gradient evaluations. This dependence on dimension is sharp, and our results significantly extend and improve previous quantitative bounds on the mixing of HMC.

math.PR↗

arXiv2017

Mixing Times for a Constrained Ising Process on the Two-Dimensional Torus at Low Density

We study a kinetically constrained Ising process (KCIP) associated with a graph $G$ and density parameter $p$; this process is an interacting particle system with state space $\{ 0, 1 \}^{G}$, the location of the particles. The `constraint' in the name of the process refers to the rule that a vertex cannot change its state unless it has at least one neighbour in state `1'. The KCIP has been proposed by statistical physicists as a model for the glass transition. In this note, we study the mixing time of a KCIP on the 2-dimensional torus $G = \mathbb{Z}_{L}^{2}$ in the low-density regime $p = \frac{c}{L^{2}}$ for arbitrary $0 < c < \infty$, extending our previous results for the analogous process on the torus $\mathbb{Z}_{L}^{d}$ in dimension $d \geq 3$. Our general approach is similar, but the extension requires more delicate bounds on the behaviour of the process at intermediate densities.

math.PR↗

arXiv2017

The first supermassive black holes

We briefly review the historical development of the ideas regarding the first supermassive black hole seeds, the physics of their formation and radiative feedback, recent theoretical and observational progress, and our outlook for the future.

astro-ph.GA↗

arXiv2017

MCMC for Imbalanced Categorical Data

Many modern applications collect highly imbalanced categorical data, with some categories relatively rare. Bayesian hierarchical models combat data sparsity by borrowing information, while also quantifying uncertainty. However, posterior computation presents a fundamental barrier to routine use; a single class of algorithms does not work well in all settings and practitioners waste time trying different types of MCMC approaches. This article was motivated by an application to quantitative advertising in which we encountered extremely poor computational performance for common data augmentation MCMC algorithms but obtained excellent performance for adaptive Metropolis. To obtain a deeper understanding of this behavior, we give strong theory results on computational complexity in an infinitely imbalanced asymptotic regime. Our results show computational complexity of Metropolis is logarithmic in sample size, while data augmentation is polynomial in sample size. The root cause of poor performance of data augmentation is a discrepancy between the rates at which the target density and MCMC step sizes concentrate. In general, MCMC algorithms that have a similar discrepancy will fail in large samples - a result with substantial practical impact.

math.ST↗

arXiv2017

Bounds on Lifting Continuous Markov Chains to Speed Up Mixing

It is often possible to speed up the mixing of a Markov chain $\{ X_{t} \}_{t \in \mathbb{N}}$ on a state space $Ω$ by \textit{lifting}, that is, running a more efficient Markov chain $\{ \hat{X}_{t} \}_{t \in \mathbb{N}}$ on a larger state space $\hatΩ \supset Ω$ that projects to $\{ X_{t} \}_{t \in \mathbb{N}}$ in a certain sense. In [CLP99], Chen, Lov{á}sz and Pak prove that for Markov chains on finite state spaces, the mixing time of any lift of a Markov chain is at least the square root of the mixing time of the original chain, up to a factor that depends on the stationary measure. Unfortunately, this extra factor makes the bound in [CLP99] very loose for Markov chains on large state spaces and useless for Markov chains on continuous state spaces. In this paper, we develop an extension of the evolving set method that allows us to refine this extra factor and find bounds for Markov chains on continuous state spaces that are analogous to the bounds in [CLP99]. These bounds also allow us to improve on the bounds in [CLP99] for some chains on finite state spaces.

math.PR↗

arXiv2016

Lyman-alpha radiation hydrodynamics of galactic winds before cosmic reionization

The dynamical impact of Lyman-alpha (Lyα) radiation pressure on galaxy formation depends on the rate and duration of momentum transfer between Lyα photons and neutral hydrogen gas. Although photon trapping has the potential to multiply the effective force, ionizing radiation from stellar sources may relieve the Lyα pressure before appreciably affecting the kinematics of the host galaxy or efficiently coupling Lyα photons to the outflow. We present self-consistent Lyα radiation-hydrodynamics simulations of high-$z$ galaxy environments by coupling the Cosmic Lyα Transfer code (COLT) with spherically symmetric Lagrangian frame hydrodynamics. The accurate but computationally expensive Monte-Carlo radiative transfer calculations are feasible under the one-dimensional approximation. The initial starburst drives an expanding shell of gas from the centre and in certain cases Lyα feedback significantly enhances the shell velocity. Radiative feedback alone is capable of ejecting baryons into the intergalactic medium (IGM) for protogalaxies with a virial mass of $M_{\rm vir} \lesssim 10^8~{\rm M}_\odot$. We compare the Lyα signatures of Population III stars with $10^5$ K blackbody emission to that of direct collapse black holes with a nonthermal Compton-thick spectrum and find substantial differences if the Lyα spectra are shaped by gas pushed by Lyα radiation-driven winds. For both sources, the flux emerging from the galaxy is reprocessed by the IGM such that the observed Lyα luminosity is reduced significantly and the time-averaged velocity offset of the Lyα peak is shifted redward.

astro-ph.GA↗

arXiv2016

Parallel Markov Chain Monte Carlo via Spectral Clustering

As it has become common to use many computer cores in routine applications, finding good ways to parallelize popular algorithms has become increasingly important. In this paper, we present a parallelization scheme for Markov chain Monte Carlo (MCMC) methods based on spectral clustering of the underlying state space, generalizing earlier work on parallelization of MCMC methods by state space partitioning. We show empirically that this approach speeds up MCMC sampling for multimodal distributions and that it can be usefully applied in greater generality than several related algorithms. Our algorithm converges under reasonable conditions to an `optimal' MCMC algorithm. We also show that our approach can be asymptotically far more efficient than naive parallelization, even in situations such as completely flat target distributions where no unique optimal algorithm exists. Finally, we combine theoretical and empirical bounds to provide practical guidance on the choice of tuning parameters.

stat.ME↗