arXiv subjects

Heiner Kremer

Publications and source records attributed to Heiner Kremer.

8 recordsLinked to original sources

arXiv2025

Implicit Language Models are RNNs: Balancing Parallelization and Expressivity

State-space models (SSMs) and transformers dominate the language modeling landscape. However, they are constrained to a lower computational complexity than classical recurrent neural networks (RNNs), limiting their expressivity. In contrast, RNNs lack parallelization during training, raising fundamental questions about the trade off between parallelization and expressivity. We propose implicit SSMs, which iterate a transformation until convergence to a fixed point. Theoretically, we show that implicit SSMs implement the non-linear state-transitions of RNNs. Empirically, we find that only approximate fixed-point convergence suffices, enabling the design of a scalable training curriculum that largely retains parallelization, with full convergence required only for a small subset of tokens. Our approach demonstrates superior state-tracking capabilities on regular languages, surpassing transformers and SSMs. We further scale implicit SSMs to natural language reasoning tasks and pretraining of large-scale language models up to 1.3B parameters on 207B tokens representing, to our knowledge, the largest implicit model trained to date. Notably, our implicit models outperform their explicit counterparts on standard benchmarks. Our code is publicly available at http://github.com/microsoft/implicit_languagemodels .

cs.LG↗

arXiv2024

Geometry-Aware Instrumental Variable Regression

Instrumental variable (IV) regression can be approached through its formulation in terms of conditional moment restrictions (CMR). Building on variants of the generalized method of moments, most CMR estimators are implicitly based on approximating the population data distribution via reweightings of the empirical sample. While for large sample sizes, in the independent identically distributed (IID) setting, reweightings can provide sufficient flexibility, they might fail to capture the relevant information in presence of corrupted data or data prone to adversarial attacks. To address these shortcomings, we propose the Sinkhorn Method of Moments, an optimal transport-based IV estimator that takes into account the geometry of the data manifold through data-derivative information. We provide a simple plug-and-play implementation of our method that performs on par with related estimators in standard settings but improves robustness against data corruption and adversarial attacks.

cs.LG↗

arXiv2024

Functional Generalized Empirical Likelihood Estimation for Conditional Moment Restrictions

Important problems in causal inference, economics, and, more generally, robust machine learning can be expressed as conditional moment restrictions, but estimation becomes challenging as it requires solving a continuum of unconditional moment restrictions. Previous works addressed this problem by extending the generalized method of moments (GMM) to continuum moment restrictions. In contrast, generalized empirical likelihood (GEL) provides a more general framework and has been shown to enjoy favorable small-sample properties compared to GMM-based estimators. To benefit from recent developments in machine learning, we provide a functional reformulation of GEL in which arbitrary models can be leveraged. Motivated by a dual formulation of the resulting infinite dimensional optimization problem, we devise a practical method and explore its asymptotic properties. Finally, we provide kernel- and neural network-based implementations of the estimator, which achieve state-of-the-art empirical performance on two conditional moment restriction problems.

cs.LG↗

arXiv2023

Estimation Beyond Data Reweighting: Kernel Method of Moments

Moment restrictions and their conditional counterparts emerge in many areas of machine learning and statistics ranging from causal inference to reinforcement learning. Estimators for these tasks, generally called methods of moments, include the prominent generalized method of moments (GMM) which has recently gained attention in causal inference. GMM is a special case of the broader family of empirical likelihood estimators which are based on approximating a population distribution by means of minimizing a $φ$-divergence to an empirical distribution. However, the use of $φ$-divergences effectively limits the candidate distributions to reweightings of the data samples. We lift this long-standing limitation and provide a method of moments that goes beyond data reweighting. This is achieved by defining an empirical likelihood estimator based on maximum mean discrepancy which we term the kernel method of moments (KMM). We provide a variant of our estimator for conditional moment restrictions and show that it is asymptotically first-order optimal for such problems. Finally, we show that our method achieves competitive performance on several conditional moment restriction tasks.

cs.LG↗

arXiv2022

Quantifying the Effects of Contact Tracing, Testing, and Containment Measures in the Presence of Infection Hotspots

Multiple lines of evidence strongly suggest that infection hotspots, where a single individual infects many others, play a key role in the transmission dynamics of COVID-19. However, most of the existing epidemiological models fail to capture this aspect by neither representing the sites visited by individuals explicitly nor characterizing disease transmission as a function of individual mobility patterns. In this work, we introduce a temporal point process modeling framework that specifically represents visits to the sites where individuals get in contact and infect each other. Under our model, the number of infections caused by an infectious individual naturally emerges to be overdispersed. Using an efficient sampling algorithm, we demonstrate how to estimate the transmission rate of infectious individuals at the sites they visit and in their households using Bayesian optimization and longitudinal case data. Simulations using fine-grained and publicly available demographic data and site locations from Bern, Switzerland showcase the flexibility of our framework. To facilitate research and analyses of other cities and regions, we release an open-source implementation of our framework.

cs.LG↗

arXiv2022

Directed Acoustic Assembly in 3D

The creation of whole 3D objects in one shot is an ultimate goal for rapid prototyping, most notably biofabrication, where conventional methods are typically slow and apply mechanical or chemical stress on biological cells. Here, we demonstrate one-step assembly of matter to form compact 3D shapes using acoustic forces, which is enabled by the superposition of multiple holographic fields. The technique is contactless and shown to work with solid microparticles, hydrogel beads and biological cells inside standard labware. The structures can be fixed via gelation of the surrounding medium. In contrast to previous work, this approach handles matter with positive acoustic contrast and does not require opposing waves, supporting surfaces or scaffolds. We envision promising applications in tissue engineering and additive manufacturing.

cond-mat.soft↗

arXiv2022

Maximum Mean Discrepancy Distributionally Robust Nonlinear Chance-Constrained Optimization with Finite-Sample Guarantee

This paper is motivated by addressing open questions in distributionally robust chance-constrained programs (DRCCP) using the popular Wasserstein ambiguity sets. Specifically, the computational techniques for those programs typically place restrictive assumptions on the constraint functions and the size of the Wasserstein ambiguity sets is often set using costly cross-validation (CV) procedures or conservative measure concentration bounds. In contrast, we propose a practical DRCCP algorithm using kernel maximum mean discrepancy (MMD) ambiguity sets, which we term MMD-DRCCP, to treat general nonlinear constraints without using ad-hoc reformulation techniques. MMD-DRCCP can handle general nonlinear and non-convex constraints with a proven finite-sample constraint satisfaction guarantee of a dimension-independent $\mathcal{O}(\frac{1}{\sqrt{N}})$ rate, achievable by a practical algorithm. We further propose an efficient bootstrap scheme for constructing sharp MMD ambiguity sets in practice without resorting to CV. Our algorithm is validated numerically on a portfolio optimization problem and a tube-based distributionally robust model predictive control problem with non-convex constraints.

math.OC↗

arXiv2017

Nonequilibrium photon production in partonic transport simulations

We discuss the implementation of leading-order photon production in nonequilibrium partonic transport simulations. In this framework photons are produced by microscopic scatterings, where we include the exact matrix elements of Compton scattering, quark-antiquark annihilation, and bremsstrahlung processes. We show how the hard-thermal loop inspired screening of propagators has to be modified such that the microscopic production rate agrees well with the analytically known resummed leading-order rate. We model the complete quark-gluon plasma phase of heavy-ion collisions by using the partonic transport approach called the Boltzmann approach to multiparton scatterings (BAMPS), which solves the ultrarelativistic Boltzmann equation with Monte Carlo methods. We show photon spectra and elliptic flow of photons from BAMPS and discuss nonequilibrium effects. Due to the slow quark chemical equilibration in BAMPS, the yield is lower than the results from other groups; in turn we see a strong effect from scatterings of energetic jet-like partons with the medium. This nonequilibrium photon production can dominate the thermal emission, such that the spectra are harder and the photonic elliptic flow of the quark-gluon plasma becomes negative.

hep-ph↗