arXiv subjects

Michael Eickenberg

Publications and source records attributed to Michael Eickenberg.

At least 19 recordsLinked to original sources

arXiv2025

Surfer 2: The Next Generation of Cross-Platform Computer Use Agents

Building agents that generalize across web, desktop, and mobile environments remains an open challenge, as prior systems rely on environment-specific interfaces that limit cross-platform deployment. We introduce Surfer 2, a unified architecture operating purely from visual observations that achieves state-of-the-art performance across all three environments. Surfer 2 integrates hierarchical context management, decoupled planning and execution, and self-verification with adaptive recovery, enabling reliable operation over long task horizons. Our system achieves 97.1% accuracy on WebVoyager, 69.6% on WebArena, 60.1% on OSWorld, and 87.1% on AndroidWorld, outperforming all prior systems without task-specific fine-tuning. With multiple attempts, Surfer 2 exceeds human performance on all benchmarks. These results demonstrate that systematic orchestration amplifies foundation model capabilities and enables general-purpose computer control through visual interaction alone, while calling for a next-generation vision language model to achieve Pareto-optimal cost-efficiency.

cs.AI↗

arXiv2025

Mitigating Model Misspecification in Simulation-Based Inference for Galaxy Clustering

Simulation-based inference (SBI) has become an important tool in cosmology for extracting additional information from observational data using simulations. However, all cosmological simulations are approximations of the actual universe, and SBI methods can be sensitive to model misspecification - particularly when the observational data lie outside the support of the training distribution. We present a method to improve the robustness of cosmological analyses under such conditions. Our approach first identifies and discards components of the summary statistics that exhibit inconsistency across related simulators, then learns a transformation that brings the observation back within the support of the training distribution. We apply our method in the context of a recent SimBIG SBI galaxy clustering analysis using the wavelet scattering transform (WST) summary statistic. The original analysis struggled to produce robust constraints for certain subsets of WST coefficients, where the observational data appeared out-of-distribution (OOD) relative to the training data. We show that our method enables robust cosmological inference and resolves OOD issues, while preserving most of the constraining power. In particular, the improved SimBIG WST analysis yields $Λ$CDM constraints of $Ω_m = 0.32^{+0.02}_{-0.02}$ and $σ_8 = 0.80^{+0.02}_{-0.02}$, which are respectively $1.4\times$ and $3.1\times$ tighter than those from a standard perturbation-theory-based power spectrum analysis, confirming the significant information gain of WST summary statistics. The proposed method is easily applicable to other cosmological SBI contexts and represents a step toward more robust SBI pipelines.

astro-ph.CO↗

arXiv2025

Surfer-H Meets Holo1: Cost-Efficient Web Agent Powered by Open Weights

We present Surfer-H, a cost-efficient web agent that integrates Vision-Language Models (VLM) to perform user-defined tasks on the web. We pair it with Holo1, a new open-weight collection of VLMs specialized in web navigation and information extraction. Holo1 was trained on carefully curated data sources, including open-access web content, synthetic examples, and self-produced agentic data. Holo1 tops generalist User Interface (UI) benchmarks as well as our new web UI localization benchmark, WebClick. When powered by Holo1, Surfer-H achieves a 92.2% state-of-the-art performance on WebVoyager, striking a Pareto-optimal balance between accuracy and cost-efficiency. To accelerate research advancement in agentic systems, we are open-sourcing both our WebClick evaluation dataset and the Holo1 model weights.

cs.AI↗

arXiv2025

PETRA: Parallel End-to-end Training with Reversible Architectures

Reversible architectures have been shown to be capable of performing on par with their non-reversible architectures, being applied in deep learning for memory savings and generative modeling. In this work, we show how reversible architectures can solve challenges in parallelizing deep model training. We introduce PETRA, a novel alternative to backpropagation for parallelizing gradient computations. PETRA facilitates effective model parallelism by enabling stages (i.e., a set of layers) to compute independently on different devices, while only needing to communicate activations and gradients between each other. By decoupling the forward and backward passes and keeping a single updated version of the parameters, the need for weight stashing is also removed. We develop a custom autograd-like training framework for PETRA, and we demonstrate its effectiveness on CIFAR-10, ImageNet32, and ImageNet, achieving competitive accuracies comparable to backpropagation using ResNet-18, ResNet-34, and ResNet-50 models.

cs.LG↗

arXiv2024

xVal: A Continuous Numerical Tokenization for Scientific Language Models

Due in part to their discontinuous and discrete default encodings for numbers, Large Language Models (LLMs) have not yet been commonly used to process numerically-dense scientific datasets. Rendering datasets as text, however, could help aggregate diverse and multi-modal scientific data into a single training corpus, thereby potentially facilitating the development of foundation models for science. In this work, we introduce xVal, a strategy for continuously tokenizing numbers within language models that results in a more appropriate inductive bias for scientific applications. By training specially-modified language models from scratch on a variety of scientific datasets formatted as text, we find that xVal generally outperforms other common numerical tokenization strategies on metrics including out-of-distribution generalization and computational efficiency.

stat.ML↗

arXiv2024

Multiple Physics Pretraining for Physical Surrogate Models

We introduce multiple physics pretraining (MPP), an autoregressive task-agnostic pretraining approach for physical surrogate modeling of spatiotemporal systems with transformers. In MPP, rather than training one model on a specific physical system, we train a backbone model to predict the dynamics of multiple heterogeneous physical systems simultaneously in order to learn features that are broadly useful across systems and facilitate transfer. In order to learn effectively in this setting, we introduce a shared embedding and normalization strategy that projects the fields of multiple systems into a shared embedding space. We validate the efficacy of our approach on both pretraining and downstream tasks over a broad fluid mechanics-oriented benchmark. We show that a single MPP-pretrained transformer is able to match or outperform task-specific baselines on all pretraining sub-tasks without the need for finetuning. For downstream tasks, we demonstrate that finetuning MPP-trained models results in more accurate predictions across multiple time-steps on systems with previously unseen physical components or higher dimensional systems compared to training from scratch or finetuning pretrained video foundation models. We open-source our code and model weights trained at multiple scales for reproducibility.

cs.LG↗

arXiv2024

Galaxy Clustering Analysis with SimBIG and the Wavelet Scattering Transform

The non-Gaussisan spatial distribution of galaxies traces the large-scale structure of the Universe and therefore constitutes a prime observable to constrain cosmological parameters. We conduct Bayesian inference of the $Λ$CDM parameters $Ω_m$, $Ω_b$, $h$, $n_s$, and $σ_8$ from the BOSS CMASS galaxy sample by combining the wavelet scattering transform (WST) with a simulation-based inference approach enabled by the ${\rm S{\scriptsize IM}BIG}$ forward model. We design a set of reduced WST statistics that leverage symmetries of redshift-space data. Posterior distributions are estimated with a conditional normalizing flow trained on 20,000 simulated ${\rm S{\scriptsize IM}BIG}$ galaxy catalogs with survey realism. We assess the accuracy of the posterior estimates using simulation-based calibration and quantify generalization and robustness to the change of forward model using a suite of 2,000 test simulations. When probing scales down to $k_{\rm max}=0.5~h/\text{Mpc}$, we are able to derive accurate posterior estimates that are robust to the change of forward model for all parameters, except $σ_8$. We mitigate the robustness issues with $σ_8$ by removing the WST coefficients that probe scales smaller than $k \sim 0.3~h/\text{Mpc}$. Applied to the BOSS CMASS sample, our WST analysis yields seemingly improved constraints obtained from a standard PT-based power spectrum analysis with $k_{\rm max}=0.25~h/\text{Mpc}$ for all parameters except $h$. However, we still raise concerns on these results. The observational predictions significantly vary across different normalizing flow architectures, which we interpret as a form of model misspecification. This highlights a key challenge for forward modeling approaches when using summary statistics that are sensitive to detailed model-specific or observational imprints on galaxy clustering.

astro-ph.CO↗

arXiv2024

AstroCLIP: A Cross-Modal Foundation Model for Galaxies

We present AstroCLIP, a single, versatile model that can embed both galaxy images and spectra into a shared, physically meaningful latent space. These embeddings can then be used - without any model fine-tuning - for a variety of downstream tasks including (1) accurate in-modality and cross-modality semantic similarity search, (2) photometric redshift estimation, (3) galaxy property estimation from both images and spectra, and (4) morphology classification. Our approach to implementing AstroCLIP consists of two parts. First, we embed galaxy images and spectra separately by pretraining separate transformer-based image and spectrum encoders in self-supervised settings. We then align the encoders using a contrastive loss. We apply our method to spectra from the Dark Energy Spectroscopic Instrument and images from its corresponding Legacy Imaging Survey. Overall, we find remarkable performance on all downstream tasks, even relative to supervised baselines. For example, for a task like photometric redshift prediction, we find similar performance to a specifically-trained ResNet18, and for additional tasks like physical property estimation (stellar mass, age, metallicity, and sSFR), we beat this supervised baseline by 19\% in terms of $R^2$. We also compare our results to a state-of-the-art self-supervised single-modal model for galaxy images, and find that our approach outperforms this benchmark by roughly a factor of two on photometric redshift estimation and physical property prediction in terms of $R^2$, while remaining roughly in-line in terms of morphology classification. Ultimately, our approach represents the first cross-modal self-supervised model for galaxies, and the first self-supervised transformer-based architectures for galaxy images and spectra.

astro-ph.IM↗

arXiv2024

MoMo: Momentum Models for Adaptive Learning Rates

Training a modern machine learning architecture on a new task requires extensive learning-rate tuning, which comes at a high computational cost. Here we develop new Polyak-type adaptive learning rates that can be used on top of any momentum method, and require less tuning to perform well. We first develop MoMo, a Momentum Model based adaptive learning rate for SGD-M (stochastic gradient descent with momentum). MoMo uses momentum estimates of the losses and gradients sampled at each iteration to build a model of the loss function. Our model makes use of any known lower bound of the loss function by using truncation, e.g. most losses are lower-bounded by zero. The model is then approximately minimized at each iteration to compute the next step. We show how MoMo can be used in combination with any momentum-based method, and showcase this by developing MoMo-Adam, which is Adam with our new model-based adaptive learning rate. We show that MoMo attains a $\mathcal{O}(1/\sqrt{K})$ convergence rate for convex problems with interpolation, needing knowledge of no problem-specific quantities other than the optimal value. Additionally, for losses with unknown lower bounds, we develop on-the-fly estimates of a lower bound, that are incorporated in our model. We show that MoMo and MoMo-Adam improve over SGD-M and Adam in terms of robustness to hyperparameter tuning for training image classifiers on MNIST, CIFAR, and Imagenet, for recommender systems on Criteo, for a transformer model on the translation task IWSLT14, and for a diffusion model.

cs.LG↗

arXiv2024

From Feature Visualization to Visual Circuits: Effect of Adversarial Model Manipulation

Understanding the inner working functionality of large-scale deep neural networks is challenging yet crucial in several high-stakes applications. Mechanistic inter- pretability is an emergent field that tackles this challenge, often by identifying human-understandable subgraphs in deep neural networks known as circuits. In vision-pretrained models, these subgraphs are usually interpreted by visualizing their node features through a popular technique called feature visualization. Recent works have analyzed the stability of different feature visualization types under the adversarial model manipulation framework. This paper starts by addressing limitations in existing works by proposing a novel attack called ProxPulse that simultaneously manipulates the two types of feature visualizations. Surprisingly, when analyzing these attacks under the umbrella of visual circuits, we find that visual circuits show some robustness to ProxPulse. We, therefore, introduce a new attack based on ProxPulse that unveils the manipulability of visual circuits, shedding light on their lack of robustness. The effectiveness of these attacks is validated using pre-trained AlexNet and ResNet-50 models on ImageNet.

cs.CV↗

arXiv2024

Contextual Counting: A Mechanistic Study of Transformers on a Quantitative Task

Transformers have revolutionized machine learning across diverse domains, yet understanding their behavior remains crucial, particularly in high-stakes applications. This paper introduces the contextual counting task, a novel toy problem aimed at enhancing our understanding of Transformers in quantitative and scientific contexts. This task requires precise localization and computation within datasets, akin to object detection or region-based scientific analysis. We present theoretical and empirical analysis using both causal and non-causal Transformer architectures, investigating the influence of various positional encodings on performance and interpretability. In particular, we find that causal attention is much better suited for the task, and that no positional embeddings lead to the best accuracy, though rotary embeddings are competitive and easier to train. We also show that out of distribution performance is tightly linked to which tokens it uses as a bias term.

cs.LG↗

arXiv2024

Channel-Selective Normalization for Label-Shift Robust Test-Time Adaptation

Deep neural networks have useful applications in many different tasks, however their performance can be severely affected by changes in the data distribution. For example, in the biomedical field, their performance can be affected by changes in the data (different machines, populations) between training and test datasets. To ensure robustness and generalization to real-world scenarios, test-time adaptation has been recently studied as an approach to adjust models to a new data distribution during inference. Test-time batch normalization is a simple and popular method that achieved compelling performance on domain shift benchmarks. It is implemented by recalculating batch normalization statistics on test batches. Prior work has focused on analysis with test data that has the same label distribution as the training data. However, in many practical applications this technique is vulnerable to label distribution shifts, sometimes producing catastrophic failure. This presents a risk in applying test time adaptation methods in deployment. We propose to tackle this challenge by only selectively adapting channels in a deep network, minimizing drastic adaptation that is sensitive to label shifts. Our selection scheme is based on two principles that we empirically motivate: (1) later layers of networks are more sensitive to label shift (2) individual features can be sensitive to specific classes. We apply the proposed technique to three classification tasks, including CIFAR10-C, Imagenet-C, and diagnosis of fatty liver, where we explore both covariate and label distribution shifts. We find that our method allows to bring the benefits of TTA while significantly reducing the risk of failure common in other methods, while being robust to choice in hyperparameters.

cs.CV↗

arXiv2024

{\sc SimBIG}: Cosmological Constraints using Simulation-Based Inference of Galaxy Clustering with Marked Power Spectra

We present the first $Λ$CDM cosmological analysis performed on a galaxy survey using marked power spectra. The marked power spectrum is the two-point function of a marked field, where galaxies are weighted by a function that depends on their local density. The presence of the mark leads these statistics to contain higher-order information of the original galaxy field, making them a good candidate to exploit the non-Gaussian information of a galaxy catalog. In this work we make use of \simbig, a forward modeling framework for galaxy clustering analyses, and perform simulation-based inference using normalizing flows to infer the posterior distribution of the $Λ$CDM cosmological parameters. We consider different mark configurations (ways to weight the galaxy field) and deploy them in the \simbig~pipeline to analyze the corresponding marked power spectra measured from a subset of the BOSS galaxy sample. We analyze the redshift-space mark power spectra decomposed in $\ell = 0, 2, 4$ multipoles and include scales up to the non-linear regime. Among the various mark configurations considered, the ones that give the most stringent cosmological constraints produce posterior median and $68\%$ confidence limits on the growth of structure parameters equal to $Ω_m=0.273^{+0.040}_{-0.030}$ and $σ_8=0.777^{+0.077}_{-0.071}$. Compared to a perturbation theory analysis using the power spectrum of the same dataset, the \simbig~marked power spectra constraints on $σ_8$ are up to $1.2\times$ tighter, while no improvement is seen for the other cosmological parameters.

astro-ph.CO↗

arXiv2024

Statistical Component Separation for Targeted Signal Recovery in Noisy Mixtures

Separating signals from an additive mixture may be an unnecessarily hard problem when one is only interested in specific properties of a given signal. In this work, we tackle simpler "statistical component separation" problems that focus on recovering a predefined set of statistical descriptors of a target signal from a noisy mixture. Assuming access to samples of the noise process, we investigate a method devised to match the statistics of the solution candidate corrupted by noise samples with those of the observed mixture. We first analyze the behavior of this method using simple examples with analytically tractable calculations. Then, we apply it in an image denoising context employing 1) wavelet-based descriptors, 2) ConvNet-based descriptors on astrophysics and ImageNet data. In the case of 1), we show that our method better recovers the descriptors of the target data than a standard denoising method in most situations. Additionally, despite not constructed for this purpose, it performs surprisingly well in terms of peak signal-to-noise ratio on full signal reconstruction. In comparison, representation 2) appears less suitable for image denoising. Finally, we extend this method by introducing a diffusive stepwise algorithm which gives a new perspective to the initial method and leads to promising results for image denoising under specific circumstances.

stat.ML↗

arXiv2024

${\rm S{\scriptsize IM}BIG}$: Cosmological Constraints from the Redshift-Space Galaxy Skew Spectra

Extracting the non-Gaussian information of the cosmic large-scale structure (LSS) is vital in unlocking the full potential of the rich datasets from the upcoming stage-IV galaxy surveys. Galaxy skew spectra serve as efficient beyond-two-point statistics, encapsulating essential bispectrum information with computational efficiency akin to power spectrum analysis. This paper presents the first cosmological constraints from analyzing the full set of redshift-space galaxy skew spectra of the data from the SDSS-III BOSS, accessing cosmological information down to nonlinear scales. Employing the ${\rm S{\scriptsize IM}BIG}$ forward modeling framework and simulation-based inference via normalizing flows, we analyze the CMASS-SGC sub-sample, which constitute approximately 10\% of the full BOSS data. Analyzing the scales up to $k_{\rm max}=0.5 \, {\rm Mpc}^{-1}h$, we find that the skew spectra improve the constraints on $Ω_{\rm m}, Ω_{\rm b}, h$, and $n_s$ by 34\%, 35\%, 18\%, 10\%, respectively, compared to constraints from previous ${\rm S{\scriptsize IM}BIG}$ power spectrum multipoles analysis, yielding $Ω_{\rm m}=0.288^{+0.024}_{-0.034}$, $Ω_{\rm b}= 0.043^{+0.005}_{-0.007}$, $h=0.759^{+0.104}_{-0.050}$, $n_{\rm s} = 0.918^{+0.041}_{-0.090}$ (at 68\% confidence limit). On the other hand, the constraints on $σ_8$ are weaker than from the power spectrum. Including the Big Bang Nucleosynthesis (BBN) prior on baryon density reduces the uncertainty on the Hubble parameter further, achieving $h=0.750^{+0.034}_{-0.032}$, which is a 38\% improvement over the constraint from the power spectrum with the same prior. Compared to the ${\rm S{\scriptsize IM}BIG}$ bispectrum (monopole) analysis, skew spectra offer comparable constraints on larger scales ($k_{\rm max}<0.3\, {\rm Mpc}^{-1}h$) for most parameters except for $σ_8$.

astro-ph.CO↗

arXiv2023

${\rm S{\scriptsize IM}BIG}$: The First Cosmological Constraints from the Non-Linear Galaxy Bispectrum

We present the first cosmological constraints from analyzing higher-order galaxy clustering on non-linear scales. We use ${\rm S{\scriptsize IM}BIG}$, a forward modeling framework for galaxy clustering analyses that employs simulation-based inference to perform highly efficient cosmological inference using normalizing flows. It leverages the predictive power of high-fidelity simulations and robustly extracts cosmological information from regimes inaccessible with current standard analyses. In this work, we apply ${\rm S{\scriptsize IM}BIG}$ to a subset of the BOSS galaxy sample and analyze the redshift-space bispectrum monopole, $B_0(k_1, k_2, k_3)$, to $k_{\rm max}=0.5\,h/{\rm Mpc}$. We achieve 1$σ$ constraints of $Ω_m=0.293^{+0.027}_{-0.027}$ and $σ_8= 0.783^{+0.040}_{-0.038}$, which are more than 1.2 and 2.4$\times$ tighter than constraints from standard power spectrum analyses of the same dataset. We also derive 1.4, 1.4, 1.7$\times$ tighter constraints on $Ω_b$, $h$, $n_s$. This improvement comes from additional cosmological information in higher-order clustering on non-linear scales and, for $σ_8$, is equivalent to the gain expected from a standard analysis on a $\sim$4$\times$ larger galaxy sample. Even with our BOSS subsample, which only spans 10% of the full BOSS volume, we derive competitive constraints on the growth of structure: $S_8 = 0.774^{+0.056}_{-0.053}$. Our constraint is consistent with results from both cosmic microwave background and weak lensing. Combined with a $ω_b$ prior from Big Bang Nucleosynthesis, we also derive a constraint on $H_0=67.6^{+2.2}_{-1.8}\,{\rm km\,s^{-1}\,Mpc^{-1}}$ that is consistent with early universe constraints.

astro-ph.CO↗

arXiv2023

${\rm S{\scriptsize IM}BIG}$: The First Cosmological Constraints from Non-Gaussian and Non-Linear Galaxy Clustering

The 3D distribution of galaxies encodes detailed cosmological information on the expansion and growth history of the Universe. We present the first cosmological constraints that exploit non-Gaussian cosmological information on non-linear scales from galaxy clustering, inaccessible with current standard analyses. We analyze a subset of the BOSS galaxy survey using ${\rm S{\scriptsize IM}BIG}$, a new framework for cosmological inference that leverages high-fidelity simulations and deep generative models. We use two clustering statistics beyond the standard power spectrum: the bispectrum and a convolutional neural network based summary of the galaxy field. We infer constraints on $Λ$CDM parameters, $Ω_b$, $h$, $n_s$, $Ω_m$, and $σ_8$, that are 1.6, 1.5, 1.7, 1.2, and 2.3$\times$ tighter than power spectrum analyses. With this increased precision, we derive constraints on the Hubble constant, $H_0$, and $S_8 = σ_8 \sqrt{Ω_m/0.3}$ that are competitive with other cosmological probes, even with a sample that only spans 10% of the full BOSS volume. Our $H_0$ constraints, imposing the Big Bang Nucleosynthesis prior on the baryon density, are consistent with the early time constraints from the cosmic microwave background (CMB). Meanwhile, our $S_8$ constraints are consistent with weak lensing experiments and similarly lie below CMB constraints. Lastly, we present forecasts to show that future work extending ${\rm S{\scriptsize IM}BIG}$ to upcoming spectroscopic galaxy surveys (DESI, PFS, Euclid) will produce leading $H_0$ and $S_8$ constraints that bridge the gap between early and late time measurements and shed light on current cosmic tensions.

astro-ph.CO↗

arXiv2023

SimBIG: Field-level Simulation-Based Inference of Galaxy Clustering

We present the first simulation-based inference (SBI) of cosmological parameters from field-level analysis of galaxy clustering. Standard galaxy clustering analyses rely on analyzing summary statistics, such as the power spectrum, $P_\ell$, with analytic models based on perturbation theory. Consequently, they do not fully exploit the non-linear and non-Gaussian features of the galaxy distribution. To address these limitations, we use the {\sc SimBIG} forward modelling framework to perform SBI using normalizing flows. We apply SimBIG to a subset of the BOSS CMASS galaxy sample using a convolutional neural network with stochastic weight averaging to perform massive data compression of the galaxy field. We infer constraints on $Ω_m = 0.267^{+0.033}_{-0.029}$ and $σ_8=0.762^{+0.036}_{-0.035}$. While our constraints on $Ω_m$ are in-line with standard $P_\ell$ analyses, those on $σ_8$ are $2.65\times$ tighter. Our analysis also provides constraints on the Hubble constant $H_0=64.5 \pm 3.8 \ {\rm km / s / Mpc}$ from galaxy clustering alone. This higher constraining power comes from additional non-Gaussian cosmological information, inaccessible with $P_\ell$. We demonstrate the robustness of our analysis by showcasing our ability to infer unbiased cosmological constraints from a series of test simulations that are constructed using different forward models than the one used in our training dataset. This work not only presents competitive cosmological constraints but also introduces novel methods for leveraging additional cosmological information in upcoming galaxy surveys like DESI, PFS, and Euclid.

astro-ph.CO↗