Searcharxiv⌕ Search

arXiv subjects

William T. Freeman

Publications and source records attributed to William T. Freeman.

At least 19 recordsLinked to original sources

Latent Lattice Language Models

Language models typically increase per-token computation through deeper or looped layers and latent reasoning steps, while richer residual connections improve information flow across layers. We instead increase computational capacity: the number of active computational states a model maintains. We introduce the Lattice of Latents, a set of per-layer horizontal latent streams that run recurrently along time, alongside the vertical residual stream that flows through depth. Like hyper-connections, mHC, and attention or value residuals, the Lattice enhances the residual stream, but rather than adding or rerouting residual pathways through depth, it provides companion computational states that persist across tokens. Unlike hybrid Transformer-SSM architectures, whose recurrent states primarily serve as internal memory for sequence mixing, these latents directly condition the vertical stream's attention, KV-cache writes, MLP computation, and decoding. On Transformers trained on 20B FineWeb-Edu tokens, a 406M-parameter Lattice model lowers validation perplexity from 15.41 to 14.40 over a parameter-matched Transformer. These gains extend across architectural variants. The Lattice composes with prior residual-stream enhancements: with hyper-connections and input residuals, it lowers validation perplexity from 15.41 (base Transformer) and 14.22 (Transformer with the same enhancements) to 13.61. On Gated DeltaNet, Mamba-2, Triadic GDN, and GDN-Mamba-2 hybrids, it improves perplexity at every tested native state size, with Lattice-GDN and Lattice-Triadic-GDN variants outperforming pure baselines with 4x larger native states. These results identify persistent per-layer state as a complementary axis for scaling language-model computation.

cs.LG↗

Imaging exoplanets below the telescope stability limit

Exoplanet imaging is the main method for detailed characterization of planets orbiting beyond 1 au but it requires suppressing the bright glare of the star to see the faint planet light. As well as sophisticated instrumentation, effective image processing is necessary for starlight suppression, which continues to be a substantial limitation. Standard algorithms use reference images for starlight modelling and thus they are limited by inevitable telescope instabilities that change from reference to science observation. Addressing this challenge will increase the number of planets amenable to direct imaging and will be a notable step towards imaging exo-Earths with future observatories. Here we present a new physics-based method for starlight subtraction. Our method combines a multistep optimization with a flexible physical model to produce a starlight model, allowing modelling and compensation of instrument instabilities. We use automatic differentiation (autodiff) to optimize millions of model parameters. We demonstrate our method on observations from the James Webb Space Telescope (JWST) Near-InfraRed Camera (NIRCam) coronagraph instrument. We achieve sensitivity 2-15 times superior to standard techniques, including the first detection of 51 Eridani b at 4.3 $μ$m. This is the first time, to our knowledge, that such an approach has demonstrated substantially improved sensitivity on real, on-sky coronagraphic imaging. Our framework can be extended to other current and future ground-based and space-based observatories, including the Nancy Grace Roman Space Telescope and the Habitable Worlds Observatory (HWO), potentially relaxing the unprecedented stability requirements needed to make these a success.

astro-ph.EP↗

Learning What's Real: Disentangling Signal and Measurement Artifacts in Multi-Sensor Data, with Applications to Astrophysics

Data collected from the physical world is always a combination of multiple sources: an underlying signal from the physical process of interest and a signal from measurement-dependent artifacts from the sensor or instrument. This secondary signal acts as a confounding factor, limiting our ability to extract information about the physics underlying the phenomena we observe. Furthermore, it complicates the combination of observations in heterogeneous or multi-instrument settings. We propose a deep learning framework that leverages overlapping observations, a dual-encoder architecture, and a counterfactual generation objective to disentangle these factors of variation. The resulting representations explicitly separate intrinsic signals from sensor-specific distortions and noise, and can be used for counterfactual view generation, parameter inference unconfounded by measurement distortions, and instrument-independent similarity search. We demonstrate the effectiveness of our approach on astrophysical galaxy images from the DESI Legacy Imaging Survey (Legacy) and the Hyper Suprime-Cam (HSC) Survey as a representative multi-instrument setting. This framework provides a general recipe for scientific and multi-modal self-supervised pretraining: construct training pairs from overlapping observations of the same physical system, treat sensor- or modality-specific effects as augmentations, and learn invariant representations through counterfactual generation.

astro-ph.IM↗

Everything at Every Scale: Scale-Invariant Diffusion with Continuous Super-Resolution

Creating images from noise is image generation; reconstructing fine details from coarse inputs is super-resolution. Despite their practical differences, both can be understood as reversing information loss across scales. We introduce $\textbf{SKILD}$, a $\textbf{S}$cale-invariant $\textbf{K}$-Space $\textbf{I}$mage $\textbf{L}$earning $\textbf{D}$iffusion model that unifies generation and continuous super-resolution within a single unconditional framework. Both natural images and critical physical systems exhibit scale invariance, and we leverage it to design a forward process that attenuates image content from fine to coarse scales while injecting spectrum-matched Gaussian noise, making scale an explicit coordinate of the diffusion dynamics. The same trained reverse process performs generation and continuous super-resolution by varying only the starting timestep: $\textit{no task-specific architecture, no conditioning branch, no classifier-free guidance, no retraining per scale factor}$. Empirically, SKILD reaches FID $2.61$ and Inception Score $9.65$ on unconditional CIFAR-10, performs $2\times$--$8\times$ super-resolution on ImageNet from a single unconditional checkpoint while outperforming conditional models across perceptual metrics, and reconstructs critical Ising models whose connected four-point correlations closely track the ground truth.

cs.CV↗

Exploring perceptual straightness in learned visual representations

Humans have been shown to use a "straightened" encoding to represent the natural visual world as it evolves in time (Hénaff et al. 2019). In the context of discrete video sequences, "straightened" means that changes between frames follow a more linear path in representation space at progressively deeper levels of processing. While deep convolutional networks are often proposed as models of human visual processing, many do not straighten natural videos. In this paper, we explore the relationship between network architecture, differing types of robustness, biologically-inspired filtering mechanisms, and representational straightness in response to time-varying input; we identify strengths and limitations of straightness as a useful way of evaluating neural network representations. We find that (1) adversarial training leads to straighter representations in both CNN and transformer-based architectures but (2) this effect is task-dependent, not generalizing to tasks such as segmentation and frame-prediction, where straight representations are not favorable for predictions; and nor to other types of robustness. In addition, (3) straighter representations impart temporal stability to class predictions, even for out-of-distribution data. Finally, (4) biologically-inspired elements increase straightness in the early stages of a network, but do not guarantee increased straightness in downstream layers of CNNs. We show that straightness is an easily computed measure of representational robustness and stability, as well as a hallmark of human representations with benefits for computer vision models.

cs.LG↗

EpiCon: Collective Agent Learning through Co-Evolving Multimodal Memory

Agents can learn from past executions, but enabling different agents to reuse and build on one another's experience remains challenging. We introduce EpiCon, a shared multimodal memory framework for agent collective learning without updating host model parameters. EpiCon links question-level memory evolution to a persistent experience bank through two independently trained 2B models: a memory controller and a tree self-organizer. The controller jointly refines textual guidance and visual evidence across attempts and selectively includes visual memory. The self-organizer consolidates lessons hierarchically and retrieves experience and rules for new problems. We evaluate EpiCon on eleven benchmarks spanning four multimodal task domains, using two harnesses and multiple backbones. A frozen bank improves other systems even with a single solving attempt. A second harness raises the original system's macro-average score by 2.6 points across eleven benchmarks. Across four host configurations, EpiCon improves macro-average scores by 1.7 to 4.9 points over No Memory and reduces memory-operation time by 67\% to 74\% relative to backbone-sized memory models.

cs.CV↗

Broadband multiwavelength properties of the archetypal blazar 3C 279 during the 2017 Event Horizon Telescope campaign

The archetypal blazar 3C 279 hosts a prominent relativistic jet and exhibits strong broadband variability across the electromagnetic spectrum. In April 2017, the Event Horizon Telescope (EHT) observed 3C 279, alongside one of the most extensive quasi-simultaneous multiwavelength (MWL) campaigns ever conducted. With the aim of investigating the physical processes governing 3C 279, we analyzed individual observations and multiband light curves, and constructed a new quasi-simultaneous MWL spectrum. We also performed phenomenological modeling using the turbulent extreme multi-zone (TEMZ) model to constrain the fundamental physical properties of the source. The EHT observations reveal a clear flux increase in the innermost core between April 5 and 11, 2017. Over a broader timescale, radio measurements at longer wavelengths show concurrent enhancements in core flux and polarization around mid-April, coinciding with the ejection of a superluminal knot. Record UV-optical flares with strong polarization variability occurred in late March, followed by gamma-ray activity that declined before the end of the EHT observing period. During this interval, the source remained in a low X-ray state and showed no detectable VHE emission. The TEMZ modeling suggests that the broadband spectrum and variability of 3C 279 can be explained within a jet scenario in which turbulent plasma cells are compressed by a stationary conical shock. However, alternative interpretations, such as magnetic reconnection or a moving shock-in-jet event, remain plausible. This coordinated MWL campaign advances our understanding of the origin of jet and gamma-ray emission in 3C 279, while also providing a comprehensive publicly available dataset that will serve as a valuable reference for future studies.

astro-ph.HE↗

MathNet: a Global Multimodal Benchmark for Mathematical Reasoning and Retrieval

Mathematical problem solving remains a challenging test of reasoning for large language and multimodal models, yet existing benchmarks are limited in size, language coverage, and task diversity. We introduce MathNet, a high-quality, large-scale, multimodal, and multilingual dataset of Olympiad-level math problems together with a benchmark for evaluating mathematical reasoning in generative models and mathematical retrieval in embedding-based systems. MathNet spans 47 countries, 17 languages, and two decades of competitions, comprising 30,676 expert-authored problems with solutions across diverse domains. In addition to the core dataset, we construct a retrieval benchmark consisting of mathematically equivalent and structurally similar problem pairs curated by human experts. MathNet supports three tasks: (i) Problem Solving, (ii) Math-Aware Retrieval, and (iii) Retrieval-Augmented Problem Solving. Experimental results show that even state-of-the-art reasoning models (78.4% for Gemini-3.1-Pro and 69.3% for GPT-5) remain challenged, while embedding models struggle to retrieve equivalent problems. We further show that retrieval-augmented generation performance is highly sensitive to retrieval quality; for example, DeepSeek-V3.2-Speciale achieves gains of up to 12%, obtaining the highest scores on the benchmark. MathNet provides the largest high-quality Olympiad dataset together with the first benchmark for evaluating mathematical problem retrieval, and we publicly release both the dataset and benchmark at https://mathnet.mit.edu.

cs.AI↗

Comparison of Polarized Radiative Transfer Codes used by the EHT Collaboration

Interpretation of resolved polarized images of black holes by the Event Horizon Telescope (EHT) requires predictions of the polarized emission observable by an Earth-based instrument for a particular model of the black hole accretion system. Such predictions are generated by general relativistic radiative transfer (GRRT) codes, which integrate the equations of polarized radiative transfer in curved spacetime. A selection of ray-tracing GRRT codes used within the EHT collaboration is evaluated for accuracy and consistency in producing a selection of test images, demonstrating that the various methods and implementations of radiative transfer calculations are highly consistent. When imaging an analytic accretion model, we find that all codes produce images similar within a pixel-wise normalized mean squared error (NMSE) of 0.012 in the worst case. When imaging a snapshot from a cell-based magnetohydrodynamic simulation, we find all test images to be similar within NMSEs of 0.02, 0.04, 0.04, and 0.12 in Stokes I, Q, U , and V respectively. We additionally find the values of several image metrics relevant to published EHT results to be in agreement to much better precision than measurement uncertainties.

astro-ph.HE↗

Dynamics and geometry of the inner sub-parsec-scale jet in 3C 279 observed with the Event Horizon Telescope

The 2021 Event Horizon Telescope observations resolve the innermost jet region of the blazar 3C279 with unprecedented detail. The reconstructed images consistently reveal a compact core elongated nearly orthogonal to the large-scale jet axis. This rarely observed morphology recurs across multiple epochs and from 22-230 GHz and is therefore intrinsic rather than an imaging artifact. Geometric model fitting identifies several components with apparent speeds up to 10c, requiring bulk Lorentz factors greater than 10.3 and constraining viewing angles to extremely small values (smaller than one degree). Rest-frame brightness temperatures are systematically low (between 10^9 and 10^10 K), consistent with optically thin emission at 230 GHz. These results suggest that the jet bends toward the observer on sub-parsec scales, producing strong relativistic beaming. Possible drivers of the observed jet bending and temporal evolution include the jet's interaction with the interstellar medium, kink or Kelvin--Helmholtz instabilities, magnetic reconnection near the horizon, or binary-induced precession. However, the current temporal coverage of VLBI data remains insufficient to distinguish between these mechanisms. Continued multifrequency VLBI monitoring will be essential to constraining the dynamics and geometry of the jet base in 3C279.

astro-ph.GA↗

GenMatter: Perceiving Physical Objects with Generative Matter Models

Human visual perception offers valuable insights for understanding computational principles of motion-based scene interpretation. Humans robustly detect and segment moving entities that constitute independently moveable chunks of matter, whether observing sparse moving dots, textured surfaces, or naturalistic scenes. In contrast, existing computer vision systems lack a unified approach that works across these diverse settings. Inspired by principles of human perception, we propose a generative model that hierarchically groups low-level motion cues and high-level appearance features into particles (small Gaussians representing local matter), and groups particles into clusters capturing coherently and independently moveable physical entities. We develop a hardware-accelerated inference algorithm based on parallelized block Gibbs sampling to recover stable particle motion and groupings. Our model operates on different kinds of inputs (random dots, stylized textures, or naturalistic RGB video), enabling it to work across settings where biological vision succeeds but existing computer vision approaches do not. We validate this unified framework across three domains: on 2D random dot kinematograms, our approach captures human object perception including graded uncertainty across ambiguous conditions; on a Gestalt-inspired dataset of camouflaged rotating objects, our approach recovers correct 3D structure from motion and thereby accurate 2D object segmentation; and on naturalistic RGB videos, our model tracks the moving 3D matter that makes up deforming objects, enabling robust object-level scene understanding. This work thus establishes a general framework for motion-based perception grounded in principles of human vision.

cs.CV↗

Dynamic Black-hole Emission Tomography with Physics-informed Neural Fields

With the success of static black-hole imaging, the next frontier is the dynamic and 3D imaging of black holes. Recovering the dynamic 3D gas near a black hole would reveal previously-unseen parts of the universe and inform new physics models. However, only sparse radio measurements from a single viewpoint are possible, making the dynamic 3D reconstruction problem significantly ill-posed. Previously, BH-NeRF addressed the ill-posed problem by assuming Keplerian dynamics of the gas, but this assumption breaks down near the black hole, where the strong gravitational pull of the black hole and increased electromagnetic activity complicate fluid dynamics. To overcome the restrictive assumptions of BH-NeRF, we propose PI-DEF, a physics-informed approach that uses differentiable neural rendering to fit a 4D (time + 3D) emissivity field given EHT measurements. Our approach jointly reconstructs the 3D velocity field with the 4D emissivity field and enforces the velocity as a soft constraint on the dynamics of the emissivity. In experiments on simulated data, we find significantly improved reconstruction accuracy over both BH-NeRF and a physics-agnostic approach. We demonstrate how our method may be used to estimate other physics parameters of the black hole, such as its spin.

gr-qc↗

Large Video Planner Enables Generalizable Robot Control

General-purpose robots require decision-making models that generalize across diverse tasks and environments. Recent works build robot foundation models by extending multimodal large language models (MLLMs) with action outputs, creating vision-language-action (VLA) systems. These efforts are motivated by the intuition that MLLMs' large-scale language and image pretraining can be effectively transferred to the action output modality. In this work, we explore an alternative paradigm of using large-scale video pretraining as a primary modality for building robot foundation models. Unlike static images and language, videos capture spatio-temporal sequences of states and actions in the physical world that are naturally aligned with robotic behavior. We curate an internet-scale video dataset of human activities and task demonstrations, and train, for the first time at a foundation-model scale, an open video model for generative robotics planning. The model produces zero-shot video plans for novel scenes and tasks, which we post-process to extract executable robot actions. We evaluate task-level generalization through third-party selected tasks in the wild and real-robot experiments, demonstrating successful physical execution. Together, these results show robust instruction following, strong generalization, and real-world feasibility. We release both the model and dataset to support open, reproducible video-based robot learning. Our website is available at https://www.boyuan.space/large-video-planner/.

cs.RO↗

Full-polarization millimeter wavelength variability of Sagittarius A* during the 2018 EHT campaign

Sagittarius A* (Srg A*), the supermassive black hole at the center of the Milky Way, provides a unique laboratory to study accretion dynamics and plasma processes near the event horizon. We investigated the variability and polarization properties of Srg A* using ALMA observations during the 2018 Event Horizon Telescope campaign. We analyzed high-cadence full-polarization light curves from ALMA at millimeter wavelengths, performed time-series analysis, and investigated the temporal behavior during an X-ray flare observed by Chandra on 2018 April 24. The variability characteristics are compared with expectations from standard accretion flow models. We find low variability in total intensity ($σ/μ< 10\%$), but significantly higher variability in linear and circular polarization (~ 30% and ~ 50%, respectively). A time-series analysis reveals red-noise variability, with power spectral densities between -2 and -3 across all Stokes parameters. Polarized intensity shows stable intra-day timescales, while total intensity exhibits more variable timescales, suggesting distinct emission regions, with polarization likely arising from a coherent structure. On April 24, a statistically significant inter-band delay in polarized intensity coincides with a near-simultaneous X-ray and millimeter peak that deviates from the typical delayed flare scenario. This event also features enhanced millimeter variability and coherent polarization loop evolution. The observed simultaneity challenges standard models of transient synchrotron emission with cooling delays, favoring instead a scenario of continuous energy injection in an optically thin region. Our results offer new constraints on the physical mechanisms driving variability in Srg A*, and provide key observational input for refining theoretical models of accretion and plasma behavior in the vicinity of supermassive black holes.

astro-ph.GA↗

Ring Asymmetry and Spin in M87*

Event Horizon Telescope (EHT) images of the supermassive black hole M87* depict an asymmetric ring of emission. General relativistic magnetohydrodynamic (GRMHD) models of M87* and its accretion disk predict that the amplitude and location of the ring's peak brightness asymmetry should fluctuate due to turbulence in the source plasma. We compare the observed distribution of brightness asymmetry amplitudes to the simulated distribution in GRMHD models, across varying black hole spin $a_{*}$. We show that, for strongly magnetized (MAD) models, three epochs of EHT data marginally disfavor $|a_{*}| \lesssim 0.2$. This is consistent with the Blandford-Znajek model for M87's jet, which predicts that M87* should have nonzero spin. We show quantitatively how future observations could improve spin constraints, and discuss how improved spin constraints could distinguish between differing jet-launching mechanisms and black hole growth scenarios.

astro-ph.HE↗

End-to-End Training for Unified Tokenization and Latent Denoising

Latent diffusion models (LDMs) enable high-fidelity synthesis by operating in learned latent spaces. However, training state-of-the-art LDMs requires complex staging: a tokenizer must be trained first, before the diffusion model can be trained in the frozen latent space. We propose UNITE - an autoencoder architecture for unified tokenization and latent diffusion. UNITE consists of a Generative Encoder that serves as both image tokenizer and latent generator via weight sharing. Our key insight is that tokenization and generation can be viewed as the same latent inference problem under different conditioning regimes: tokenization infers latents from fully observed images, whereas generation infers them from noise together with text or class conditioning. Motivated by this, we introduce a single-stage training procedure that jointly optimizes both tasks via two forward passes through the same Generative Encoder. The shared parameters enable gradients to jointly shape the latent space, encouraging a "common latent language". Across image and molecule modalities, UNITE achieves near state of the art performance without adversarial losses or pretrained encoders (e.g., DINO), reaching FID 2.12 and 1.73 for Base and Large models on ImageNet 256 x 256. We further analyze the Generative Encoder through the lenses of representation alignment and compression. These results show that single stage joint training of tokenization & generation from scratch is feasible.

cs.CV↗

Locating the missing large-scale emission in the jet of M87* with short EHT baselines

In Very-Long Baseline Interferometric arrays, nearly co-located stations probe the largest scales and typically cannot resolve the observed source. In the absence of large-scale structure, closure phases constructed with these stations are zero and, since they are independent of station-based errors, they can be used to probe data issues. Here, we show with an expansion about co-located stations, how these trivial closure phases become non-zero with brightness distribution on smaller scales than their short baseline would suggest. When applied to sources that are made up of a bright compact and large-scale diffuse component, the trivial closure phases directly measure the centroid relative to the compact source and higher-order image moments. We present a technique to measure these image moments with minimal model assumptions and validate it on synthetic Event Horizon Telescope (EHT) data. We then apply this technique to 2017 and 2018 EHT observations of M87* and find a weak preference for extended emission in the direction of the large-scale jet. We also apply it to 2021 EHT data and measure the source centroid about 1 mas northwest of the compact ring, consistent with the jet observed at lower frequencies.

astro-ph.HE↗

Probing jet base emission of M87* with the 2021 Event Horizon Telescope observations

We investigate the presence and spatial characteristics of the jet base emission in M87* at 230 GHz, enabled by the enhanced uv coverage in the 2021 Event Horizon Telescope (EHT) observations. The addition of the 12-m Kitt Peak Telescope and NOEMA provides two key intermediate-length baselines to SMT and the IRAM 30-m, giving sensitivity to emission structures at scales of $\sim250~μ$as and $\sim2500~μ$as (0.02 pc and 0.2 pc). Without these baselines, earlier EHT observations lacked the capability to constrain emission on large scales, where a "missing flux" of order $\sim1$ Jy is expected. To probe these scales, we analyzed closure phases, robust against station-based gain errors, and modeled the jet base emission using a simple Gaussian offset from the compact ring emission at separations $>100~μ$as. Our analysis reveals a Gaussian feature centered at ($Δ$RA $\approx320~μ$as, $Δ$Dec $\approx60~μ$as), a projected separation of $\approx5500$ AU, with a flux density of only $\sim60$ mJy, implying that most of the missing flux in previous studies must arise from larger scales. Brighter emission at these scales is ruled out, and the data do not favor more complex models. This component aligns with the inferred direction of the large-scale jet and is consistent with emission from the jet base. While our findings indicate detectable jet base emission at 230 GHz, coverage from only two intermediate baselines limits reconstruction of its morphology. We therefore treat the recovered Gaussian as an upper limit on the jet base flux density. Future EHT observations with expanded intermediate-baseline coverage will be essential to constrain the structure and nature of this component.

astro-ph.HE↗