arXiv subjects

Elias Ellison

Publications and source records attributed to Elias Ellison.

3 recordsLinked to original sources

arXiv2026

AutoParallel: Composable, PyTorch-Native Automatic Parallelism for Production Training

Training large models requires multiple interacting forms of parallelism. As models and clusters scale, finding and maintaining efficient parallelization plans becomes essential for training performance. Diverse model architectures, workloads, and hardware topologies demand flexible strategies, yet production PyTorch systems typically rely on hand-written recipes that are costly to adapt and tend to reuse familiar patterns, which may be suboptimal for a given workload. We present AutoParallel, a PyTorch-native planner for distributed model training. AutoParallel searches parallelization choices jointly across the training computation, allowing established techniques to be combined and unconventional strategies to emerge. It expresses its decisions using native PyTorch abstractions and produces ordinary PyTorch programs, avoiding a separate execution runtime while composing directly with the PyTorch compilation and training stack. AutoParallel also composes automatic decisions with model-specific distributed code, allowing dynamic or specialized regions to remain under user control. Plans are inspectable and replayable artifacts. Across dense, hybrid, and mixture-of-experts models on meshes of up to four dimensions and 128 NVIDIA H100 GPUs, AutoParallel matches or exceeds expert-written TorchTitan recipes on most configurations, with throughput gains of up to 18.3%, and discovers plans that uniform expert recipes do not express.

cs.DC↗

arXiv2026

Optimus: A Generic Operator-Level PyTorch Model Transformation Framework

In large-scale industrial applications, deep learning models that power recommendation and ranking have complex and diverse model architectures. These models are continuously developed and refined by large teams of machine learning engineers, rendering manual optimization infeasible. Consequently, graph-based optimization techniques have become an industry standard for boosting performance, with PyTorch FX transformations leading the charge. These transformations typically rely on a set of human-engineered module-level rewrite rules which are not scalable to diverse model architectures. To address this limitation, we introduce Optimus, a general-purpose model transformation framework built in the PyTorch 2.x (PT2) machine learning compiler. With a concise set of predefined patterns, Optimus applies an efficient greedy search algorithm for pattern matching and replacement, while preserving model semantic. It is designed and implemented as a highly customizable and extensible framework integrated into the PT2 stack. Our evaluation shows that the framework can achieve up to 63% speedup, 6% peak memory reduction, and over 400 second compile time decrease for our industry-scale recommendation models compared to baselines. Optimus is open-sourced together with PyTorch 2.x as a customizable model transformation layer.

cs.PF↗

arXiv2013

A Systematic Retrieval Analysis of Secondary Eclipse Spectra I: A Comparison of Atmospheric Retrieval Techniques

Spectra of exoplanet atmospheres provide us the opportunity to improve our understanding of these objects just as remote sensing in our own solar system has increased our understanding of the solar system bodies. The challenge is to quantitatively determine the range of temperatures and species abundances allowed by the data. This challenge is often difficult given the low information content of most exoplanet spectra which commonly leads to degeneracies in the interpretation. A variety of temperature and abundance retrieval approaches have been applied to exoplanet spectra, but no previous investigations have sought to compare these approaches. In this investigation we compare three different retrieval methods: Optimal Estimation, Differential Evolution Markov Chain Monte Carlo, and Bootstrap Monte Carlo. We call our suite of retrieval algorithms the Caltech Inverse Modeling and Retrieval Algorithms (CHIMERA). We discuss what we can expect in terms of uncertainties in abundances and temperatures given current observations as well as potential future observations and what conclusions can be drawn given those uncertainties. In general we find that the three approaches agree for high quality spectra expected to come from potential future spaceborne missions, but disagree for low quality spectra representative of current observations. We also show that the Gaussian posterior probability distribution assumption made in the Optimal Estimation approach is valid for high quality spectral data. We also discuss the implications of our models for the inferred C to O ratios of exoplanetary atmospheres, which of course are important for understanding formation environments. More specifically we show that in the observational limit of a few photometric points, the retrieved C/O is biased towards values near solar and near one simply due to the assumption of uninformative priors.

astro-ph.EP↗