arXiv subjects

Thomas Schulthess

Publications and source records attributed to Thomas Schulthess.

13 recordsLinked to original sources

arXiv2026

An Engineering Journey Training Large Language Models at Scale on Alps: The Apertus Experience

Large Language Models (LLMs) have surged as a transformative technology for science and society, prompting governments worldwide to pursue sovereign AI capabilities that ensure data compliance and cultural representation. However, the associated capital costs and engineering complexity required to train these models have largely restricted such capabilities to the private sector, leaving a significant gap for public institutions. This paper details the engineering journey behind training Apertus, a fully open multilingual foundation model, on the Alps supercomputer. Representing a first-of-its-kind achievement for academia at the 70B parameter scale, we successfully deployed a massive pre-training campaign on one of Europe's largest systems for open science, powered by NVIDIA GH200 Grace Hopper Superchips. We detail the challenges encountered in readying HPC infrastructure for training AI models, from overcoming storage bottlenecks to stabilizing large-scale interconnects, and the lessons learned in transforming a supercomputer into a resilient software-defined Machine Learning Platform. Finally, we discuss the post-training requirements and evolution of our Machine Learning platform, outlining how this initial release lays the groundwork for a sustained, iterative operational capability, in particular for fine tuning foundation models, that extends well beyond a single model training run.

cs.DC↗

arXiv2025

Apertus: Democratizing Open and Compliant LLMs for Global Language Environments

We present Apertus, a fully open suite of large language models (LLMs) designed to address two systemic shortcomings in today's open model ecosystem: data compliance and multilingual representation. Unlike many prior models that release weights without reproducible data pipelines or regard for content-owner rights, Apertus models are pretrained exclusively on openly available data, retroactively respecting `robots.txt` exclusions and filtering for non-permissive, toxic, and personally identifiable content. To mitigate risks of memorization, we adopt the Goldfish objective during pretraining, strongly suppressing verbatim recall of data while retaining downstream task performance. The Apertus models also expand multilingual coverage, training on 15T tokens from over 1800 languages, with ~40% of pretraining data allocated to non-English content. Released at 8B and 70B scales, Apertus approaches state-of-the-art results among fully open models on multilingual benchmarks, rivalling or surpassing open-weight counterparts. Beyond model weights, we release all scientific artifacts from our development cycle with a permissive license, including data preparation scripts, checkpoints, evaluation suites, and training code, enabling transparent audit and extension.

cs.CL↗

arXiv2024

Understanding Data Movement in Tightly Coupled Heterogeneous Systems: A Case Study with the Grace Hopper Superchip

Heterogeneous supercomputers have become the standard in HPC. GPUs in particular have dominated the accelerator landscape, offering unprecedented performance in parallel workloads and unlocking new possibilities in fields like AI and climate modeling. With many workloads becoming memory-bound, improving the communication latency and bandwidth within the system has become a main driver in the development of new architectures. The Grace Hopper Superchip (GH200) is a significant step in the direction of tightly coupled heterogeneous systems, in which all CPUs and GPUs share a unified address space and support transparent fine grained access to all main memory on the system. We characterize both intra- and inter-node memory operations on the Quad GH200 nodes of the new Swiss National Supercomputing Centre Alps supercomputer, and show the importance of careful memory placement on example workloads, highlighting tradeoffs and opportunities.

cs.DC↗

arXiv2024

Towards Specialized Supercomputers for Climate Sciences: Computational Requirements of the Icosahedral Nonhydrostatic Weather and Climate Model

We discuss the computational challenges and requirements for high-resolution climate simulations using the Icosahedral Nonhydrostatic Weather and Climate Model (ICON). We define a detailed requirements model for ICON which emphasizes the need for specialized supercomputers to accurately predict climate change impacts and extreme weather events. Based on the requirements model, we outline computational demands for km-scale simulations, and suggests machine learning techniques to enhance model accuracy and efficiency. Our findings aim to guide the design of future supercomputers for advanced climate science.

physics.ao-ph↗

arXiv2024

XaaS: Acceleration as a Service to Enable Productive High-Performance Cloud Computing

HPC and Cloud have evolved independently, specializing their innovations into performance or productivity. Acceleration as a Service (XaaS) is a recipe to empower both fields with a shared execution platform that provides transparent access to computing resources, regardless of the underlying cloud or HPC service provider. Bridging HPC and cloud advancements, XaaS presents a unified architecture built on performance-portable containers. Our converged model concentrates on low-overhead, high-performance communication and computing, targeting resource-intensive workloads from climate simulations to machine learning. XaaS lifts the restricted allocation model of Function-as-a-Service (FaaS), allowing users to benefit from the flexibility and efficient resource utilization of serverless while supporting long-running and performance-sensitive workloads from HPC.

cs.DC↗

arXiv2023

Earth Virtualization Engines -- A Technical Perspective

Participants of the Berlin Summit on Earth Virtualization Engines (EVEs) discussed ideas and concepts to improve our ability to cope with climate change. EVEs aim to provide interactive and accessible climate simulations and data for a wide range of users. They combine high-resolution physics-based models with machine learning techniques to improve the fidelity, efficiency, and interpretability of climate projections. At their core, EVEs offer a federated data layer that enables simple and fast access to exabyte-sized climate data through simple interfaces. In this article, we summarize the technical challenges and opportunities for developing EVEs, and argue that they are essential for addressing the consequences of climate change.

physics.ao-ph↗

arXiv2022

All-Electron APW+${lo}$ calculation of magnetic molecules with the SIRIUS domain-specific package

We report APW+${lo}$ (augmented plane wave plus local orbital) density functional theory (DFT) calculations of molecule systems using the domain specific SIRIUS multi-functional DFT package. Compared to other packages the additional APW and FLAPW task and data parallelism and the additional eigensystem solver provided by the SIRIUS package can be exploited for performance gains in in the ground state Kohn-Sham calculation. This is in contrast with the use of SIRIUS as a library backend to some other APW+${lo}$ or FLAPW (full-potential linearized AWP) code. We benchmark the code and demonstrate performance on several magnetic molecule and metal organic framework systems. We show that the SIRIUS package in itself is capable of handling systems as large as a few hundreds of atoms in the unit cell without losing the accuracy needed for magnetic systems.

cond-mat.mtrl-sci↗

arXiv2022

Productive Performance Engineering for Weather and Climate Modeling with Python

Earth system models are developed with a tight coupling to target hardware, often containing specialized code predicated on processor characteristics. This coupling stems from using imperative languages that hard-code computation schedules and layout. We present a detailed account of optimizing the Finite Volume Cubed-Sphere Dynamical Core (FV3), improving productivity and performance. By using a declarative Python-embedded stencil domain-specific language and data-centric optimization, we abstract hardware-specific details and define a semi-automated workflow for analyzing and optimizing weather and climate applications. The workflow utilizes both local and full-program optimization, as well as user-guided fine-tuning. To prune the infeasible global optimization space, we automatically utilize repeating code motifs via a novel transfer tuning approach. On the Piz Daint supercomputer, we scale to 2,400 GPUs, achieving speedups of up to 3.92x over the tuned production implementation at a fraction of the original code.

cs.DC↗

arXiv2019

DFT+DMFT calculations of the complex band and tunneling behavior for the transition metal monoxides MnO, FeO, CoO and NiO

We report complex band structure (CBS) calculations for the four late transition metal monoxides, MnO, FeO, CoO and NiO, in their paramagnetic phase. The CBS is obtained from density functional theory plus dynamical mean field theory (DMFT) calculations to take into account correlation effects. The so-called $β$ parameters, governing the exponential decay of the transmission probability in the non-resonant tunneling regime of these oxides, are extracted from the CBS. Different model constructions are examined in the DMFT part of the calculation. The calculated $β$ parameters provide theoretical estimation for the decay length in the evanescent channel, which would be useful for tunnel junction applications of these materials.

cond-mat.str-el↗

arXiv2019

SimFS: A Simulation Data Virtualizing File System Interface

Nowadays simulations can produce petabytes of data to be stored in parallel filesystems or large-scale databases. This data is accessed over the course of decades often by thousands of analysts and scientists. However, storing these volumes of data for long periods of time is not cost effective and, in some cases, practically impossible. We propose to transparently virtualize the simulation data, relaxing the storage requirements by not storing the full output and re-simulating the missing data on demand. We develop SimFS, a file system interface that exposes a virtualized view of the simulation output to the analysis applications and manages the re-simulations. SimFS monitors the access patterns of the analysis applications in order to (1) decide the data to keep stored for faster accesses and (2) to employ prefetching strategies to reduce the access time of missing data. Virtualizing simulation data allows us to trade storage for computation: this paradigm becomes similar to traditional on-disk analysis (all data is stored) or in situ (no data is stored) according with the storage resources that are assigned to SimFS. Overall, by exploiting the growing computing power and relaxing the storage capacity requirements, SimFS offers a viable path towards exa-scale simulations.

cs.DC↗

arXiv2014

Two-particle correlations in a dynamic cluster approximation with continuous momentum dependence: Superconductivity in the 2D Hubbard model

The DCA$^+$ algortihm was recently introduced to extend the dynamic cluster approximation (DCA) with a continuous lattice self-energy in order to achieve better convergence with cluster size. Here we extend the DCA$^+$ algorithm to the calculation of two-particle correlation functions by introducing irreducible vertex functions with continuous momentum dependence consistent with the DCA$^+$ self-energy. This enables a significantly more controlled and reliable study of phase transitions than with the DCA. We test the new method by calculating the superconducting transition temperature $T_{c}$ in the attractive Hubbard model and show that it reproduces previous high-precision determinantal quantum Monte Carlo results. We then calculate $T_c$ in the doped repulsive Hubbard model, for which previous DCA calculations could only access the weak-coupling ($U=4t$) regime for large clusters. We show that the new algorithm provides access to much larger clusters and delivers asymptotically converged results for $T_c$ for both the weak ($U=4t$) and intermediate ($U=7t$) coupling regimes, and thereby enables the accurate determination of the exact infinite cluster size result.

cond-mat.str-el↗

arXiv2013

The Continuous-Pole-Expansion method to obtain spectra of electronic lattice models

We present a new algorithm to analytically continue the self-energy of quantum many-body systems from Matsubara frequencies to the real axis. The method allows straightforward, unambiguous computation of electronic spectra for lattice models of strongly correlated systems from self-energy data that has been collected with state-of-the are continuous time solvers within dynamical mean field simulations. Using well-known analytical properties of the self-energy, the analytic continuation is cast into a constrained minimization problem that can be formulated as a quadratic programmable optimization with linear constraints. The algorithm is validated against exactly solvable finite size problems, showing that all features of the spectral function near the Femi level are very well reproduced and coarse features are reproduced for all energies. The method is applied to two well known lattice problems, the two-dimensional Hubbard model at half filling where the momentum dependence of the gap formation is studied, as well as a multi-band model of NiO, for which the spectral function can be directly compared to experiment. Agreement with results published results is very good.

cond-mat.str-el↗

arXiv2011

Sub-matrix updates for the Continuous-Time Auxiliary Field algorithm

We present a sub-matrix update algorithm for the continuous-time auxiliary field method that allows the simulation of large lattice and impurity problems. The algorithm takes optimal advantage of modern CPU architectures by consistently using matrix instead of vector operations, resulting in a speedup of a factor of $\approx 8$ and thereby allowing access to larger systems and lower temperature. We illustrate the power of our algorithm at the example of a cluster dynamical mean field simulation of the Néel transition in the three-dimensional Hubbard model, where we show momentum dependent self-energies for clusters with up to 100 sites.

cond-mat.str-el↗