Searcharxiv⌕ Search

arXiv subjects

Felix Wolf

Publications and source records attributed to Felix Wolf.

At least 19 recordsLinked to original sources
arXiv2026

ShardMeter: Sharded and Geo-Distributed Training Without the Guesswork

Training large-scale AI models often outgrows a single data center, demanding sharded, multi-cluster, and decentralized training. However, the huge space of resource allocations makes exhaustive benchmarking and manual tuning impractical, while performance depends on tightly coupled factors like model size, GPU memory, batch size, bandwidth, and sharding strategy. We introduce ShardMeter, a lightweight analytical performance model that predicts the end-to-end runtime of transformer-based workloads across arbitrary sharded, distributed, and even decentralized training. Given a model's characteristics and a target hardware topology, ShardMeter estimates per-GPU and per-island throughput, training cost, total wall-clock time, and identifies performance bottlenecks. Our analysis reveals diminishing-return regimes as island size increases, quantifies transitions between compute- and communication-bound scaling, evaluates hyperparameter trade-offs, and models cost-throughput for large-scale decentralized training. ShardMeter exposes these insights to quickly explore the configuration space, choose near-optimal deployment plans, and avoid costly trial and error.

cs.DC↗
arXiv2026

HeLoCo: Efficient asynchronous low-communication training under data and device heterogeneity

Distributed Low-Communication (DiLoCo) training reduces communication overhead by allowing workers to perform multiple local optimization steps before sending pseudo-gradients to a global outer update. Its asynchronous variant further improves hardware utilization by removing synchronization barriers, but at the cost of stale pseudo-gradients computed from outdated model states. As a result, these updates can become misaligned with the current global optimization direction, particularly in heterogeneous systems. This issue becomes even more pronounced when data are non-IID, a setting that has not been well studied in asynchronous low-communication training. To address this limitation, we propose \textbf{HeLoCo}, a direction-aware correction method for asynchronous low-communication training that uses outer momentum as a reference for the current optimization trajectory and selectively adjusts incoming pseudo-gradients before the outer update. Updates that remain aligned are preserved, while directionally conflicting components are corrected. On multilingual language-model training with heterogeneous workers and non-IID data, HeLoCo consistently improves validation loss. It outperforms existing asynchronous DiLoCo-based baselines by up to 7.5\% at a fixed token budget, exceeds asynchronous momentum look-ahead by up to 3.3\% at a fixed wall-clock budget, and surpasses the synchronous baseline by up to 22.1\% under severe system heterogeneity. Our analysis further shows how staleness, worker speed, and data heterogeneity shape update quality and convergence in highly decentralized and heterogeneous training setups.

cs.DC↗
arXiv2025

When AI Bends Metal: AI-Assisted Optimization of Design Parameters in Sheet Metal Forming

Numerical simulations have revolutionized the industrial design process by reducing prototyping costs, design iterations, and enabling product engineers to explore the design space more efficiently. However, the growing scale of simulations demands substantial expert knowledge, computational resources, and time. A key challenge is identifying input parameters that yield optimal results, as iterative simulations are costly and can have a large environmental impact. This paper presents an AI-assisted workflow that reduces expert involvement in parameter optimization through the use of Bayesian optimization. Furthermore, we present an active learning variant of the approach, assisting the expert if desired. A deep learning model provides an initial parameter estimate, from which the optimization cycle iteratively refines the design until a termination condition (e.g.,energy budget or iteration limit) is met. We demonstrate our approach, based on a sheet metal forming process, and show how it enables us to accelerate the exploration of the design space while reducing the need for expert involvement.

cs.AI↗
arXiv2025

I Like To Move It -- Computation Instead of Data in the Brain

The detailed functioning of the human brain remains incompletely understood. Large-scale brain simulations complement experimental research but face substantial computational challenges: the human brain comprises approximately $10^{11}$ neurons connected by $10^{14}$ synapses, collectively forming the connectome. Empirical evidence indicates that modifications of the connectome -- specifically the formation and elimination of synapses, referred to as structural plasticity -- are essential for processes such as learning and memory formation. Connectivity updates can be computed efficiently using a Barnes--Hut-inspired approximation that reduces computational complexity from $O(n^2)$ to $O(n \log n)$, where $n$ denotes the number of neurons. Despite this improvement, communication overhead still limits scalability. Synapse updates rely heavily on remote memory access (RMA), and spike transmission requires all-to-all communication at every simulation time step. We introduce a novel algorithm that reduces communication by migrating computation rather than data. This approach reduces connectivity update time by a factor of 6 and spike transmission time by more than 2 orders of magnitude.

cs.DC↗
arXiv2025

Navigating the Energy Doldrums: Can We Exploit Energy-Price Volatility To Lower the Cost of Computing?

Energy costs are a major factor in the total cost of ownership (TCO) for high-performance computing (HPC) systems. The rise of intermittent green energy sources and reduced reliance on fossil fuels have introduced volatility into electricity markets, complicating energy budgeting. This paper explores variable capacity as a strategy for managing HPC energy costs -- dynamically adjusting compute resources in response to fluctuating electricity prices. While this approach can lower energy expenses, it risks underutilizing costly hardware. To evaluate this trade-off, we present a simple model that helps operators estimate the TCO impact of variable capacity strategies using key system parameters. We apply this model to real data from a university HPC cluster and assess how different scenarios could affect the cost-effectiveness of this approach in the future.

cs.DC↗
arXiv2025

Denoising Application Performance Models with Noise-Resilient Priors

As parallel codes are scaled to larger computing systems, performance models play a crucial role in identifying potential bottlenecks. However, constructing these models analytically is often challenging. Empirical models based on performance measurements provide a practical alternative, but measurements on high-performance computing (HPC) systems are frequently affected by noise, which can lead to misleading predictions. To mitigate the impact of noise, we introduce application-specific dynamic priors into the modeling process. These priors are derived from noise-resilient measurements of computational effort, combined with domain knowledge about common algorithms used in communication routines. By incorporating these priors, we effectively constrain the model's search space, eliminating complexity classes that capture noise rather than true performance characteristics. This approach keeps the models closely aligned with theoretical expectations and substantially enhances their predictive accuracy. Moreover, it reduces experimental overhead by cutting the number of repeated measurements by half.

cs.PF↗
arXiv2024

A Low-Frequency-Stable Higher-Order Isogeometric Discretization of the Augmented Electric Field Integral Equation

This contribution investigates the connection between isogeometric analysis and integral equation methods for full-wave electromagnetic problems up to the low-frequency limit. The proposed spline-based integral equation method allows for an exact representation of the model geometry described in terms of non-uniform rational B-splines without meshing. This is particularly useful when high accuracy is required or when meshing is cumbersome for instance during optimization of electric components. The augmented electric field integral equation is adopted and the deflation method is applied, so the low-frequency breakdown is avoided. The extension to higher-order basis functions is analyzed and the convergence rate is discussed. Numerical experiments on academic and realistic test cases demonstrate the high accuracy of the proposed approach.

cs.CE↗
arXiv2023

Capturing Periodic I/O Using Frequency Techniques

Many HPC applications perform their I/O in bursts that follow a periodic pattern. This allows for making predictions as to when a burst occurs. System providers can take advantage of such knowledge to reduce file-system contention by actively scheduling I/O bandwidth. The effectiveness of this approach, however, depends on the ability to detect and quantify the periodicity of I/O patterns online. In this paper, we introduce FTIO, an online method to detect periodic I/O phases, which is based on discrete Fourier transform (DFT), combined with outlier detection. We provide metrics that gauge the confidence in the output and tell how far from being periodic the signal is. We validate our approach with large-scale experiments on a production system and examine its limitations extensively. Our experiments show that FTIO has a mean error below 11%. Finally, we demonstrate that FTIO allowed the I/O scheduler Set- 10 to boost system utilization by 26% and reduce I/O slowdown by 56%.

cs.DC↗
arXiv2023

Accelerating Brain Simulations with the Fast Multipole Method

The brain is probably the most complex organ in the human body. To understand processes such as learning or healing after brain lesions, we need suitable tools for brain simulations. The Model of Structural Plasticity offers a solution to that problem. It provides a way to model the brain bottom-up by specifying the behavior of the neurons and using structural plasticity to form the synapses. However, its original formulation involves a pairwise evaluation of attraction kernels, which drastically limits scalability. While this complexity has recently been decreased to $O(n\cdot \log^2 n)$ after reformulating the task as a variant of an n-body problem and solving it using an adapted version of the Barnes-Hut approximation, we propose an even faster approximation based on the fast multipole method (FMM). The fast multipole method was initially introduced to solve pairwise interactions in linear time. Our adaptation achieves this time complexity, and it is also faster in practice than the previous approximation.

cs.DC↗
arXiv2022

Simulating Structural Plasticity of the Brain more Scalable than Expected

Structural plasticity of the brain describes the creation of new and the deletion of old synapses over time. Rinke et al. (JPDC 2018) introduced a scalable algorithm that simulates structural plasticity for up to one billion neurons on current hardware using a variant of the Barnes-Hut algorithm. They demonstrate good scalability and prove a runtime complexity of $O(n \log^2 n)$. In this comment paper, we show that with careful consideration of the algorithm and a rigorous proof, the theoretical runtime can even be classified as $O(n \log n)$.

cs.DC↗
arXiv2021

Learning to Make Compiler Optimizations More Effective

Because loops execute their body many times, compiler developers place much emphasis on their optimization. Nevertheless, in view of highly diverse source code and hardware, compilers still struggle to produce optimal target code. The sheer number of possible loop optimizations, including their combinations, exacerbates the problem further. Today's compilers use hard-coded heuristics to decide when, whether, and which of a limited set of optimizations to apply. Often, this leads to highly unstable behavior, making the success of compiler optimizations dependent on the precise way a loop has been written. This paper presents LoopLearner, which addresses the problem of compiler instability by predicting which way of writing a loop will lead to efficient compiled code. To this end, we train a neural network to find semantically invariant source-level transformations for loops that help the compiler generate more efficient code. Our model learns to extract useful features from the raw source code and predicts the speedup that a given transformation is likely to yield. We evaluate LoopLearner with 1,895 loops from various performance-relevant benchmarks. Applying the transformations that our model deems most favorable prior to compilation yields an average speedup of 1.14x. When trying the top-3 suggested transformations, the average speedup even increases to 1.29x. Comparing the approach with an exhaustive search through all available code transformations shows that LoopLearner helps to identify the most beneficial transformations in several orders of magnitude less time.

cs.PL↗
arXiv2020

Extracting Clean Performance Models from Tainted Programs

Performance models are well-known instruments to understand the scaling behavior of parallel applications. They express how performance changes as key execution parameters, such as the number of processes or the size of the input problem, vary. Besides reasoning about program behavior, such models can also be automatically derived from performance data. This is called empirical performance modeling. While this sounds simple at the first glance, this approach faces several serious interrelated challenges, including expensive performance measurements, inaccuracies inflicted by noisy benchmark data, and overall complex experiment design, starting with the selection of the right parameters. The more parameters one considers, the more experiments are needed and the stronger the impact of noise. In this paper, we show how taint analysis, a technique borrowed from the domain of computer security, can substantially improve the modeling process, lowering its cost, improving model quality, and help validate performance models and experimental setups.

cs.DC↗
arXiv2020

Static Neural Compiler Optimization via Deep Reinforcement Learning

The phase-ordering problem of modern compilers has received a lot of attention from the research community over the years, yet remains largely unsolved. Various optimization sequences exposed to the user are manually designed by compiler developers. In designing such a sequence developers have to choose the set of optimization passes, their parameters and ordering within a sequence. Resulting sequences usually fall short of achieving optimal runtime for a given source code and may sometimes even degrade the performance when compared to unoptimized version. In this paper, we employ a deep reinforcement learning approach to the phase-ordering problem. Provided with sub-sequences constituting LLVM's O3 sequence, our agent learns to outperform the O3 sequence on the set of source codes used for training and achieves competitive performance on the validation set, gaining up to 1.32x speedup on previously-unseen programs. Notably, our approach differs from autotuning methods by not depending on one or more test runs of the program for making successful optimization decisions. It has no dependence on any dynamic feature, but only on the statically-attainable intermediate representation of the source code. We believe that the models trained using our approach can be integrated into modern compilers as neural optimization agents, at first to complement, and eventually replace the hand-crafted optimization sequences.

cs.LG↗
arXiv2019

Multipatch Approximation of the de Rham Sequence and its Traces in Isogeometric Analysis

We define a conforming B-spline discretisation of the de Rham complex on multipatch geometries. We introduce and analyse the properties of interpolation operators onto these spaces which commute w.r.t. the surface differential operators. Using these results as a basis, we derive new convergence results of optimal order w.r.t. the respective energy spaces and provide approximation properties of the spline discretisations of trace spaces for application in the theory of isogeometric boundary element methods. Our analysis allows for a straightforward generalisation to finite element methods.

math.NA↗
arXiv2019

Isogeometric Boundary Elements in Electromagnetism: Rigorous Analysis, Fast Methods, and Examples

We present a new approach to three-dimensional electromagnetic scattering problems via fast isogeometric boundary element methods. Starting with an investigation of the theoretical setting around the electric field integral equation within the isogeometric framework, we show existence, uniqueness, and quasi-optimality of the isogeometric approach. For a fast and efficient computation, we then introduce and analyze an interpolation-based fast multipole method tailored to the isogeometric setting, which admits competitive algorithmic and complexity properties. This is followed by a series of numerical examples of industrial scope, together with a detailed presentation and interpretation of the results.

math.NA↗
arXiv2019

Bisimulation Invariant Monadic-Second Order Logic in the Finite

We consider bisimulation-invariant monadic second-order logic over various classes of finite transition systems. We present several combinatorial characterisations of when the expressive power of this fragment coincides with that of the modal mu-calculus. Using these characterisations we prove for some simple classes of transition systems that this is indeed the case. In particular, we show that, over the class of all finite transition systems with Cantor-Bendixson rank at most k, bisimulation-invariant MSO coincides with L_mu.

cs.LO↗
arXiv2018

A Numerical Comparison of an Isogeometric and a Classical Higher-Order Approach to the Electric Field Integral Equation

In this paper, we advocate a novel spline-based isogeometric approach for boundary elements and its efficient implementation. We compare solutions obtained by both an isogeometric approach, and a classical parametric higher-order approach via Raviart-Thomas elements to the solution of the electric field integral equation; i.e., the solution to an electromagnetic scattering problem, promising high convergence orders w.r.t. pointwise error. We discuss both, the obtained accuracy per DOF, as well as the effort required to solve the corresponding system iteratively, on three numerical examples of varying complexity.

cs.CE↗