arXiv subjects

Felix Stollenwerk

Publications and source records attributed to Felix Stollenwerk.

13 recordsLinked to original sources

arXiv2026

Modalities, a PyTorch-native Framework For Large-scale LLM Training and Research

Today's LLM (pre-) training and research workflows typically allocate a significant amount of compute to large-scale ablation studies. Despite the substantial compute costs of these ablations, existing open-source frameworks provide limited tooling for these experiments, often forcing researchers to write their own wrappers and scripts. We propose Modalities, an end-to-end PyTorch-native framework that integrates data-driven LLM research with large-scale model training from two angles. Firstly, by integrating state-of-the-art parallelization strategies, it enables both efficient pretraining and systematic ablations at trillion-token and billion-parameter scale. Secondly, Modalities adopts modular design with declarative, self-contained configuration, enabling reproducibility and extensibility levels that are difficult to achieve out-of-the-box with existing LLM training frameworks.

cs.LG↗

arXiv2026

Output Embedding Centering for Stable LLM Pretraining

Pretraining of large language models is not only expensive but also prone to certain training instabilities. A specific instability that often occurs at the end of training is output logit divergence. The most widely used mitigation strategies, z-loss and logit soft-capping, merely address the symptoms rather than the underlying cause of the problem. In this paper, we analyze the instability from the perspective of the output embeddings' geometry and identify anisotropic embeddings as its source. Based on this, we propose output embedding centering (OEC) as a new mitigation strategy, and demonstrate that it suppresses output logit divergence. OEC can be implemented in two different ways: as a deterministic operation called $\mu$-centering, or a regularization method called $\mu$-loss. Our experiments show that both variants outperform z-loss in terms of training stability, while being on par with logit soft-capping. This holds true both in the presence and the absence of weight tying. As a secondary result, we find that $\mu$-loss is significantly less sensitive to regularization hyperparameter tuning than z-loss.

cs.LG↗

arXiv2025

On the Mathematical Relationship Between Layer Normalization and Dynamic Activation Functions

Layer normalization (LN) is an essential component of modern neural networks. While many alternative techniques have been proposed, none of them have succeeded in replacing LN so far. The latest suggestion in this line of research is a dynamic activation function called Dynamic Tanh (DyT). Although it is empirically well-motivated and appealing from a practical point of view, it lacks a theoretical foundation. In this work, we shed light on the mathematical relationship between LN and dynamic activation functions. In particular, we derive DyT from the LN variant RMSNorm, and show that a well-defined decoupling in derivative space as well as an approximation are needed to do so. By applying the same decoupling procedure directly in function space, we are able to omit the approximation and obtain the exact element-wise counterpart of RMSNorm, which we call Dynamic Inverse Square Root Unit (DyISRU). We demonstrate numerically that DyISRU reproduces the normalization effect on outliers more accurately than DyT does.

cs.LG↗

arXiv2025

Better Embeddings with Coupled Adam

Despite their remarkable capabilities, LLMs learn word representations that exhibit the undesirable yet poorly understood feature of anisotropy. In this paper, we argue that the second moment in Adam is a cause of anisotropic embeddings, and suggest a modified optimizer called Coupled Adam to mitigate the problem. Our experiments demonstrate that Coupled Adam significantly improves the quality of embeddings, while also leading to better upstream and downstream performance on large enough datasets.

cs.CL↗

arXiv2023

nerblackbox: A High-level Library for Named Entity Recognition in Python

We present nerblackbox, a python library to facilitate the use of state-of-the-art transformer-based models for named entity recognition. It provides simple-to-use yet powerful methods to access data and models from a wide range of sources, for fully automated model training and evaluation as well as versatile model inference. While many technical challenges are solved and hidden from the user by default, nerblackbox also offers fine-grained control and a rich set of customizable features. It is thus targeted both at application-oriented developers as well as machine learning experts and researchers.

cs.CL↗

arXiv2023

Annotated Job Ads with Named Entity Recognition

We have trained a named entity recognition (NER) model that screens Swedish job ads for different kinds of useful information (e.g. skills required from a job seeker). It was obtained by fine-tuning KB-BERT. The biggest challenge we faced was the creation of a labelled dataset, which required manual annotation. This paper gives an overview of the methods we employed to make the annotation process more efficient and to ensure high quality data. We also report on the performance of the resulting model.

cs.CL↗

arXiv2023

Text Annotation Handbook: A Practical Guide for Machine Learning Projects

This handbook is a hands-on guide on how to approach text annotation tasks. It provides a gentle introduction to the topic, an overview of theoretical concepts as well as practical advice. The topics covered are mostly technical, but business, ethical and regulatory issues are also touched upon. The focus lies on readability and conciseness rather than completeness and scientific rigor. Experience with annotation and knowledge of machine learning are useful but not required. The document may serve as a primer or reference book for a wide range of professions such as team leaders, project managers, IT architects, software developers and machine learning engineers.

cs.CL↗

arXiv2023

GPT-SW3: An Autoregressive Language Model for the Nordic Languages

This paper details the process of developing the first native large generative language model for the Nordic languages, GPT-SW3. We cover all parts of the development process, from data collection and processing, training configuration and instruction finetuning, to evaluation and considerations for release strategies. We hope that this paper can serve as a guide and reference for other researchers that undertake the development of large generative models for smaller languages.

cs.CL↗

arXiv2023

Training and Evaluation of a Multilingual Tokenizer for GPT-SW3

This paper provides a detailed discussion of the multilingual tokenizer used for GPT-SW3. It was trained on the Nordic Pile using the SentencePiece library and the BPE algorithm. We outline the tokenizer's most important features and share details on its learned vocabulary. In addition, we systematically analyze the properties and evaluate the performance of the tokenizer with regard to the different languages present in the data.

cs.CL↗

arXiv2022

Adaptive Fine-Tuning of Transformer-Based Language Models for Named Entity Recognition

The current standard approach for fine-tuning transformer-based language models includes a fixed number of training epochs and a linear learning rate schedule. In order to obtain a near-optimal model for the given downstream task, a search in optimization hyperparameter space is usually required. In particular, the number of training epochs needs to be adjusted to the dataset size. In this paper, we introduce adaptive fine-tuning, which is an alternative approach that uses early stopping and a custom learning rate schedule to dynamically adjust the number of training epochs to the dataset size. For the example use case of named entity recognition, we show that our approach not only makes hyperparameter search with respect to the number of training epochs redundant, but also leads to improved results in terms of performance, stability and efficiency. This holds true especially for small datasets, where we outperform the state-of-the-art fine-tuning method by a large margin.

cs.CL↗

arXiv2018

Symanzik Improvement with Dynamical Charm: A 3+1 Scheme for Wilson Quarks

We discuss the problem of lattice artefacts in QCD simulations enhanced by the introduction of dynamical charmed quarks. In particular, we advocate the use of a massive renormalization scheme with a close to realistic charm mass. To maintain O(a) improvement for Wilson type fermions in this case we define a finite size scheme and carry out a nonperturbative estimation of the clover coefficient $c_\mathrm{sw}$. It is summarized in a fit formula $c_\mathrm{sw}(g_0^2)$ that defines an improved action suitable for future dynamical charm simulations.

hep-lat↗

arXiv2015

Determination of c_sw in N_f=3+1 Lattice QCD with massive Wilson fermions

We develop a strategy for the non-perturbative determination of the O(a)-improvement coefficient c_sw for Wilson fermions with massive sea quarks. The improvement condition is defined via the PCAC relation in the Schrödinger functional. It is imposed along a line of constant physics designed to be close to the correct mass of the charm quark. The numerical work uses the tree-level improved Lüscher-Weisz gauge action in N_f=3+1 Lattice QCD.

hep-lat↗

arXiv2013

Critical slowing down and the gradient flow coupling in the Schrödinger functional

We study the sensitivity of the gradient flow coupling to sectors of different topological charge and its implications in practical situations. Furthermore, we investigate an alternative definition of the running coupling that is expected to be less sensitive to the problems of the HMC algorithm to efficiently sample all topological sectors.

hep-lat↗