Searcharxiv⌕ Search

arXiv subjects

Stefan Kesselheim

Publications and source records attributed to Stefan Kesselheim.

28 records · Page 2Linked to original sources
arXiv2024

Scaling Image Tokenizers with Grouped Spherical Quantization

Vision tokenizers have gained a lot of attraction due to their scalability and compactness; previous works depend on old-school GAN-based hyperparameters, biased comparisons, and a lack of comprehensive analysis of the scaling behaviours. To tackle those issues, we introduce Grouped Spherical Quantization (GSQ), featuring spherical codebook initialization and lookup regularization to constrain codebook latent to a spherical surface. Our empirical analysis of image tokenizer training strategies demonstrates that GSQ-GAN achieves superior reconstruction quality over state-of-the-art methods with fewer training iterations, providing a solid foundation for scaling studies. Building on this, we systematically examine the scaling behaviours of GSQ, specifically in latent dimensionality, codebook size, and compression ratios, and their impact on model performance. Our findings reveal distinct behaviours at high and low spatial compression levels, underscoring challenges in representing high-dimensional latent spaces. We show that GSQ can restructure high-dimensional latent into compact, low-dimensional spaces, thus enabling efficient scaling with improved quality. As a result, GSQ-GAN achieves a 16x down-sampling with a reconstruction FID (rFID) of 0.50.

cs.CV↗
arXiv2024

Tokenizer Choice For LLM Training: Negligible or Crucial?

The recent success of Large Language Models (LLMs) has been predominantly driven by curating the training dataset composition, scaling of model architectures and dataset sizes and advancements in pretraining objectives, leaving tokenizer influence as a blind spot. Shedding light on this underexplored area, we conduct a comprehensive study on the influence of tokenizer choice on LLM downstream performance by training 24 mono- and multilingual LLMs at a 2.6B parameter scale, ablating different tokenizer algorithms and parameterizations. Our studies highlight that the tokenizer choice can significantly impact the model's downstream performance and training costs. In particular, we find that the common tokenizer evaluation metrics fertility and parity are not always predictive of model downstream performance, rendering these metrics a questionable proxy for the model's downstream performance. Furthermore, we show that multilingual tokenizers trained on the five most frequent European languages require vocabulary size increases of factor three in comparison to English. While English-centric tokenizers have been applied to the training of multi-lingual LLMs in the past, we find that this approach results in a severe downstream performance degradation and additional training costs of up to 68%, due to an inefficient tokenization vocabulary.

cs.LG↗
arXiv2023

Physics informed Neural Networks applied to the description of wave-particle resonance in kinetic simulations of fusion plasmas

The Vlasov-Poisson system is employed in its reduced form version (1D1V) as a test bed for the applicability of Physics Informed Neural Network (PINN) to the wave-particle resonance. Two examples are explored: the Landau damping and the bump-on-tail instability. PINN is first tested as a compression method for the solution of the Vlasov-Poisson system and compared to the standard neural networks. Second, the application of PINN to solving the Vlasov-Poisson system is also presented with the special emphasis on the integral part, which motivates the implementation of a PINN variant, called Integrable PINN (I-PINN), based on the automatic-differentiation to solve the partial differential equation and on the automatic-integration to solve the integral equation.

physics.comp-ph↗
arXiv2023

A Comparative Study on Generative Models for High Resolution Solar Observation Imaging

Solar activity is one of the main drivers of variability in our solar system and the key source of space weather phenomena that affect Earth and near Earth space. The extensive record of high resolution extreme ultraviolet (EUV) observations from the Solar Dynamics Observatory (SDO) offers an unprecedented, very large dataset of solar images. In this work, we make use of this comprehensive dataset to investigate capabilities of current state-of-the-art generative models to accurately capture the data distribution behind the observed solar activity states. Starting from StyleGAN-based methods, we uncover severe deficits of this model family in handling fine-scale details of solar images when training on high resolution samples, contrary to training on natural face images. When switching to the diffusion based generative model family, we observe strong improvements of fine-scale detail generation. For the GAN family, we are able to achieve similar improvements in fine-scale generation when turning to ProjectedGANs, which uses multi-scale discriminators with a pre-trained frozen feature extractor. We conduct ablation studies to clarify mechanisms responsible for proper fine-scale handling. Using distributed training on supercomputers, we are able to train generative models for up to 1024x1024 resolution that produce high quality samples indistinguishable to human experts, as suggested by the evaluation we conduct. We make all code, models and workflows used in this study publicly available at \url{https://github.com/SLAMPAI/generative-models-for-highres-solar-images}.

cs.CV↗
arXiv2022

Hearts Gym: Learning Reinforcement Learning as a Team Event

Amidst the COVID-19 pandemic, the authors of this paper organized a Reinforcement Learning (RL) course for a graduate school in the field of data science. We describe the strategy and materials for creating an exciting learning experience despite the ubiquitous Zoom fatigue and evaluate the course qualitatively. The key organizational features are a focus on a competitive hands-on setting in teams, supported by a minimum of lectures providing the essential background on RL. The practical part of the course revolved around Hearts Gym, an RL environment for the card game Hearts that we developed as an entry-level tutorial to RL. Participants were tasked with training agents to explore reward shaping and other RL hyperparameters. For a final evaluation, the agents of the participants competed against each other.

cs.CY↗
arXiv2021

JUWELS Booster -- A Supercomputer for Large-Scale AI Research

In this article, we present JUWELS Booster, a recently commissioned high-performance computing system at the Jülich Supercomputing Center. With its system architecture, most importantly its large number of powerful Graphics Processing Units (GPUs) and its fast interconnect via InfiniBand, it is an ideal machine for large-scale Artificial Intelligence (AI) research and applications. We detail its system architecture, parallel, distributed model training, and benchmarks indicating its outstanding performance. We exemplify its potential for research application by presenting large-scale AI research highlights from various scientific fields that require such a facility.

cs.DC↗
arXiv2013

Hydrodynamic Correlations slow down Crystallization of Soft Colloids

Crystallization is often assumed to be a quasi-static process that is unaffected by details of particle transport other than the bulk diffusion coefficient. Therefore colloidal suspensions are frequently argued to be an ideal toy model for experimentally more difficult systems such as metal melts. In this letter, we want to challenge this assumption. To this aim, we have considered molecular dynamics simulations of the crystallization in a suspension of Yukawa-type colloids. In order to investigate the role of hydrodynamic interactions (HIs) mediated by the solvent, we modeled the solvent both implicitly and explicitly, using Langevin dynamics and the fluctuating Lattice Boltzmann method, respectively. Our simulations show a dramatic reduction of the crystal growth velocity due to HIs even at moderate hydrodynamic coupling. A detailed analysis shows that this slowdown is due to the wall-like properties of the crystal surface, which reduces the colloidal diffusion towards the crystal surface by hydrodynamic screening. Crystallization in suspensions therefore differs strongly from pure melts, making them less useful as a toy model than previously thought.

cond-mat.stat-mech↗
arXiv2012

Investigation of tracer diffusion in crowded cylindrical channel

Based on a coarse-grained model, we carry out molecular dynamics simulations to analyze the diffusion of a small tracer particle inside a cylindrical channel whose inner wall is covered with randomly grafted short polymeric chains. We observe an interesting transient subdiffusive behavior along the cylindrical axis at high attraction between the tracer and the chains, however, the long time diffusion is always normal. This process is found to be enhanced for the case that we immobilize the grafted chains, i.e. the sub-diffusive behavior sets in at an earlier time and spans over a longer time period before becoming diffusive. Even if the grafted chains are replaced with a frozen sea of repulsive, non-connected particles in the background, the transient subdiffusion is observed. The intermediate subdiffusive behavior only disappears when the grafted chains are replaced with a mobile background sea of mutually repulsive particles. Overall, the long time diffusion coefficient of the tracer along the cylinder axis decreases with the increase in system volume fraction, strength of attraction between the tracer and the background and also on freezing the background. We believe that the simple model presented here could be useful for a qualitative understanding of the process of macromolecular diffusion inside the nuclear pore complex.

cond-mat.soft↗
arXiv2010

Influence of pore dielectric boundaries on the translocation barrier of DNA

We investigate the impact of dielectric boundary forces on the translocation process of charged rigid DNA segments through solid neutral nanopores. We assess the electrostatic contribution to the translocation free energy barrier of a model DNA segment by evaluating the potential of mean force in absence and presence of polarization effects by means of coarse-grained molecular dynamics simulations. The effect of induced polarization charges has been taken into account by employing ICC*, a recently developed algorithm that can efficiently compute induced polarization charges induced on suitably discretized dielectric boundaries. Since water has a higher dielectric constant than the pore walls, polarization effects repel charged objects in the vicinity of the interface, with the effect of significantly increasing the free energy barrier. Another investigated side effect is the change of the counterion distribution around the charged polymer in presence of the induced pore charges. Furthermore we investigate the influence of adding salt to the solution.

cond-mat.soft↗