arXiv subjects

Judith Mueller

Publications and source records attributed to Judith Mueller.

4 recordsLinked to original sources

arXiv2026

Better Models, Faster Training: Sigmoid Attention for single-cell Foundation Models

Training stable biological foundation models requires rethinking attention mechanisms: we find that using sigmoid attention as a drop in replacement for softmax attention a) produces better learned representations: on six diverse single-cell datasets, sigmoid achieves 25% higher cell-type separation, better cell-type cohesion metrics, and lower validation loss, b) faster training, models with sigmoid attention train up to 10% faster than their softmax counterparts, and c) more stable training by eliminating inherent sources of instability in softmax attention. We establish that sigmoid attention has globally bounded derivatives ($\leq 0.25$) as opposed to softmax, and a diagonal Jacobian structure in contrast with softmax's dense coupling, which together help alleviate training instabilities. In stress tests on 160M-parameter bidirectional attention models trained without gradient clipping on 8K-token sequences, softmax diverges catastrophically, with gradients exploding by four orders of magnitude, while sigmoid remains stable. Finally, we implement and open-source TritonSigmoid, an efficient GPU kernel that achieves 515 TFLOPS on H100 GPUs, outperforming both FlashAttention-2 and FlashSigmoid, with native padding support, which is essential for biological sequences. Our results establish sigmoid attention as both theoretically grounded and empirically superior for biological foundation models. Code is available at https://github.com/MSDLLCpapers/triton-sigmoid

cs.LG↗

arXiv2025

TEDDY: A Family Of Foundation Models For Understanding Single Cell Biology

Understanding the biological mechanisms of disease is crucial for medicine, and in particular, for drug discovery. AI-powered analysis of genome-scale biological data holds great potential in this regard. The increasing availability of single-cell RNA sequencing data has enabled the development of large foundation models for disease biology. However, existing foundation models only modestly improve over task-specific models in downstream applications. Here, we explored two avenues for improving single-cell foundation models. First, we scaled the pre-training data to a diverse collection of 116 million cells, which is larger than those used by previous models. Second, we leveraged the availability of large-scale biological annotations as a form of supervision during pre-training. We trained the \model family of models comprising six transformer-based state-of-the-art single-cell foundation models with 70 million, 160 million, and 400 million parameters. We vetted our models on several downstream evaluation tasks, including identifying the underlying disease state of held-out donors not seen during training, distinguishing between diseased and healthy cells for disease conditions and donors not seen during training, and probing the learned representations for known biology. Our models showed substantial improvement over existing works, and scaling experiments showed that performance improved predictably with both data volume and parameter count.

cs.LG↗

arXiv2000

Phase field under stress

A phase-field approach describing the dynamics of a strained solid in contact with its melt is developed. By rigorous asymptotic analysis we show that the sharp-interface limit of this model recovers the continuum model equations for the Grinfeld instability. Moreover, we use our approach to derive hitherto unknown sharp-interface equations for a situation including a field of body forces. The numerical utility of the phase-field approach is demonstrated by comparison with a sharp-interface simulation. We then investigate the dynamics of extended systems within the phase-field model which contains an inherent lower length cutoff, thus avoiding cusp singularities. It is found that a periodic array of grooves generically evolves into a superstructure which arises from a series of imperfect period doublings. For wavenumbers close to the fastest-growing mode of the linear instability, the first period doubling can be obtained analytically. Both the dynamics of an initially periodic array and a random initial structure can be described as a coarsening process with winning grooves temporarily accelerating whereas losing ones decelerate and even reverse their direction of motion.

cond-mat.stat-mech↗

arXiv1998

Model of surface instabilities induced by stress

We propose a model based on a Ginzburg-Landau approach to study a strain relief mechanism at a free interface of a non-hydrostatically stressed solid, commonly observed in thin-film growth. The evolving instability, known as the Grinfeld instability, is studied numerically in two and three dimensions. Inherent in the description is the proper treatment of nonlinearities. We find these nonlinearities can lead to competitive coarsening of interfacial structures, corresponding to different wavenumbers, as strain is relieved. We suggest ways to experimentally measure this coarsening.

cond-mat.mtrl-sci↗