arXiv subjects

Vincent Abbott

Publications and source records attributed to Vincent Abbott.

6 recordsLinked to original sources

arXiv2026

Weaves, Wires, and Morphisms: Formalizing and Implementing the Algebra of Deep Learning

Despite deep learning models running well-defined mathematical functions, we lack a formal mathematical framework for describing model architectures. Ad-hoc notation, diagrams, and pseudocode poorly handle nonlinear broadcasting and the relationship between individual components and composed models. This paper introduces a categorical framework for deep learning models that formalizes broadcasting through the novel axis-stride and array-broadcasted categories. This allows the mathematical function underlying architectures to be precisely expressed and manipulated in a compositional manner. These mathematical definitions are translated into human manageable diagrams and machine manageable data structures. We provide a mirrored implementation in Python (pyncd) and TypeScript (tsncd) to show the universal aspect of our framework, along with features including algebraic construction, graph conversion, PyTorch compilation and diagram rendering. This lays the foundation for a systematic, formal approach to deep learning model design and analysis.

cs.LG↗

arXiv2025

Accelerating Machine Learning Systems via Category Theory: Applications to Spherical Attention for Gene Regulatory Networks

How do we enable artificial intelligence models to improve themselves? This is central to exponentially improving generalized artificial intelligence models, which can improve their own architecture to handle new problem domains in an efficient manner that leverages the latest hardware. However, current automated compilation methods are poor, and efficient algorithms require years of human development. In this paper, we use neural circuit diagrams, based in category theory, to prove a general theorem related to deep learning algorithms, guide the development of a novel attention algorithm catered to the domain of gene regulatory networks, and produce a corresponding efficient kernel. The algorithm we propose, spherical attention, shows that neural circuit diagrams enable a principled and systematic method for reasoning about deep learning architectures and providing high-performance code. By replacing SoftMax with an $L^2$ norm as suggested by diagrams, it overcomes the special function unit bottleneck of standard attention while retaining the streaming property essential to high-performance. Our diagrammatically derived \textit{FlashSign} kernel achieves comparable performance to the state-of-the-art, fine-tuned FlashAttention algorithm on an A100, and $3.6\times$ the performance of PyTorch. Overall, this investigation shows neural circuit diagrams' suitability as a high-level framework for the automated development of efficient, novel artificial intelligence architectures.

math.CT↗

arXiv2024

FlashAttention on a Napkin: A Diagrammatic Approach to Deep Learning IO-Awareness

Optimizing deep learning algorithms currently requires slow, manual derivation, potentially leaving much performance untapped. Methods like FlashAttention have achieved a x6 performance improvement over native PyTorch by avoiding unnecessary data transfers, but required three iterations over three years to be developed. Automated compiled methods have consistently lagged behind. This paper extends Neural Circuit Diagrams for deep learning models to consider resource usage and the distribution of tasks across a GPU hierarchy. We show how diagrams can use simple relabellings to derive high-level streaming and tiling optimization strategies along with performance models. We show how this high-level performance model allows the effects of quantization and multi-level GPU hierarchies to be readily considered. We develop a methodology for representing intermediate-level pseudocode with diagrams, allowing hardware-aware algorithms to be derived step-by-step. Finally, we show how our methodology can be used to better understand existing techniques like FlashAttention. This work uses a theoretical framework to link assumptions about GPU behaviour to claims about performance. We aim to lay the groundwork for a scientific approach to GPU optimization where experiments can address clear hypotheses rather than post-hoc rationalizations.

cs.LG↗

arXiv2024

Diagrammatic Negative Information

The flow of information through a complex system can be readily understood with category theory. However, negative information (e.g., what is not possible) does not have an immediately evident categorical representation. The formalization of nategories using unconventional composition addresses this issue, and lets imposed limitations on categories be considered. However, traditional nategories abandon core categorical constructs and rely on extensive mathematical development. This creates a divide between the consideration of positive and negative information composition. In this work, we show that negative information can be considered in a natural categorical manner. This is aided by functor string diagrams, a novel flexible diagrammatic approach that can intuitively show the operation of hom-functors and natural transformations in expressions. This insight reveals how to consider the composition of negative information with foundational categorical constructs without relying on enrichment. We present diagrammatic means to consider not only nategories, but preorders more broadly. This paper introduces diagrammatic methods for the consideration of triangle inequalities and co-designs $\mathbf{DP/Feas_{Bool}}$, showing how important cases of negative information composition can be categorically and diagrammatically approached. In particular, we develop systematic tools to rigorously consider imposed limitations on systems, advancing our mathematical understanding, and present intuitive diagrams which motivate widespread adoption and usage for various applications.

math.CT↗

arXiv2024

Functor String Diagrams: A Novel Approach to Flexible Diagrams for Applied Category Theory

The study of abstraction and composition - the focus of category theory - naturally leads to sophisticated diagrams which can encode complex algebraic semantics. Consequently, these diagrams facilitate a clearer visual comprehension of diverse theoretical and applied systems. Complex algebraic structures - otherwise represented by a forest of symbols - can be encoded into diagrams with intuitive graphical rules. The prevailing paradigm for diagrammatic category theory are monoidal string diagrams whose specification reflects the axioms of monoidal categories. However, such diagrams struggle in accurately portraying crucial categorical constructs such as functors or natural transformations, obscuring central concepts such as the Yoneda lemma or the simultaneous consideration of hom-functors and products. In this work, we introduce functor string diagrams, a systematic approach for the development of categorical diagrams which allows functors, natural transformations, and products to be clearly represented. We validate their practicality in multiple dimensions. We show their usefulness for theoretical manipulations by proving the Yoneda lemma, show that they encompass monoidal string diagrams and hence their helpful properties, and end by showing their exceptional applied utility by leveraging them to underpin neural circuit diagrams, a method which, at last, allows deep learning architectures to be comprehensively and rigorously expressed.

math.CT↗

arXiv2024

Neural Circuit Diagrams: Robust Diagrams for the Communication, Implementation, and Analysis of Deep Learning Architectures

Diagrams matter. Unfortunately, the deep learning community has no standard method for diagramming architectures. The current combination of linear algebra notation and ad-hoc diagrams fails to offer the necessary precision to understand architectures in all their detail. However, this detail is critical for faithful implementation, mathematical analysis, further innovation, and ethical assurances. I present neural circuit diagrams, a graphical language tailored to the needs of communicating deep learning architectures. Neural circuit diagrams naturally keep track of the changing arrangement of data, precisely show how operations are broadcast over axes, and display the critical parallel behavior of linear operations. A lingering issue with existing diagramming methods is the inability to simultaneously express the detail of axes and the free arrangement of data, which neural circuit diagrams solve. Their compositional structure is analogous to code, creating a close correspondence between diagrams and implementation. In this work, I introduce neural circuit diagrams for an audience of machine learning researchers. After introducing neural circuit diagrams, I cover a host of architectures to show their utility and breed familiarity. This includes the transformer architecture, convolution (and its difficult-to-explain extensions), residual networks, the U-Net, and the vision transformer. I include a Jupyter notebook that provides evidence for the close correspondence between diagrams and code. Finally, I examine backpropagation using neural circuit diagrams. I show their utility in providing mathematical insight and analyzing algorithms' time and space complexities.

cs.LG↗