arXiv subjects

Thomas Gebhart

Publications and source records attributed to Thomas Gebhart.

14 recordsLinked to original sources

arXiv2025

Opening Knowledge Gaps Drives Scientific Progress

Knowledge production is often viewed as an endogenous process in which discovery arises through the recombination of existing theories, findings, and concepts. Yet given the vast space of potential recombinations, not all are equally valuable, and identifying those that may prove most generative remains challenging. We argue that a crucial form of recombination occurs when linking concepts creates knowledge gaps-empty regions in the conceptual landscape that focus scientific attention on proximal, unexplored connections and signal promising directions for future research. Using computational topology, we develop a method to systematically identify knowledge gaps in science at scale. Applying this approach to millions of articles from Microsoft Academic Graph (n = 34,363,623) over a 120-year period (1900-2020), we uncover papers that create topological gaps in concept networks, tracking how these gap-opening works reshape the scientific knowledge landscape. Our results indicate that gap-opening papers are more likely to rank among the most highly cited works (top 1-20%) compared with papers that do not introduce novel concept pairings. In contrast, papers that introduce novel combinations without opening gaps are not more likely to rank in the top 1% for citation counts, and are even less likely than baseline papers to appear in the top 5% to 20%. Our findings also suggest that gap-opening papers are more disruptive, highlighting their generative role in stimulating new directions for scientific inquiry.

cs.CY↗

arXiv2025

Sheaf theory: from deep geometry to deep learning

This paper provides an overview of the applications of sheaf theory in deep learning, data science, and computer science in general. The primary text of this work serves as a friendly introduction to applied and computational sheaf theory accessible to those with modest mathematical familiarity. We describe intuitions and motivations underlying sheaf theory shared by both theoretical researchers and practitioners, bridging classical mathematical theory and its more recent implementations within signal processing and deep learning. We observe that most notions commonly considered specific to cellular sheaves translate to sheaves on arbitrary posets, providing an interesting avenue for further generalization of these methods in applications, and we present a new algorithm to compute sheaf cohomology on arbitrary finite posets in response. By integrating classical theory with recent applications, this work reveals certain blind spots in current machine learning practices. We conclude with a list of problems related to sheaf-theoretic applications that we find mathematically insightful and practically instructive to solve. To ensure the exposition of sheaf theory is self-contained, a rigorous mathematical introduction is provided in appendices which moves from an introduction of diagrams and sheaves to the definition of derived functors, higher order cohomology, sheaf Laplacians, sheaf diffusion, and interconnections of these subjects therein.

math.AT↗

arXiv2024

ICML Topological Deep Learning Challenge 2024: Beyond the Graph Domain

This paper describes the 2nd edition of the ICML Topological Deep Learning Challenge that was hosted within the ICML 2024 ELLIS Workshop on Geometry-grounded Representation Learning and Generative Modeling (GRaM). The challenge focused on the problem of representing data in different discrete topological domains in order to bridge the gap between Topological Deep Learning (TDL) and other types of structured datasets (e.g. point clouds, graphs). Specifically, participants were asked to design and implement topological liftings, i.e. mappings between different data structures and topological domains --like hypergraphs, or simplicial/cell/combinatorial complexes. The challenge received 52 submissions satisfying all the requirements. This paper introduces the main scope of the challenge, and summarizes the main results and findings.

cs.LG↗

arXiv2023

Feature Propagation on Knowledge Graphs using Cellular Sheaves

Many inference tasks on knowledge graphs, including relation prediction, operate on knowledge graph embeddings -- vector representations of the vertices (entities) and edges (relations) that preserve task-relevant structure encoded within the underlying combinatorial object. Such knowledge graph embeddings can be modeled as an approximate global section of a cellular sheaf, an algebraic structure over the graph. Using the diffusion dynamics encoded by the corresponding sheaf Laplacian, we optimally propagate known embeddings of a subgraph to inductively represent new entities introduced into the knowledge graph at inference time. We implement this algorithm via an efficient iterative scheme and show that on a number of large-scale knowledge graph embedding benchmarks, our method is competitive with -- and in some scenarios outperforms -- more complex models derived explicitly for inductive knowledge graph reasoning tasks.

cs.AI↗

arXiv2023

A Mathematical Framework for Citation Disruption

Many theories of scientific and technological progress imagine science as an iterative, developmental process periodically interrupted by innovations which disrupt and restructure the status quo. Due to the immense societal value created by these disruptive scientific and technological innovations, accurately operationalizing this perspective into quantifiable terms represents a key challenge for researchers seeking to understand the history and mechanisms underlying scientific and technological progress. Researchers have recently proposed a number of quantitative measures that seek to quantify the extent to which works in science and technology are disruptive with respect to their scientific context. While these disruption measures show promise in their ability to quantify potentially disruptive works of science and technology, their definitions are bespoke to the science of science and lack a broader theoretical framework, obscuring their interrelationships and limiting their adoption within broader network science paradigms. We propose a mathematical framework for conceptualizing and measuring disruptive scientific contributions within citation networks through the lens of network centrality, and formally relate the CD Index disruption measure and its variants to betweenness centrality. By reinterpreting disruption through the lens of centrality, we unify a number of existing citation-based disruption measures while simultaneously providing natural generalizations which enjoy empirical and computational efficiencies. We validate these theoretical observations by computing a variety of disruption measures on real citation data and find that computing these centrality-based disruption measures over ego networks of increasing radius results in better discernment of award-winning scientific innovations relative to conventional disruption metrics which rely on local citation context alone.

cs.SI↗

arXiv2023

Knowledge Sheaves: A Sheaf-Theoretic Framework for Knowledge Graph Embedding

Knowledge graph embedding involves learning representations of entities -- the vertices of the graph -- and relations -- the edges of the graph -- such that the resulting representations encode the known factual information represented by the knowledge graph and can be used in the inference of new relations. We show that knowledge graph embedding is naturally expressed in the topological and categorical language of \textit{cellular sheaves}: a knowledge graph embedding can be described as an approximate global section of an appropriate \textit{knowledge sheaf} over the graph, with consistency constraints induced by the knowledge graph's schema. This approach provides a generalized framework for reasoning about knowledge graph embedding models and allows for the expression of a wide range of prior constraints on embeddings. Further, the resulting embeddings can be easily adapted for reasoning over composite relations without special training. We implement these ideas to highlight the benefits of the extensions inspired by this new perspective.

cs.LG↗

arXiv2022

Graph Convolutional Networks from the Perspective of Sheaves and the Neural Tangent Kernel

Graph convolutional networks are a popular class of deep neural network algorithms which have shown success in a number of relational learning tasks. Despite their success, graph convolutional networks exhibit a number of peculiar features, including a bias towards learning oversmoothed and homophilic functions, which are not easily diagnosed due to the complex nature of these algorithms. We propose to bridge this gap in understanding by studying the neural tangent kernel of sheaf convolutional networks--a topological generalization of graph convolutional networks. To this end, we derive a parameterization of the neural tangent kernel for sheaf convolutional networks which separates the function into two parts: one driven by a forward diffusion process determined by the graph, and the other determined by the composite effect of nodes' activations on the output layer. This geometrically-focused derivation produces a number of immediate insights which we discuss in detail.

cs.LG↗

arXiv2021

Go with the Flow? A Large-Scale Analysis of Health Care Delivery Networks in the United States Using Hodge Theory

Health care delivery is a collaborative process, requiring close coordination among networks of providers with specialized expertise. Yet in the United States, care is often spread across multiple disconnected providers (e.g., primary care physicians, specialists), leading to fragmented care delivery networks, and contributing to higher costs and lower quality. While this problem is well known, there are relatively few quantitative tools available for characterizing care delivery networks at scale, thereby inhibiting deeper understanding of care fragmentation and efforts to address it. In this, study, we conduct a large-scale analysis of care delivery networks across the United States using the discrete Hodge decomposition, an emerging method of topological data analysis. Using this technique, we decompose networks of patient flows among physicians into three orthogonal subspaces: gradient (acyclic flow), harmonic (global cyclic flow), and curl (local cyclic flow). We document substantial variation in the relative importance of each subspace, suggesting that there may be systematic differences in the organization of care delivery networks across health care markets. Moreover, we find that the relative importance of each subspace is predictive of local care cost and quality, with outcomes tending to be better with greater curl flow and worse with greater harmonic flow.

cs.SI↗

arXiv2021

A Unified Paths Perspective for Pruning at Initialization

A number of recent approaches have been proposed for pruning neural network parameters at initialization with the goal of reducing the size and computational burden of models while minimally affecting their training dynamics and generalization performance. While each of these approaches have some amount of well-founded motivation, a rigorous analysis of the effect of these pruning methods on network training dynamics and their formal relationship to each other has thus far received little attention. Leveraging recent theoretical approximations provided by the Neural Tangent Kernel, we unify a number of popular approaches for pruning at initialization under a single path-centric framework. We introduce the Path Kernel as the data-independent factor in a decomposition of the Neural Tangent Kernel and show the global structure of the Path Kernel can be computed efficiently. This Path Kernel decomposition separates the architectural effects from the data-dependent effects within the Neural Tangent Kernel, providing a means to predict the convergence dynamics of a network from its architecture alone. We analyze the use of this structure in approximating training and generalization performance of networks in the absence of data across a number of initialization pruning approaches. Observing the relationship between input data and paths and the relationship between the Path Kernel and its natural norm, we additionally propose two augmentations of the SynFlow algorithm for pruning at initialization.

cs.LG↗

arXiv2020

Sheaf Neural Networks

We present a generalization of graph convolutional networks by generalizing the diffusion operation underlying this class of graph neural networks. These sheaf neural networks are based on the sheaf Laplacian, a generalization of the graph Laplacian that encodes additional relational structure parameterized by the underlying graph. The sheaf Laplacian and associated matrices provide an extended version of the diffusion operation in graph convolutional networks, providing a proper generalization for domains where relations between nodes are non-constant, asymmetric, and varying in dimension. We show that the resulting sheaf neural networks can outperform graph convolutional networks in domains where relations between nodes are asymmetric and signed.

cs.LG↗

arXiv2020

The Emergence of Higher-Order Structure in Scientific and Technological Knowledge Networks

The growth of science and technology is a recombinative process, wherein new discoveries and inventions are built from prior knowledge. Yet relatively little is known about the manner in which scientific and technological knowledge develop and coalesce into larger structures that enable or constrain future breakthroughs. Network science has recently emerged as a framework for measuring the structure and dynamics of knowledge. While helpful, existing approaches struggle to capture the global properties of the underlying networks, leading to conflicting observations about the nature of scientific and technological progress. We bridge this methodological gap using tools from algebraic topology to characterize the higher-order structure of knowledge networks in science and technology across scale. We observe rapid growth in the higher-order structure of knowledge in many scientific and technological fields. This growth is not observable using traditional network measures. We further demonstrate that the emergence of higher-order structure coincides with decline in lower-order structure, and has historically far outpaced the corresponding emergence of higher-order structure in scientific and technological collaboration networks. Up to a point, increases in higher-order structure are associated with better outcomes, as measured by the novelty and impact of papers and patents. However, the nature of science and technology produced under higher-order regimes also appears to be qualitatively different from that produced under lower-order ones, with the former exhibiting greater linguistic abstractness and greater tendencies for building upon prior streams of knowledge.

cs.SI↗

arXiv2019

Path homologies of deep feedforward networks

We provide a characterization of two types of directed homology for fully-connected, feedforward neural network architectures. These exact characterizations of the directed homology structure of a neural network architecture are the first of their kind. We show that the directed flag homology of deep networks reduces to computing the simplicial homology of the underlying undirected graph, which is explicitly given by Euler characteristic computations. We also show that the path homology of these networks is non-trivial in higher dimensions and depends on the number and size of the layers within the network. These results provide a foundation for investigating homological differences between neural network architectures and their realized structure as implied by their parameters.

math.AT↗

arXiv2019

Characterizing the Shape of Activation Space in Deep Neural Networks

The representations learned by deep neural networks are difficult to interpret in part due to their large parameter space and the complexities introduced by their multi-layer structure. We introduce a method for computing persistent homology over the graphical activation structure of neural networks, which provides access to the task-relevant substructures activated throughout the network for a given input. This topological perspective provides unique insights into the distributed representations encoded by neural networks in terms of the shape of their activation structures. We demonstrate the value of this approach by showing an alternative explanation for the existence of adversarial examples. By studying the topology of network activations across multiple architectures and datasets, we find that adversarial perturbations do not add activations that target the semantic structure of the adversarial class as previously hypothesized. Rather, adversarial examples are explainable as alterations to the dominant activation structures induced by the original image, suggesting the class representations learned by deep networks are problematically sparse on the input space.

cs.LG↗

arXiv2017

Adversary Detection in Neural Networks via Persistent Homology

We outline a detection method for adversarial inputs to deep neural networks. By viewing neural network computations as graphs upon which information flows from input space to out- put distribution, we compare the differences in graphs induced by different inputs. Specifically, by applying persistent homology to these induced graphs, we observe that the structure of the most persistent subgraphs which generate the first homology group differ between adversarial and unperturbed inputs. Based on this observation, we build a detection algorithm that depends only on the topological information extracted during training. We test our algorithm on MNIST and achieve 98% detection adversary accuracy with F1-score 0.98.

cs.LG↗