arXiv subjects

Thomas Markovich

Publications and source records attributed to Thomas Markovich.

16 recordsLinked to original sources

arXiv2025

Uncertainty Estimation on Graphs with Structure Informed Stochastic Partial Differential Equations

Graph Neural Networks have achieved impressive results across diverse network modeling tasks, but accurately estimating uncertainty on graphs remains difficult, especially under distributional shifts. Unlike traditional uncertainty estimation, graph-based uncertainty must account for randomness arising from both the graph's structure and its label distribution, which adds complexity. In this paper, making an analogy between the evolution of a stochastic partial differential equation (SPDE) driven by Matern Gaussian Process and message passing using GNN layers, we present a principled way to design a novel message passing scheme that incorporates spatial-temporal noises motivated by the Gaussian Process approach to SPDE. Our method simultaneously captures uncertainty across space and time and allows explicit control over the covariance kernel smoothness, thereby enhancing uncertainty estimates on graphs with both low and high label informativeness. Our extensive experiments on Out-of-Distribution (OOD) detection on graph datasets with varying label informativeness demonstrate the soundness and superiority of our model to existing approaches.

cs.LG↗

arXiv2025

Understanding the Design Principles of Link Prediction in Directed Settings

Link prediction is a widely studied task in Graph Representation Learning (GRL) for modeling relational data. The early theories in GRL were based on the assumption of a symmetric adjacency matrix, reflecting an undirected setting. As a result, much of the following state-of-the-art research has continued to operate under this symmetry assumption, even though real-world data often involve crucial information conveyed through the direction of relationships. This oversight limits the ability of these models to fully capture the complexity of directed interactions. In this paper, we focus on the challenge of directed link prediction by evaluating key heuristics that have been successful in undirected settings. We propose simple but effective adaptations of these heuristics to the directed link prediction task and demonstrate that these modifications produce competitive performance compared to the leading Graph Neural Networks (GNNs) originally designed for undirected graphs. Through an extensive set of experiments, we derive insights that inform the development of a novel framework for directed link prediction, which not only surpasses baseline methods but also outperforms state-of-the-art GNNs on multiple benchmarks.

cs.LG↗

arXiv2024

Scaling Laws for Discriminative Classification in Large Language Models

Modern large language models (LLMs) represent a paradigm shift in what can plausibly be expected of machine learning models. The fact that LLMs can effectively generate sensible answers to a diverse range of queries suggests that they would be useful in customer support applications. While powerful, LLMs have been observed to be prone to hallucination which unfortunately makes their near term use in customer support applications challenging. To address this issue we present a system that allows us to use an LLM to augment our customer support advocates by re-framing the language modeling task as a discriminative classification task. In this framing, we seek to present the top-K best template responses for a customer support advocate to use when responding to a customer. We present the result of both offline and online experiments where we observed offline gains and statistically significant online lifts for our experimental system. Along the way, we present observed scaling curves for validation loss and top-K accuracy, resulted from model parameter ablation studies. We close by discussing the space of trade-offs with respect to model size, latency, and accuracy as well as and suggesting future applications to explore.

cs.CL↗

arXiv2023

Understanding convolution on graphs via energies

Graph Neural Networks (GNNs) typically operate by message-passing, where the state of a node is updated based on the information received from its neighbours. Most message-passing models act as graph convolutions, where features are mixed by a shared, linear transformation before being propagated over the edges. On node-classification tasks, graph convolutions have been shown to suffer from two limitations: poor performance on heterophilic graphs, and over-smoothing. It is common belief that both phenomena occur because such models behave as low-pass filters, meaning that the Dirichlet energy of the features decreases along the layers incurring a smoothing effect that ultimately makes features no longer distinguishable. In this work, we rigorously prove that simple graph-convolutional models can actually enhance high frequencies and even lead to an asymptotic behaviour we refer to as over-sharpening, opposite to over-smoothing. We do so by showing that linear graph convolutions with symmetric weights minimize a multi-particle energy that generalizes the Dirichlet energy; in this setting, the weight matrices induce edge-wise attraction (repulsion) through their positive (negative) eigenvalues, thereby controlling whether the features are being smoothed or sharpened. We also extend the analysis to non-linear GNNs, and demonstrate that some existing time-continuous GNNs are instead always dominated by the low frequencies. Finally, we validate our theoretical findings through ablations and real-world experiments.

cs.LG↗

arXiv2023

kNN-Embed: Locally Smoothed Embedding Mixtures For Multi-interest Candidate Retrieval

Candidate retrieval is the first stage in recommendation systems, where a light-weight system is used to retrieve potentially relevant items for an input user. These candidate items are then ranked and pruned in later stages of recommender systems using a more complex ranking model. As the top of the recommendation funnel, it is important to retrieve a high-recall candidate set to feed into downstream ranking models. A common approach is to leverage approximate nearest neighbor (ANN) search from a single dense query embedding; however, this approach this can yield a low-diversity result set with many near duplicates. As users often have multiple interests, candidate retrieval should ideally return a diverse set of candidates reflective of the user's multiple interests. To this end, we introduce kNN-Embed, a general approach to improving diversity in dense ANN-based retrieval. kNN-Embed represents each user as a smoothed mixture over learned item clusters that represent distinct "interests" of the user. By querying each of a user's mixture component in proportion to their mixture weights, we retrieve a high-diversity set of candidates reflecting elements from each of a user's interests. We experimentally compare kNN-Embed to standard ANN candidate retrieval, and show significant improvements in overall recall and improved diversity across three datasets. Accompanying this work, we open source a large Twitter follow-graph dataset (https://huggingface.co/datasets/Twitter/TwitterFollowGraph), to spur further research in graph-mining and representation learning for recommender systems.

cs.IR↗

arXiv2023

QDC: Quantum Diffusion Convolution Kernels on Graphs

Graph convolutional neural networks (GCNs) operate by aggregating messages over local neighborhoods given the prediction task under interest. Many GCNs can be understood as a form of generalized diffusion of input features on the graph, and significant work has been dedicated to improving predictive accuracy by altering the ways of message passing. In this work, we propose a new convolution kernel that effectively rewires the graph according to the occupation correlations of the vertices by trading on the generalized diffusion paradigm for the propagation of a quantum particle over the graph. We term this new convolution kernel the Quantum Diffusion Convolution (QDC) operator. In addition, we introduce a multiscale variant that combines messages from the QDC operator and the traditional combinatorial Laplacian. To understand our method, we explore the spectral dependence of homophily and the importance of quantum dynamics in the construction of a bandpass filter. Through these studies, as well as experiments on a range of datasets, we observe that QDC improves predictive performance on the widely used benchmark datasets when compared to similar methods.

cs.LG↗

arXiv2023

Graph Neural Networks for Link Prediction with Subgraph Sketching

Many Graph Neural Networks (GNNs) perform poorly compared to simple heuristics on Link Prediction (LP) tasks. This is due to limitations in expressive power such as the inability to count triangles (the backbone of most LP heuristics) and because they can not distinguish automorphic nodes (those having identical structural roles). Both expressiveness issues can be alleviated by learning link (rather than node) representations and incorporating structural features such as triangle counts. Since explicit link representations are often prohibitively expensive, recent works resorted to subgraph-based methods, which have achieved state-of-the-art performance for LP, but suffer from poor efficiency due to high levels of redundancy between subgraphs. We analyze the components of subgraph GNN (SGNN) methods for link prediction. Based on our analysis, we propose a novel full-graph GNN called ELPH (Efficient Link Prediction with Hashing) that passes subgraph sketches as messages to approximate the key components of SGNNs without explicit subgraph construction. ELPH is provably more expressive than Message Passing GNNs (MPNNs). It outperforms existing SGNN models on many standard LP benchmarks while being orders of magnitude faster. However, it shares the common GNN limitation that it is only efficient when the dataset fits in GPU memory. Accordingly, we develop a highly scalable model, called BUDDY, which uses feature precomputation to circumvent this limitation without sacrificing predictive performance. Our experiments show that BUDDY also outperforms SGNNs on standard LP benchmarks while being highly scalable and faster than ELPH.

cs.LG↗

arXiv2023

TwERC: High Performance Ensembled Candidate Generation for Ads Recommendation at Twitter

Recommendation systems are a core feature of social media companies with their uses including recommending organic and promoted contents. Many modern recommendation systems are split into multiple stages - candidate generation and heavy ranking - to balance computational cost against recommendation quality. We focus on the candidate generation phase of a large-scale ads recommendation problem in this paper, and present a machine learning first heterogeneous re-architecture of this stage which we term TwERC. We show that a system that combines a real-time light ranker with sourcing strategies capable of capturing additional information provides validated gains. We present two strategies. The first strategy uses a notion of similarity in the interaction graph, while the second strategy caches previous scores from the ranking stage. The graph based strategy achieves a 4.08% revenue gain and the rankscore based strategy achieves a 1.38% gain. These two strategies have biases that complement both the light ranker and one another. Finally, we describe a set of metrics that we believe are valuable as a means of understanding the complex product trade offs inherent in industrial candidate generation systems.

cs.IR↗

arXiv2023

Causally-guided Regularization of Graph Attention Improves Generalizability

Graph attention networks estimate the relational importance of node neighbors to aggregate relevant information over local neighborhoods for a prediction task. However, the inferred attentions are vulnerable to spurious correlations and connectivity in the training data, hampering the generalizability of the model. We introduce CAR, a general-purpose regularization framework for graph attention networks. Embodying a causal inference approach, CAR aligns the attention mechanism with the causal effects of active interventions on graph connectivity in a scalable manner. CAR is compatible with a variety of graph attention architectures, and we show that it systematically improves generalizability on various node classification tasks. Our ablation studies indicate that CAR hones in on the aspects of graph structure most pertinent to the prediction (e.g., homophily), and does so more effectively than alternative approaches. Finally, we also show that CAR enhances interpretability of attention weights by accentuating node-neighbor relations that point to causal hypotheses. For social media network-sized graphs, a CAR-guided graph rewiring approach could allow us to combine the scalability of graph convolutional methods with the higher performance of graph attention.

cs.LG↗

arXiv2022

TwHIN: Embedding the Twitter Heterogeneous Information Network for Personalized Recommendation

Social networks, such as Twitter, form a heterogeneous information network (HIN) where nodes represent domain entities (e.g., user, content, advertiser, etc.) and edges represent one of many entity interactions (e.g, a user re-sharing content or "following" another). Interactions from multiple relation types can encode valuable information about social network entities not fully captured by a single relation; for instance, a user's preference for accounts to follow may depend on both user-content engagement interactions and the other users they follow. In this work, we investigate knowledge-graph embeddings for entities in the Twitter HIN (TwHIN); we show that these pretrained representations yield significant offline and online improvement for a diverse range of downstream recommendation and classification tasks: personalized ads rankings, account follow-recommendation, offensive content detection, and search ranking. We discuss design choices and practical challenges of deploying industry-scale HIN embeddings, including compressing them to reduce end-to-end model latency and handling parameter drift across versions.

cs.SI↗

arXiv2016

Calibration of the Many-Body Dispersion Range-Separation Parameter

Recent work has shown that a fully many-body treatment of noncovalent interactions, such as that given by the method of many-body dispersion (MBD), is vital to accurately modeling the structure and energetics of many molecular systems with density functional theory (DFT). To avoid double counting the correlation contributions of DFT and the MBD correction, a single-parameter range-separation scheme is typically employed. Coupling the MBD correction to a given exchange-correlation functional therefore requires calibrating the range-separation parameter. We perform this calibration for 24 popular DFT functionals by optimizing against the S66x8 benchmark set. Additionally, we report a linear equation that predicts near optimal range-separation parameters, dependent only on the class of the exchange functional and the value of the gradient enhancement factor. When a calibrated MBD correction is employed, most of the exchange-correlation functionals considered are capable of achieving agreement with CCSD(T)/CBS interaction energies in the S66x8 set to better than 1~kcal/mol mean absolute error.

physics.chem-ph↗

arXiv2015

Analytical nuclear gradients for the range-separated many-body dispersion model of noncovalent interactions

Accurate treatment of the long-range electron correlation energy, including van der Waals (vdW) or dispersion interactions, is essential for describing the structure, dynamics, and function of a wide variety of systems. Among the most accurate models for including dispersion into density functional theory (DFT) is the range-separated many-body dispersion (MBD) method [A. Ambrossetti et al., J. Chem. Phys. 140, 18A508 (2014)], in which the correlation energy is modeled at short-range by a semi-local density functional and at long-range by a model system of coupled quantum harmonic oscillators. In this work, we develop analytical gradients of the MBD energy with respect to nuclear coordinates, including all implicit coordinate dependencies arising from the partitioning of the charge density into Hirshfeld effective volumes. To demonstrate the efficiency and accuracy of these MBD gradients for geometry optimizations of systems with intermolecular and intramolecular interactions, we optimized conformers of the benzene dimer and isolated small peptides with aromatic side-chains. We find excellent agreement with the wavefunction theory reference geometries of these systems (at a fraction of the computational cost) and find that MBD consistently outperforms the popular TS and D3(BJ) dispersion corrections. To demonstrate the performance of the MBD model on a larger system with supramolecular interactions, we optimized the C$_{60}$@C$_{60}$H$_{28}$ buckyball catcher host-guest complex. Finally, we find that neglecting the implicit nuclear coordinate dependence arising from the charge density partitioning, as has been done in prior numerical treatments, leads to an unacceptable error in the MBD forces, with relative errors of ~20% (on average) that can extend well beyond 100%.

physics.comp-ph↗

arXiv2015

Benchmarking Compressed Sensing, Super-Resolution, and Filter Diagonalization

Signal processing techniques have been developed that use different strategies to bypass the Nyquist sampling theorem in order to recover more information than a traditional discrete Fourier transform. Here we examine three such methods: filter diagonalization, compressed sensing, and super-resolution. We apply them to a broad range of signal forms commonly found in science and engineering in order to discover when and how each method can be used most profitably. We find that filter diagonalization provides the best results for Lorentzian signals, while compressed sensing and super-resolution perform better for arbitrary signals.

physics.data-an↗

arXiv2013

More accurate and efficient bath spectral densities from super-resolution

Quantum transport and other phenomena are typically modeled by coupling the system of interest to an environment, or bath, held at thermal equilibrium. Realistic bath models are at least as challenging to construct as models for the quantum systems themselves, since they must incorporate many degrees of freedom that interact with the system on a wide range of timescales. Owing to computational limitations, the environment is often modeled with simple functional forms, with a few parameters fit to experiment to yield semi-quantitative results. Growing computational resources have enabled the construction of more realistic bath models from molecular dynamics (MD) simulations. In this paper, we develop a numerical technique to construct these atomistic bath models with better accuracy and decreased cost. We apply a novel signal processing technique, known as super-resolution, combined with a dictionary of physically-motivated bath modes to derive spectral densities from MD simulations. Our approach reduces the required simulation time and provides a more accurate spectral density than can be obtained via standard Fourier transform methods. Moreover, the spectral density is provided as a convenient closed-form expression which yields an analytic time-dependent bath kernel. Exciton dynamics of the Fenna-Matthews-Olsen light-harvesting complex are simulated with a second order time-convolutionless master equation, and spectral densities constructed via super-resolution are shown to reproduce the dynamics using only a quarter of the amount of MD data.

quant-ph↗

arXiv2012

A correlated-polaron electronic propagator: open electronic dynamics beyond the Born-Oppenheimer approximation

In this work we develop a theory of correlated many-electron dynamics dressed by the presence of a finite-temperature harmonic bath. The theory is based on the ab-initio Hamiltonian, and thus well-defined apart from any phenomenological choice of collective basis states or electronic coupling model. The equation-of-motion includes some bath effects non-perturbatively, and can be used to simulate line- shapes beyond the Markovian approximation and open electronic dynamics which are subjects of renewed recent interest. Energy conversion and transport depend critically on the ratio of electron-electron coupling to bath-electron coupling, which is a fitted parameter if a phenomenological basis of many-electron states is used to develop an electronic equation of motion. Since the present work doesn't appeal to any such basis, it avoids this ambiguity. The new theory produces a level of detail beyond the adiabatic Born-Oppenheimer states, but with cost scaling like the Born-Oppenheimer approach. While developing this model we have also applied the time-convolutionless perturbation theory to correlated molecular excitations for the first time. Resonant response properties are given by the formalism without phenomenological parameters. Example propagations with a developmental code are given demonstrating the treatment of electron-correlation in absorption spectra, vibronic structure, and decay in an open system.

quant-ph↗

arXiv2011

Supersymmetric Quantum Mechanics For Atomic Electronic Systems

We employ our new approach to non-relativistic supersymmetric quantum mechanics (SUSY-QM), (J. Phys. Chem. A 114, 8202(2010)) for any number of dimensions and distinguishable particles, to treat the hydrogen atom in full three-dimensional detail. In contrast to the standard one-dimensional radial equation SUSY-QM treatment of the hydrogen atom, where the superpotential is a scalar, in a full three-dimensional treatment, it is a vector which applies regardless of the electron angular momentum. The original scalar Schrodinger Hamiltonian operator is factored into vector "charge" operators: Q and Q^{dagger}. Using these operators, the first sector Hamiltonian is written as H1 = Q^{\dagger} * Q. The second sector Hamiltonian is a tensor given by H2 = Q Q^{\dagger} and is isospectral with H1. The second sector ground state, psi^{(2)}_0, can be used to obtain the excited state wave functions of the first sector by application of the adjoint charge operator. Alternatively, Q applied to analytical, sector one excited states yield analytical results for the sector two vector eigenstates. Several of these are plotted for illustration. We then adapt the aufbau principle to show this approach can be applied to treat the helium atom.

quant-ph↗