SearcharxivSearch

arXiv subjects

Tomasz Nowicki

Publications and source records attributed to Tomasz Nowicki.

At least 19 recordsLinked to original sources

Counting Triangles of Graphs via Randomized Trace Estimation with Incomplete Matrix-Vector Products

Counting triangles in graphs is a fundamental operation in network analysis, underpinning metrics such as clustering coefficients and serving as a signal for community detection, link prediction, and anomaly detection. The standard approach computes the trace of the cube of the adjacency matrix, but explicitly forming $\mathbf{A}^3$ is infeasible for large graphs. Hutchinson randomized trace estimator offers an efficient alternative by approximating the trace through stochastic averaging of quadratic forms, requiring only matrix vector products with $\mathbf{A}$. However, in distributed and heterogeneous computing environments, observing all entries of these products can be costly due to communication overhead and straggler effects. To address this, we propose a new variant of Hutchinson estimator that operates under partial observation constraints, where both the number and identities of observed entries are random. We provide theoretical guarantees on unbiasedness, variance bounds, and sample complexity, and demonstrate through experiments on synthetic and real world graphs that our method achieves accurate triangle count estimates while reducing synchronization costs. This work highlights the adaptability of randomized algorithms to modern computational architectures and opens avenues for efficient motif counting in large scale network analytics.

math.NA

Analysis of Power Iteration Algorithm with Partially Observed Matrix-vector Products

We consider the problem of computing the dominant eigenvector of a symmetric matrix via the power iteration algorithm subject to constraints in the computation of matrix-vector pr ucts. In particular, we focus on scenarios where the entries of matrix-vector products with the input matrix are only partially observed. Such constraints frequently arise on cloud architectures implemented via the controller-worker model where the matrix-vector products are distributed across workers on remote servers. Instead of a prolonged delay incurred by waiting for the slowest workers to return their output to the controller, a phenomenon known as straggling, a set of pre-determined values can replace the values of the delayed workers and allow the power iteration to proceed to the next iteration. In this paper, we develop two algorithms whose expected approximation converges to the true dominant eigenvector. The first algorithm relies on a probabilistic switch between two different approaches to set the omitted entries: either set them to zero or to their previous recorded value. The second algorithm relies on averaging previously generated partial power iteration approximations obtained by ignoring a set of columns of the iteration matrix. several theoretical details are discussed while numerical experiments verify the effectiveness of the two proposed schemes and demonstrate their comparative performance advantage over current state-of-the-art.

math.NA

On Hamiltonian Monte Carlo for Gaussian Random Variables with Random Hamiltonians

We study a family of (multivariate-)Gaussian Hamiltonian Monte Carlo (GHMC) operators and prove that the family of Gaussian distributions and their mixtures are invariant under such operators. Furthermore, each such operator is a contraction on the space of parameters and an explicit formulae are derived. These results then enable us to analyze the dynamics and convergences of independent and identically distributed random sequences of such operators.

math.PR

Optimality and NP-Hardness of Transformers in Learning Markovian Dynamical Functions

Transformer architectures can solve unseen tasks based on input-output pairs in a given prompt due to in-context learning (ICL). Existing theoretical studies on ICL have mainly focused on linear regression tasks, often with i.i.d. inputs. To understand how transformers express ICL when modeling dynamics-driven functions, we investigate Markovian function learning through a structured ICL setup, where we characterize the loss landscape to reveal underlying optimization behaviors. Specifically, we (1) provide the closed-form expression of the global minimizer (in an enlarged parameter space) for a single-layer linear self-attention (LSA) model; (2) prove that recovering transformer parameters that realize the optimal solution is NP-hard in general, revealing a fundamental limitation of one-layer LSA in representing structured dynamical functions; and (3) supply a novel interpretation of a multilayer LSA as performing preconditioned gradient descent to optimize multiple objectives beyond the square loss. These theoretical results are numerically validated using simplified transformers.

cs.LG

Fast Linear Solvers via AI-Tuned Markov Chain Monte Carlo-based Matrix Inversion

Large, sparse linear systems are pervasive in modern science and engineering, and Krylov subspace solvers are an established means of solving them. Yet convergence can be slow for ill-conditioned matrices, so practical deployments usually require preconditioners. Markov chain Monte Carlo (MCMC)-based matrix inversion can generate such preconditioners and accelerate Krylov iterations, but its effectiveness depends on parameters whose optima vary across matrices; manual or grid search is costly. We present an AI-driven framework recommending MCMC parameters for a given linear system. A graph neural surrogate predicts preconditioning speed from $A$ and MCMC parameters. A Bayesian acquisition function then chooses the parameter sets most likely to minimise iterations. On a previously unseen ill-conditioned system, the framework achieves better preconditioning with 50\% of the search budget of conventional methods, yielding about a 10\% reduction in iterations to convergence. These results suggest a route for incorporating MCMC-based preconditioners into large-scale systems.

cs.LG

Regenerative Ulam-von Neumann Algorithm: An Innovative Markov chain Monte Carlo Method for Matrix Inversion

This paper presents a regenerative variant of the classical Ulam-von Neumann Markov chain Monte Carlo algorithm for the approximation of the matrix inverse. The algorithm presented in this paper, termed regenerative Ulam-von Neumann algorithm, utilizes the regenerative structure of classical, non-truncated Neumann series defined by a non-singular matrix and produces an estimator of the matrix inverse via ratios of unbiased estimators of the regenerative quantities. The accuracy of the proposed algorithm depends on a single parameter that controls the total number of simulated Markov transitions, thus avoiding the challenge of balancing between the total number of Markov chain replications and their length as in the classical Ulam-von Neumann algorithm. To efficiently utilize Markov chain transition samples in the calculation of the regenerative variables, the proposed algorithm automatically quantifies the contribution of each Markov transition to all regenerative quantities by a carefully designed updating scheme that utilized three separate matrices containing the current weights, total weights, and regenerative cycle count, respectively. A probabilistic analysis of the performance of the algorithm, including the variance of the estimator, is provided. Finally, numerical experiments verify the effectiveness of the proposed scheme.

math.NA

Dynamical Behaviors of the Gradient Flows for In-Context Learning

We derive the system of differential equations for the gradient flow characterizing the training process of linear in-context learning in full generality. Next, we explore the geometric structure of the gradient flows in two instances, including identifying its invariants, optimum, and saddle points. This understanding allows us to quantify the behavior of the two gradient flows under the full generality of parameters and data.

math.DS

On The Variance of Schatten $p$-Norm Estimation with Gaussian Sketching Matrices

Monte Carlo matrix trace estimation is a popular randomized technique to estimate the trace of implicitly-defined matrices via averaging quadratic forms across several observations of a random vector. The most common approach to analyze the quality of such estimators is to consider the variance over the total number of observations. In this paper we present a procedure to compute the variance of the estimator proposed by Kong and Valiant [Ann. Statist. 45 (5), pp. 2218 - 2247] for the case of Gaussian random vectors and provide a sharper bound than previously available.

math.ST

On the iterations of some random functions with Lipschitz number one

For the iterations of $x\mapsto |x-θ|$ random functions with Lipschitz number one, we represent the dynamics as a Markov chain and prove its convergence under mild conditions. We also demonstrate that the Wasserstein metric of any two measures will not increase after the corresponding induced iterations for measures and identify conditions under which a polynomial convergence rate can be achieved in this metric. We also consider an associated nonlinear operator on the space of probability measures and identify its fixed points through an detailed analysis of their characteristic functions.

math.PR

On Convergence of the Alternating Directions SGHMC Algorithm

We study convergence rates of Hamiltonian Monte Carlo (HMC) algorithms with leapfrog integration under mild conditions on stochastic gradient oracle for the target distribution (SGHMC). Our method extends standard HMC by allowing the use of general auxiliary distributions, which is achieved by a novel procedure of Alternating Directions. The convergence analysis is based on the investigations of the Dirichlet forms associated with the underlying Markov chain driving the algorithms. For this purpose, we provide a detailed analysis on the error of the leapfrog integrator for Hamiltonian motions with both the kinetic and potential energy functions in general form. We characterize the explicit dependence of the convergence rates on key parameters such as the problem dimension, functional properties of both the target and auxiliary distributions, and the quality of the oracle.

math.ST

Multi-Function Multi-Way Analog Technology for Sustainable Machine Intelligence Computation

Numerical computation is essential to many areas of artificial intelligence (AI), whose computing demands continue to grow dramatically, yet their continued scaling is jeopardized by the slowdown in Moore's law. Multi-function multi-way analog (MFMWA) technology, a computing architecture comprising arrays of memristors supporting in-memory computation of matrix operations, can offer tremendous improvements in computation and energy, but at the expense of inherent unpredictability and noise. We devise novel randomized algorithms tailored to MFMWA architectures that mitigate the detrimental impact of imperfect analog computations while realizing their potential benefits across various areas of AI, such as applications in computer vision. Through analysis, measurements from analog devices, and simulations of larger systems, we demonstrate orders of magnitude reduction in both computation and energy with accuracy similar to digital computers.

math.NA

On Representations of Mean-Field Variational Inference

The mean field variational inference (MFVI) formulation restricts the general Bayesian inference problem to the subspace of product measures. We present a framework to analyze MFVI algorithms, which is inspired by a similar development for general variational Bayesian formulations. Our approach enables the MFVI problem to be represented in three different manners: a gradient flow on Wasserstein space, a system of Fokker-Planck-like equations and a diffusion process. Rigorous guarantees are established to show that a time-discretized implementation of the coordinate ascent variational inference algorithm in the product Wasserstein space of measures yields a gradient flow in the limit. A similar result is obtained for their associated densities, with the limit being given by a quasi-linear partial differential equation. A popular class of practical algorithms falls in this framework, which provides tools to establish convergence. We hope this framework could be used to guarantee convergence of algorithms in a variety of approaches, old and new, to solve variational inference problems.

stat.ML

On $L^q$ Convergence of the Hamiltonian Monte Carlo

We establish $L_q$ convergence for Hamiltonian Monte Carlo algorithms. More specifically, under mild conditions for the associated Hamiltonian motion, we show that the outputs of the algorithms converge (strongly for $2\le q<\infty$ and weakly for $1<q<2$) to the desired target distribution.

math.CA

Neural Network Training with Asymmetric Crosspoint Elements

Analog crossbar arrays comprising programmable nonvolatile resistors are under intense investigation for acceleration of deep neural network training. However, the ubiquitous asymmetric conductance modulation of practical resistive devices critically degrades the classification performance of networks trained with conventional algorithms. Here, we describe and experimentally demonstrate an alternative fully-parallel training algorithm: Stochastic Hamiltonian Descent. Instead of conventionally tuning weights in the direction of the error function gradient, this method programs the network parameters to successfully minimize the total energy (Hamiltonian) of the system that incorporates the effects of device asymmetry. We provide critical intuition on why device asymmetry is fundamentally incompatible with conventional training algorithms and how the new approach exploits it as a useful feature instead. Our technique enables immediate realization of analog deep learning accelerators based on readily available device technologies.

cs.LG

Hamiltonian Monte Carlo with Asymmetrical Momentum Distributions

Existing rigorous convergence guarantees for the Hamiltonian Monte Carlo (HMC) algorithm use Gaussian auxiliary momentum variables, which are crucially symmetrically distributed. We present a novel convergence analysis for HMC utilizing new dynamical and probabilistic arguments. The convergence is rigorously established under significantly weaker conditions, which among others allow for general auxiliary distributions. In our framework, we show that plain HMC with asymmetrical momentum distributions breaks a key self-adjointness requirement. We propose a modified version of HMC, that we call the Alternating Direction HMC (AD-HMC), which overcomes this difficulty. Sufficient conditions are established under which AD-HMC exhibits geometric convergence in Wasserstein distance. The geometric convergence analysis is extended to when the Hamiltonian motion is approximated by the leapfrog symplectic integrator, where an additional Metropolis-Hastings rejection step is required. Numerical experiments suggest that AD-HMC can generalize a popular dynamic auxiliary scheme to show improved performance over HMC with Gaussian auxiliaries.

stat.ML

Solving sparse linear systems with approximate inverse preconditioners on analog devices

Sparse linear system solvers are computationally expensive kernels that lie at the heart of numerous applications. This paper proposes a flexible preconditioning framework to substantially reduce the time and energy requirements of this task by utilizing a hybrid architecture that combines conventional digital microprocessors with analog crossbar array accelerators. Our analysis and experiments with a simulator for analog hardware demonstrate that an order of magnitude speedup is readily attainable without much impact on convergence, despite the noise in analog computations.

cs.ET

HMC, an Algorithms in Data Mining, the Functional Analysis approach

The main purpose of this paper is to facilitate the communication between the Analytic, Probabilistic and Algorithmic communities. We present a proof of convergence of the Hamiltonian (Hybrid) Monte Carlo algorithm from the point of view of the Dynamical Systems, where the evolving objects are densities of probability distributions and the tool are derived from the Functional Analysis.

stat.CO

Approximation Algorithms for the Joint Replenishment Problem with Deadlines

The Joint Replenishment Problem (JRP) is a fundamental optimization problem in supply-chain management, concerned with optimizing the flow of goods from a supplier to retailers. Over time, in response to demands at the retailers, the supplier ships orders, via a warehouse, to the retailers. The objective is to schedule these orders to minimize the sum of ordering costs and retailers' waiting costs. We study the approximability of JRP-D, the version of JRP with deadlines, where instead of waiting costs the retailers impose strict deadlines. We study the integrality gap of the standard linear-program (LP) relaxation, giving a lower bound of 1.207, a stronger, computer-assisted lower bound of 1.245, as well as an upper bound and approximation ratio of 1.574. The best previous upper bound and approximation ratio was 1.667; no lower bound was previously published. For the special case when all demand periods are of equal length we give an upper bound of 1.5, a lower bound of 1.2, and show APX-hardness.

cs.DS