Searcharxiv⌕ Search

arXiv subjects

Johan Jonasson

Publications and source records attributed to Johan Jonasson.

At least 19 recordsLinked to original sources
arXiv2025

Iterative Data Curation with Theoretical Guarantees

In recent years, more and more large data sets have become available. Data accuracy, the absence of verifiable errors in data, is crucial for these large materials to enable high-quality research, downstream applications, and model training. This results in the problem of how to curate or improve data accuracy in such large and growing data, especially when the data is too large for manual curation to be feasible. This paper presents a unified procedure for iterative and continuous improvement of data sets. We provide theoretical guarantees that data accuracy tests speed up error reduction and, most importantly, that the proposed approach will, asymptotically, eliminate all errors in data with probability one. We corroborate the theoretical results with simulations and a real-world use case.

stat.ME↗
arXiv2025

Posterior Sampling of Probabilistic Word Embeddings

Quantifying uncertainty in word embeddings is crucial for reliable inference from textual data. However, existing Bayesian methods such as Hamiltonian Monte Carlo (HMC) and mean-field variational inference (MFVI) are either computationally infeasible for large data or rely on restrictive assumptions. We propose a scalable Gibbs sampler using Polya-Gamma augmentation as well as Laplace approximation and compare them with MFVI and HMC for word embeddings. In addition, we address non-identifiability in word embeddings. Our Gibbs sampler and HMC correctly estimate uncertainties, while MFVI does not, and Laplace approximation only does so on large sample sizes, as expected. Applying the Gibbs sampler to the US Congress and the Movielens datasets, we demonstrate the feasibility on larger real data. Finally, as a result of having draws from the full posterior, we show that the posterior mean of word embeddings improves over maximum a posteriori (MAP) estimates in terms of hold-out likelihood, especially for smaller sampling sizes, further strengthening the need for posterior sampling of word embeddings.

cs.LG↗
arXiv2025

Variational Quantum Optimization with Continuous Bandits

We introduce a novel approach to variational Quantum algorithms (VQA) via continuous bandits. VQA are a class of hybrid Quantum-classical algorithms where the parameters of Quantum circuits are optimized by classical algorithms. Previous work has used zero and first order gradient based methods, however such algorithms suffer from the barren plateau (BP) problem where gradients and loss differences are exponentially small. We introduce an approach using bandits methods which combine global exploration with local exploitation. We show how VQA can be formulated as a best arm identification problem in a continuous space of arms with Lipschitz smoothness. While regret minimization has been addressed in this setting, existing methods for pure exploration only cover discrete spaces. We give the first results for pure exploration in a continuous setting and derive a fixed-confidence, information-theoretic, instance specific lower bound. Under certain assumptions on the expected payoff, we derive a simple algorithm, which is near-optimal with respect to our lower bound. Finally, we apply our continuous bandit algorithm to two VQA schemes: a PQC and a QAOA quantum circuit, showing that we significantly outperform the previously known state of the art methods (which used gradient based methods).

cs.LG↗
arXiv2023

Noise Sensitivity and Stability of Deep Neural Networks for Binary Classification

A first step is taken towards understanding often observed non-robustness phenomena of deep neural net (DNN) classifiers. This is done from the perspective of Boolean functions by asking if certain sequences of Boolean functions represented by common DNN models are noise sensitive or noise stable, concepts defined in the Boolean function literature. Due to the natural randomness in DNN models, these concepts are extended to annealed and quenched versions. Here we sort out the relation between these definitions and investigate the properties of two standard DNN architectures, the fully connected and convolutional models, when initiated with Gaussian weights.

stat.ML↗
arXiv2023

Optimal subsampling designs

Subsampling is commonly used to overcome computational and economical bottlenecks in the analysis of finite populations and massive datasets. Existing methods are often limited in scope and use optimality criteria (e.g., A-optimality) with well-known deficiencies, such as lack of invariance to the measurement-scale of the data and parameterisation of the model. A unified theory of optimal subsampling design is still lacking. We present a theory of optimal design for general data subsampling problems, including finite population inference, parametric density estimation, and regression modelling. Our theory encompasses and generalises most existing methods in the field of optimal subdata selection based on unequal probability sampling and inverse probability weighting. We derive optimality conditions for a general class of optimality criteria, and present corresponding algorithms for finding optimal sampling schemes under Poisson and multinomial sampling designs. We present a novel class of transformation- and parameterisation-invariant linear optimality criteria which enjoy the best of two worlds: the computational tractability of A-optimality and invariance properties similar to D-optimality. The methodology is illustrated on an application in the traffic safety domain. In our experiments, the proposed invariant linear optimality criteria achieve 92-99% D-efficiency with 90-95% lower computational demand. In contrast, the A-optimality criterion has only 46% and 60% D-efficiency on two of the examples.

math.ST↗
arXiv2022

Robust Neural Network Classification via Double Regularization

The presence of mislabeled observations in data is a notoriously challenging problem in statistics and machine learning, associated with poor generalization properties for both traditional classifiers and, perhaps even more so, flexible classifiers like neural networks. Here we propose a novel double regularization of the neural network training loss that combines a penalty on the complexity of the classification model and an optimal reweighting of training observations. The combined penalties result in improved generalization properties and strong robustness against overfitting in different settings of mislabeled training data and also against variation in initial parameter values when training. We provide a theoretical justification for our proposed method derived for a simple case of logistic regression. We demonstrate the double regularization model, here denoted by DRFit, for neural net classification of (i) MNIST and (ii) CIFAR-10, in both cases with simulated mislabeling. We also illustrate that DRFit identifies mislabeled data points with very good precision. This provides strong support for DRFit as a practical of-the-shelf classifier, since, without any sacrifice in performance, we get a classifier that simultaneously reduces overfitting against mislabeling and gives an accurate measure of the trustworthiness of the labels.

stat.ML↗
arXiv2022

Elastic Gradient Descent, an Iterative Optimization Method Approximating the Solution Paths of the Elastic Net

The elastic net combines lasso and ridge regression to fuse the sparsity property of lasso with the grouping property of ridge regression. The connections between ridge regression and gradient descent and between lasso and forward stagewise regression have previously been shown. Similar to how the elastic net generalizes lasso and ridge regression, we introduce elastic gradient descent, a generalization of gradient descent and forward stagewise regression. We theoretically analyze elastic gradient descent and compare it to the elastic net and forward stagewise regression. Parts of the analysis are based on elastic gradient flow, a piecewise analytical construction, obtained for elastic gradient descent with infinitesimal step size. We also compare elastic gradient descent to the elastic net on real and simulated data and show that it provides similar solution paths, but is several orders of magnitude faster. Compared to forward stagewise regression, elastic gradient descent selects a model that, although still sparse, provides considerably lower prediction and estimation errors.

stat.ME↗
arXiv2021

Rapid mixing in unimodal landscapes and efficient simulatedannealing for multimodal distributions

We consider nearest neighbor weighted random walks on the $d$-dimensional box $[n]^d$ that are governed by some function $g:[0,1] \ra [0,\iy)$, by which we mean that standing at $x$, a neighbor $y$ of $x$ is picked at random and the walk then moves there with probability $(1/2)g(n^{-1}y)/(g(n^{-1}y)+g(n^{-1}x))$. We do this for $g$ of the form $f^{m_n}$ for some function $f$ which assumed to be analytically well-behaved and where $m_n \ra \iy$ as $n \ra \iy$. This class of walks covers an abundance of interesting special cases, e.g., the mean-field Potts model, posterior collapsed Gibbs sampling for Latent Dirichlet allocation and certain Bayesian posteriors for models in nuclear physics. The following are among the results of this paper: \begin{itemize} \item If $f$ is unimodal with negative definite Hessian at its global maximum, then the mixing time of the random walk is $O(n\log n)$. \item If $f$ is multimodal, then the mixing time is exponential in $n$, but we show that there is a simulated annealing scheme governed by $f^K$ for an increasing sequence of $K$ that mixes in time $O(n^2)$. Using a varying step size that decreases with $K$, this can be taken down to $O(n\log n)$. \item If the process is studied on a general graph rather than the $d$-dimensional box, a simulated annealing scheme expressed in terms of conductances of the underlying network, works similarly. \end{itemize} Several examples are given, including the ones mentioned above.

math.PR↗
arXiv2020

Leave-One-Out Cross-Validation for Bayesian Model Comparison in Large Data

Recently, new methods for model assessment, based on subsampling and posterior approximations, have been proposed for scaling leave-one-out cross-validation (LOO) to large datasets. Although these methods work well for estimating predictive performance for individual models, they are less powerful in model comparison. We propose an efficient method for estimating differences in predictive performance by combining fast approximate LOO surrogates with exact LOO subsampling using the difference estimator and supply proofs with regards to scaling characteristics. The resulting approach can be orders of magnitude more efficient than previous approaches, as well as being better suited to model comparison.

stat.ME↗
arXiv2019

Bayesian leave-one-out cross-validation for large data

Model inference, such as model comparison, model checking, and model selection, is an important part of model development. Leave-one-out cross-validation (LOO) is a general approach for assessing the generalizability of a model, but unfortunately, LOO does not scale well to large datasets. We propose a combination of using approximate inference techniques and probability-proportional-to-size-sampling (PPS) for fast LOO model evaluation for large datasets. We provide both theoretical and empirical results showing good properties for large data.

stat.ML↗
arXiv2017

Fast mixing for Latent Dirichlet allocation

Markov chain Monte Carlo (MCMC) algorithms are ubiquitous in probability theory in general and in machine learning in particular. A Markov chain is devised so that its stationary distribution is some probability distribution of interest. Then one samples from the given distribution by running the Markov chain for a "long time" until it appears to be stationary and then collects the sample. However these chains are often very complex and there are no theoretical guarantees that stationarity is actually reached. In this paper we study the Gibbs sampler of the posterior distribution of a very simple case of Latent Dirichlet Allocation, the arguably most well known Bayesian unsupervised learning model for text generation and text classification. It is shown that when the corpus consists of two long documents of equal length $m$ and the vocabulary consists of only two different words, the mixing time is at most of order $m^2\log m$ (which corresponds to $m\log m$ rounds over the corpus). It will be apparent from our analysis that it seems very likely that the mixing time is not much worse in the more relevant case when the number of documents and the size of the vocabulary are also large as long as each word is represented a large number in each document, even though the computations involved may be intractable.

cs.LG↗
arXiv2015

The spectrum and convergence rates of exclusion and interchange processes on the complete graph

We give a short and completely elementary method to find the full spectrum of the exclusion process and a nicely limited superset of the spectrum of the interchange process (a.k.a.\ random transpositions) on the complete graph. In the case of the exclusion process, this gives a simple closed form expression for all the eigenvalues and their multiplicities. This result is then used to give an exact expression for the distance in $ L^2 $ from stationarity at any time and upper and lower bounds on the convergence rate for the exclusion process. In the case of the interchange process, upper and lower bounds are similarly found. Our results strengthen or reprove all known results of the mixing time for the two processes in a very simple way.

math.PR↗
arXiv2015

Card-cyclic-to-random shuffling with relabeling

The card-cyclic-to-random shuffle is the card shuffle where the $n$ cards are labeled $1,\ldots,n$ according to their starting positions. Then the cards are mixed by first picking card $1$ from the deck and reinserting it at a uniformly random position, then repeating for card $2$, then for card $3$ and so on until all cards have been reinserted in this way. Then the procedure starts over again, by first picking the card with label $1$ and reinserting, and so on. Morris, Ning and Peres \cite{MNP} recently showed that the order of the number of shuffles needed to mix the deck in this way is $n\log n$. In the present paper, we consider a variant of this shuffle with relabeling, i.e.\ a shuffle that differs from the above in that after one round, i.e.\ after all cards have been reinserted once, we relabel the cards according to the positions in the deck that they now have. The relabeling is then repeated after each round of shuffling. It is shown that even in this case, the correct order of mixing is $n\log n$.

math.PR↗
arXiv2015

Stationary random graphs on $\mathbb{Z}$ with prescribed iid degrees and finite mean connections

Let $F$ be a probability distribution with support on the non-negative integers. A model is proposed for generating stationary simple graphs on $\mathbb{Z}$ with degree distribution $F$ and it is shown for this model that the expected total length of all edges at a given vertex is finite if $F$ has finite second moment. It is not hard to see that any stationary model for generating simple graphs on $\mathbb{Z}$ will give infinite mean for the total edge length per vertex if $F$ does not have finite second moment. Hence, finite second moment of $F$ is a necessary and sufficient condition for the existence of a model with finite mean total edge length.

math.PR↗
arXiv2012

Mixing times for the interchange process

Consider the interchange process on a connected graph $G=(V,E)$ on $n$ vertices. I.e.\ shuffle a deck of cards by first placing one card at each vertex of $G$ in a fixed order and then at each tick of the clock, picking an edge uniformly at random and switching the two cards at the end vertices of the edge with probability 1/2. Well known special cases are the random transpositions shuffle, where $G$ is the complete graph, and the transposing neighbors shuffle, where $G$ is the $n$-path. Other cases that have been studied are the $d$-dimensional grid, the hypercube, lollipop graphs and Erd\H os-Rényi random graphs above the threshold for connectedness. In this paper the problem is studied for general $G$. Special attention is focused on trees, random trees and the giant component of critical and supercritical $G(N,p)$ random graphs. Upper and lower bounds on the mixing time are given. In many of the cases, we establish the exact order of the mixing time. We also mention the cases when $G$ is the hypercube and when $G$ is a bounded-degree expander, giving upper and lower bounds on the mixing time.

math.PR↗
arXiv2008

Visibility to infinity in the hyperbolic plane, despite obstacles

Suppose that $Z$ is a random closed subset of the hyperbolic plane $\H^2$, whose law is invariant under isometries of $\H^2$. We prove that if the probability that $Z$ contains a fixed ball of radius 1 is larger than some universal constant $p<1$, then there is positive probability that $Z$ contains (bi-infinite) lines. We then consider a family of random sets in $\H^2$ that satisfy some additional natural assumptions. An example of such a set is the covered region in the Poisson Boolean model. Let $f(r)$ be the probability that a line segment of length $r$ is contained in such a set $Z$. We show that if $f(r)$ decays fast enough, then there are almost surely no lines in $Z$. We also show that if the decay of $f(r)$ is not too fast, then there are almost surely lines in $Z$. In the case of the Poisson Boolean model with balls of fixed radius $R$ we characterize the critical intensity for the almost sure existence of lines in the covered region by an integral equation. We also determine when there are lines in the complement of a Poisson process on the Grassmannian of lines in $\H^2$.

math.PR↗
arXiv2008

Dynamical models for circle covering: Brownian motion and Poisson updating

We consider two dynamical variants of Dvoretzky's classical problem of random interval coverings of the unit circle, the latter having been completely solved by L. Shepp. In the first model, the centers of the intervals perform independent Brownian motions and in the second model, the positions of the intervals are updated according to independent Poisson processes where an interval of length $\ell$ is updated at rate $\ell^{-α}$ where $α\ge0$ is a parameter. For the model with Brownian motions, a special case of our results is that if the length of the $n$th interval is $c/n$, then there are times at which a fixed point is not covered if and only if $c<2$ and there are times at which the circle is not fully covered if and only if $c<3$. For the Poisson updating model, we obtain analogous results with $c<α$ and $c<α+1$ instead. We also compute the Hausdorff dimension of the set of exceptional times for some of these questions.

math.PR↗