arXiv subjects

Victor Veitch

Publications and source records attributed to Victor Veitch.

46 records · Page 3Linked to original sources

arXiv2019

Non-Vacuous Generalization Bounds at the ImageNet Scale: A PAC-Bayesian Compression Approach

Modern neural networks are highly overparameterized, with capacity to substantially overfit to training data. Nevertheless, these networks often generalize well in practice. It has also been observed that trained networks can often be "compressed" to much smaller representations. The purpose of this paper is to connect these two empirical observations. Our main technical result is a generalization bound for compressed networks based on the compressed size. Combined with off-the-shelf compression algorithms, the bound leads to state of the art generalization guarantees; in particular, we provide the first non-vacuous generalization guarantees for realistic architectures applied to the ImageNet classification problem. As additional evidence connecting compression and generalization, we show that compressibility of models that tend to overfit is limited: We establish an absolute limit on expected compressibility as a function of expected generalization error, where the expectations are over the random choice of training examples. The bounds are complemented by empirical results that show an increase in overfitting implies an increase in the number of bits required to describe a trained network.

stat.ML↗

arXiv2019

Empirical Risk Minimization and Stochastic Gradient Descent for Relational Data

Empirical risk minimization is the main tool for prediction problems, but its extension to relational data remains unsolved. We solve this problem using recent ideas from graph sampling theory to (i) define an empirical risk for relational data and (ii) obtain stochastic gradients for this empirical risk that are automatically unbiased. This is achieved by considering the method by which data is sampled from a graph as an explicit component of model design. By integrating fast implementations of graph sampling schemes with standard automatic differentiation tools, we provide an efficient turnkey solver for the risk minimization problem. We establish basic theoretical properties of the procedure. Finally, we demonstrate relational ERM with application to two non-standard problems: one-stage training for semi-supervised node classification, and learning embedding vectors for vertex attributes. Experiments confirm that the turnkey inference procedure is effective in practice, and that the sampling scheme used for model specification has a strong effect on model performance. Code is available at https://github.com/wooden-spoon/relational-ERM.

stat.ML↗

arXiv2017

Exchangeable modelling of relational data: checking sparsity, train-test splitting, and sparse exchangeable Poisson matrix factorization

A variety of machine learning tasks---e.g., matrix factorization, topic modelling, and feature allocation---can be viewed as learning the parameters of a probability distribution over bipartite graphs. Recently, a new class of models for networks, the sparse exchangeable graphs, have been introduced to resolve some important pathologies of traditional approaches to statistical network modelling; most notably, the inability to model sparsity (in the asymptotic sense). The present paper explains some practical insights arising from this work. We first show how to check if sparsity is relevant for modelling a given (fixed size) dataset by using network subsampling to identify a simple signature of sparsity. We discuss the implications of the (sparse) exchangeable subsampling theory for test-train dataset splitting; we argue common approaches can lead to biased results, and we propose a principled alternative. Finally, we study sparse exchangeable Poisson matrix factorization as a worked example. In particular, we show how to adapt mean field variational inference to the sparse exchangeable setting, allowing us to scale inference to huge datasets.

stat.ML↗

arXiv2016

Sampling and Estimation for (Sparse) Exchangeable Graphs

Sparse exchangeable graphs on $\mathbb{R}_+$, and the associated graphex framework for sparse graphs, generalize exchangeable graphs on $\mathbb{N}$, and the associated graphon framework for dense graphs. We develop the graphex framework as a tool for statistical network analysis by identifying the sampling scheme that is naturally associated with the models of the framework, and by introducing a general consistent estimator for the parameter (the graphex) underlying these models. The sampling scheme is a modification of independent vertex sampling that throws away vertices that are isolated in the sampled subgraph. The estimator is a dilation of the empirical graphon estimator, which is known to be a consistent estimator for dense exchangeable graphs; both can be understood as graph analogues to the empirical distribution in the i.i.d. sequence setting. Our results may be viewed as a generalization of consistent estimation via the empirical graphon from the dense graph regime to also include sparse graphs.

math.ST↗

arXiv2015

The Class of Random Graphs Arising from Exchangeable Random Measures

We introduce a class of random graphs that we argue meets many of the desiderata one would demand of a model to serve as the foundation for a statistical analysis of real-world networks. The class of random graphs is defined by a probabilistic symmetry: invariance of the distribution of each graph to an arbitrary relabelings of its vertices. In particular, following Caron and Fox, we interpret a symmetric simple point process on $\mathbb{R}_+^2$ as the edge set of a random graph, and formalize the probabilistic symmetry as joint exchangeability of the point process. We give a representation theorem for the class of random graphs satisfying this symmetry via a straightforward specialization of Kallenberg's representation theorem for jointly exchangeable random measures on $\mathbb{R}_+^2$. The distribution of every such random graph is characterized by three (potentially random) components: a nonnegative real $I \in \mathbb{R}_+$, an integrable function $S: \mathbb{R}_+ \to \mathbb{R}_+$, and a symmetric measurable function $W: \mathbb{R}_+^2 \to [0,1]$ that satisfies several weak integrability conditions. We call the triple $(I,S,W)$ a graphex, in analogy to graphons, which characterize the (dense) exchangeable graphs on $\mathbb{N}$. Indeed, the model we introduce here contains the exchangeable graphs as a special case, as well as the "sparse exchangeable" model of Caron and Fox. We study the structure of these random graphs, and show that they can give rise to interesting structure, including sparse graph sequences. We give explicit equations for expectations of certain graph statistics, as well as the limiting degree distribution. We also show that certain families of graphexes give rise to random graphs that, asymptotically, contain an arbitrarily large fraction of the vertices in a single connected component.

math.ST↗

arXiv2014

Contextuality supplies the magic for quantum computation

Quantum computers promise dramatic advantages over their classical counterparts, but the answer to the most basic question "What is the source of the power in quantum computing?" has remained elusive. Here we prove a remarkable equivalence between the onset of contextuality and the possibility of universal quantum computation via magic state distillation. This is a conceptually satisfying link because contextuality provides one of the fundamental characterizations of uniquely quantum phenomena and, moreover, magic state distillation is the leading model for experimentally realizing fault-tolerant quantum computation. Furthermore, this connection suggests a unifying paradigm for the resources of quantum information: the nonlocality of quantum theory is a particular kind of contextuality and nonlocality is already known to be a critical resource for achieving advantages with quantum communication. In addition to clarifying these fundamental issues, this work advances the resource framework for quantum computation, which has a number of practical applications, such as characterizing the efficiency and trade-offs between distinct theoretical and experimental schemes for achieving robust quantum computation and bounding the overhead cost for the classical simulation of quantum algorithms.

quant-ph↗

arXiv2013

The whole is greater than the sum of the parts: on the possibility of purely statistical interpretations of quantum theory

The Pusey-Barrett-Rudolph theorem (PBR) claims to rule out the possibility of a purely statistical interpretation of the quantum state under an assumption of how to represent independent operations in any hidden variable model. We show that PBR's assumption of independence encodes an assumption of local causality, which is already known to conflict with the predictions of quantum theory via Bell-type inequalities. We devise a weaker formulation of independence within a general hidden variable model that is empirically indistinguishable from the PBR assumption in situations where certain hidden variables are inaccessible. Under this weaker principle we are able to construct an explicit hidden variable model that is purely statistical and also reproduces the quantum predictions. Our results suggest that the assumption of a purely statistical interpretation is actually an innocent bystander in the PBR argument, rather than the driving force behind their contradiction.

quant-ph↗

arXiv2013

The Resource Theory of Stabilizer Computation

Recent results on the non-universality of fault-tolerant gate sets underline the critical role of resource states, such as magic states, to power scalable, universal quantum computation. Here we develop a resource theory, analogous to the theory of entanglement, for resources for stabilizer codes. We introduce two quantitative measures - monotones - for the amount of non-stabilizer resource. As an application we give absolute bounds on the efficiency of magic state distillation. One of these monotones is the sum of the negative entries of the discrete Wigner representation of a quantum state, thereby resolving a long-standing open question of whether the degree of negativity in a quasi-probability representation is an operationally meaningful indicator of quantum behaviour.

quant-ph↗

arXiv2012

Efficient simulation scheme for a class of quantum optics experiments with non-negative Wigner representation

We provide a scheme for efficient simulation of a broad class of quantum optics experiments. Our efficient simulation extends the continuous variable Gottesman-Knill theorem to a large class of non-Gaussian mixed states, thereby identifying that these non-Gaussian states are not an enabling resource for exponential quantum speed-up. Our results also provide an operationally motivated interpretation of negativity as non-classicality. We apply our scheme to the case of noisy single-photon-added-thermal-states to show that this class admits states with positive Wigner function but negative P -function that are not useful resource states for quantum computation.

quant-ph↗

arXiv2012

Negative Quasi-Probability as a Resource for Quantum Computation

A central problem in quantum information is to determine the minimal physical resources that are required for quantum computational speedup and, in particular, for fault-tolerant quantum computation. We establish a remarkable connection between the potential for quantum speed-up and the onset of negative values in a distinguished quasi-probability representation, a discrete analog of the Wigner function for quantum systems of odd dimension. This connection allows us to resolve an open question on the existence of bound states for magic-state distillation: we prove that there exist mixed states outside the convex hull of stabilizer states that cannot be distilled to non-stabilizer target states using stabilizer operations. We also provide an efficient simulation protocol for Clifford circuits that extends to a large class of mixed states, including bound universal states.

quant-ph↗