arXiv subjects

Michael Lindsey

Publications and source records attributed to Michael Lindsey.

At least 37 records · Page 2Linked to original sources

arXiv2024

Large Stepsize Gradient Descent for Non-Homogeneous Two-Layer Networks: Margin Improvement and Fast Optimization

The typical training of neural networks using large stepsize gradient descent (GD) under the logistic loss often involves two distinct phases, where the empirical risk oscillates in the first phase but decreases monotonically in the second phase. We investigate this phenomenon in two-layer networks that satisfy a near-homogeneity condition. We show that the second phase begins once the empirical risk falls below a certain threshold, dependent on the stepsize. Additionally, we show that the normalized margin grows nearly monotonically in the second phase, demonstrating an implicit bias of GD in training non-homogeneous predictors. If the dataset is linearly separable and the derivative of the activation function is bounded away from zero, we show that the average empirical risk decreases, implying that the first phase must stop in finite steps. Finally, we demonstrate that by choosing a suitably large stepsize, GD that undergoes this phase transition is more efficient than GD that monotonically decreases the risk. Our analysis applies to networks of any width, beyond the well-known neural tangent kernel and mean-field regimes.

stat.ML↗

arXiv2024

Multiscale interpolative construction of quantized tensor trains

Quantized tensor trains (QTTs) have recently emerged as a framework for the numerical discretization of continuous functions, with the potential for widespread applications in numerical analysis. However, the theory of QTT approximation is not fully understood. In this work, we advance this theory from the point of view of multiscale polynomial interpolation. This perspective clarifies why QTT ranks decay with increasing depth, quantitatively controls QTT rank in terms of smoothness of the target function, and explains why certain functions with sharp features and poor quantitative smoothness can still be well approximated by QTTs. The perspective also motivates new practical and efficient algorithms for the construction of QTTs from function evaluations on multiresolution grids.

math.NA↗

arXiv2024

Direct interpolative construction of the discrete Fourier transform as a matrix product operator

The quantum Fourier transform (QFT), which can be viewed as a reindexing of the discrete Fourier transform (DFT), has been shown to be compressible as a low-rank matrix product operator (MPO) or quantized tensor train (QTT) operator. However, the original proof of this fact does not furnish a construction of the MPO with a guaranteed error bound. Meanwhile, the existing practical construction of this MPO, based on the compression of a quantum circuit, is not as efficient as possible. We present a simple closed-form construction of the QFT MPO using the interpolative decomposition, with guaranteed near-optimal compression error for a given rank. This construction can speed up the application of the QFT and the DFT, respectively, in quantum circuit simulations and QTT applications. We also connect our interpolative construction to the approximate quantum Fourier transform (AQFT) by demonstrating that the AQFT can be viewed as an MPO constructed using a different interpolation scheme.

quant-ph↗

arXiv2023

Multimarginal generative modeling with stochastic interpolants

Given a set of $K$ probability densities, we consider the multimarginal generative modeling problem of learning a joint distribution that recovers these densities as marginals. The structure of this joint distribution should identify multi-way correspondences among the prescribed marginals. We formalize an approach to this task within a generalization of the stochastic interpolant framework, leading to efficient learning algorithms built upon dynamical transport of measure. Our generative models are defined by velocity and score fields that can be characterized as the minimizers of simple quadratic objectives, and they are defined on a simplex that generalizes the time variable in the usual dynamical transport framework. The resulting transport on the simplex is influenced by all marginals, and we show that multi-way correspondences can be extracted. The identification of such correspondences has applications to style transfer, algorithmic fairness, and data decorruption. In addition, the multimarginal perspective enables an efficient algorithm for reducing the dynamical transport cost in the ordinary two-marginal setting. We demonstrate these capacities with several numerical examples.

cs.LG↗

arXiv2023

Non-Hertz-Millis scaling of the antiferromagnetic quantum critical metal via scalable Hybrid Monte Carlo

A key component of the phase diagram of many iron-based superconductors and electron-doped cuprates is believed to be a quantum critical point (QCP), delineating the onset of antiferromagnetic spin-density wave order in a quasi-two-dimensional metal. The universality class of this QCP is believed to play a fundamental role in the description of the proximate non-Fermi liquid and superconducting phases. A minimal model for this transition is the $\mathrm{O}(3)$ spin-fermion model. Despite many efforts, a definitive characterization of its universal properties is still lacking. Here, we numerically study the $\mathrm{O}(3)$ spin-fermion model and extract the scaling exponents and functional form of the static and zero-momentum dynamical spin susceptibility. We do this using a Hybrid Monte Carlo (HMC) algorithm with a novel auto-tuning procedure, which allows us to study unprecedentedly large systems of $80 \times 80$ sites. We find a strong violation of the Hertz-Millis form, contrary to all previous results. Furthermore, the form that we do observe provides good evidence that the universal scaling is actually governed by the analytically tractable fixed point discovered near perfect ``hot-spot'" nesting, even for a larger nesting window. Our predictions can be directly tested with neutron scattering. Additionally, the HMC method we introduce is generic and can be used to study other fermionic models of quantum criticality, where there is a strong need to simulate large systems.

cond-mat.str-el↗

arXiv2023

Tensorizing flows: a tool for variational inference

Fueled by the expressive power of deep neural networks, normalizing flows have achieved spectacular success in generative modeling, or learning to draw new samples from a distribution given a finite dataset of training samples. Normalizing flows have also been applied successfully to variational inference, wherein one attempts to learn a sampler based on an expression for the log-likelihood or energy function of the distribution, rather than on data. In variational inference, the unimodality of the reference Gaussian distribution used within the normalizing flow can cause difficulties in learning multimodal distributions. We introduce an extension of normalizing flows in which the Gaussian reference is replaced with a reference distribution that is constructed via a tensor network, specifically a matrix product state or tensor train. We show that by combining flows with tensor networks on difficult variational inference tasks, we can improve on the results obtained by using either tool without the other.

cs.LG↗

arXiv2023

Fast randomized entropically regularized semidefinite programming

We develop a practical approach to semidefinite programming (SDP) that includes the von Neumann entropy, or an appropriate variant, as a regularization term. In particular we solve the dual of the regularized program, demonstrating how a carefully chosen randomized trace estimator can be used to estimate dual gradients effectively. We also introduce specialized optimization approaches for common SDP, specifically SDP with diagonal constraint and the problem of the determining the spectral projector onto the span of extremal eigenvectors. We validate our approach on such problems with applications to combinatorial optimization and spectral embedding.

math.OC↗

arXiv2022

Understanding and eliminating spurious modes in variational Monte Carlo using collective variables

The use of neural network parametrizations to represent the ground state in variational Monte Carlo (VMC) calculations has generated intense interest in recent years. However, as we demonstrate in the context of the periodic Heisenberg spin chain, this approach can produce unreliable wave function approximations. One of the most obvious signs of failure is the occurrence of random, persistent spikes in the energy estimate during training. These energy spikes are caused by regions of configuration space that are over-represented by the wave function density, which are called ``spurious modes'' in the machine learning literature. After exploring these spurious modes in detail, we demonstrate that a collective-variable-based penalization yields a substantially more robust training procedure, preventing the formation of spurious modes and improving the accuracy of energy estimates. Because the penalization scheme is cheap to implement and is not specific to the particular model studied here, it can be extended to other applications of VMC where a reasonable choice of collective variable is available.

physics.chem-ph↗

arXiv2022

Rayleigh-Gauss-Newton optimization with enhanced sampling for variational Monte Carlo

Variational Monte Carlo (VMC) is an approach for computing ground-state wavefunctions that has recently become more powerful due to the introduction of neural network-based wavefunction parametrizations. However, efficiently training neural wavefunctions to converge to an energy minimum remains a difficult problem. In this work, we analyze optimization and sampling methods used in VMC and introduce alterations to improve their performance. First, based on theoretical convergence analysis in a noiseless setting, we motivate a new optimizer that we call the Rayleigh-Gauss-Newton method, which can improve upon gradient descent and natural gradient descent to achieve superlinear convergence at no more than twice the computational cost. Second, in order to realize this favorable comparison in the presence of stochastic noise, we analyze the effect of sampling error on VMC parameter updates and experimentally demonstrate that it can be reduced by the parallel tempering method. In particular, we demonstrate that RGN can be made robust to energy spikes that occur when the sampler moves between metastable regions of configuration space. Finally, putting theory into practice, we apply our enhanced optimization and sampling methods to the transverse-field Ising and XXZ models on large lattices, yielding ground-state energy estimates with remarkably high accuracy after just 200 parameter updates.

stat.ML↗

arXiv2021

Committor functions via tensor networks

We propose a novel approach for computing committor functions, which describe transitions of a stochastic process between metastable states. The committor function satisfies a backward Kolmogorov equation, and in typical high-dimensional settings of interest, it is intractable to compute and store the solution with traditional numerical methods. By parametrizing the committor function in a matrix product state/tensor train format and using a similar representation for the equilibrium probability density, we solve the variational formulation of the backward Kolmogorov equation with linear time and memory complexity in the number of dimensions. This approach bypasses the need for sampling the equilibrium distribution, which can be difficult when the distribution has multiple modes. Numerical results demonstrate the effectiveness of the proposed method for high-dimensional problems.

math.NA↗

arXiv2021

Scalable semidefinite programming approach to variational embedding for quantum many-body problems

In quantum embedding theories, a quantum many-body system is divided into localized clusters of sites which are treated with an accurate `high-level' theory and glued together self-consistently by a less accurate `low-level' theory at the global scale. The recently introduced variational embedding approach for quantum many-body problems combines the insights of semidefinite relaxation and quantum embedding theory to provide a lower bound on the ground-state energy that improves as the cluster size is increased. The variational embedding method is formulated as a semidefinite program (SDP), which can suffer from poor computational scaling when treated with black-box solvers. We exploit the interpretation of this SDP as an embedding method to develop an algorithm which alternates parallelizable local updates of the high-level quantities with updates that enforce the low-level global constraints. Moreover, we show how translation invariance in lattice systems can be exploited to reduce the complexity of projecting a key matrix to the positive semidefinite cone.

math.OC↗

arXiv2021

Ensemble Markov chain Monte Carlo with teleporting walkers

We introduce an ensemble Markov chain Monte Carlo approach to sampling from a probability density with known likelihood. This method upgrades an underlying Markov chain by allowing an ensemble of such chains to interact via a process in which one chain's state is cloned as another's is deleted. This effective teleportation of states can overcome issues of metastability in the underlying chain, as the scheme enjoys rapid mixing once the modes of the target density have been populated. We derive a mean-field limit for the evolution of the ensemble. We analyze the global and local convergence of this mean-field limit, showing asymptotic convergence independent of the spectral gap of the underlying Markov chain, and moreover we interpret the limiting evolution as a gradient flow. We explain how interaction can be applied selectively to a subset of state variables in order to maintain advantage on very high-dimensional problems. Finally we present the application of our methodology to Bayesian hyperparameter estimation for Gaussian process regression.

stat.CO↗

arXiv2020

Multiscale semidefinite programming approach to positioning problems with pairwise structure

We consider the optimization of pairwise objective functions, i.e., objective functions of the form $H(\mathbf{x}) = H(x_1,\ldots,x_N) = \sum_{1\leq i<j \leq N} H_{ij}(x_i,x_j)$ for $x_i$ in some continuous state spaces $\mathcal{X}_i$. Global optimization in this setting is generally confounded by the possible existence of spurious local minima and the impossibility of global search due to the curse of dimensionality. In this paper, we approach such problems via convex relaxation of the marginal polytope considered in graphical modeling, proceeding in a multiscale fashion which exploits the smoothness of the cost function. We show theoretically that, compared with existing methods, such an approach is advantageous even in simple settings for sensor network localization (SNL). We successfully apply our method to SNL problems, particularly difficult instances with high noise. We also validate performance on the optimization of the Lennard-Jones potential, which is plagued by the existence of many near-optimal configurations. We demonstrate that in MMR allows us to effectively explore these configurations.

math.NA↗

arXiv2020

Towards sharp error analysis of extended Lagrangian molecular dynamics

The extended Lagrangian molecular dynamics (XLMD) method provides a useful framework for reducing the computational cost of a class of molecular dynamics simulations with constrained latent variables. The XLMD method relaxes the constraints by introducing a fictitious mass $\varepsilon$ for the latent variables, solving a set of singularly perturbed ordinary differential equations. While favorable numerical performance of XLMD has been demonstrated in several different contexts in the past decade, mathematical analysis of the method remains scarce. We propose the first error analysis of the XLMD method in the context of a classical polarizable force field model. While the dynamics with respect to the atomic degrees of freedom are general and nonlinear, the key mathematical simplification of the polarizable force field model is that the constraints on the latent variables are given by a linear system of equations. We prove that when the initial value of the latent variables is compatible in a sense that we define, XLMD converges as the fictitious mass $\varepsilon$ is made small with $\mathcal{O}(\varepsilon)$ error for the atomic degrees of freedom and with $\mathcal{O}(\sqrt{\varepsilon})$ error for the latent variables, when the dimension of the latent variable $d'$ is 1. Furthermore, when the initial value of the latent variables is improved to be optimally compatible in a certain sense, we prove that the convergence rate can be improved to $\mathcal{O}(\varepsilon)$ for the latent variables as well. Numerical results verify that both estimates are sharp not only for $d' =1$, but also for arbitrary $d'$. In the setting of general $d'$, we do obtain convergence, but with the non-sharp rate of $\mathcal{O}(\sqrt{\varepsilon})$ for both the atomic and latent variables.

math.NA↗

arXiv2020

Semidefinite relaxation of multi-marginal optimal transport for strictly correlated electrons in second quantization

We consider the strictly correlated electron (SCE) limit of the fermionic quantum many-body problem in the second-quantized formalism. This limit gives rise to a multi-marginal optimal transport (MMOT) problem. Here the marginal state space for our MMOT problem is the binary set $\{0,1\}$, and the number of marginals is the number $L$ of sites in the model. The costs of storing and computing the exact solution of the MMOT problem both scale exponentially with respect to $L$. We propose an efficient convex relaxation which can be solved by semidefinite programming (SDP). In particular, the semidefinite constraint is only of size $2L\times 2L$. Moreover, the SDP-based method yields an approximation of the dual potential needed to the perform self-consistent field iteration in the so-called Kohn-Sham SCE framework, which, once converged, yields a lower bound for the total energy of the system. We demonstrate the effectiveness of our methods on spinless and spinful Hubbard-type models. Numerical results indicate that our relaxation methods yield tight lower bounds for the optimal cost, in the sense that the error due to the semidefinite relaxation is much smaller than the intrinsic modeling error of the Kohn-Sham SCE method. We also describe how our relaxation methods generalize to arbitrary MMOT problems with pairwise cost functions.

math.OC↗

arXiv2020

Sparsity pattern of the self-energy for classical and quantum impurity problems

We prove that for various impurity models, in both classical and quantum settings, the self-energy matrix is a sparse matrix with a sparsity pattern determined by the impurity sites. In the quantum setting, such a sparsity pattern has been known since Feynman. Indeed, it underlies several numerical methods for solving impurity problems, as well as many approaches to more general quantum many-body problems, such as the dynamical mean field theory. The sparsity pattern is easily motivated by a formal perturbative expansion using Feynman diagrams. However, to the extent of our knowledge, a rigorous proof has not appeared in the literature. In the classical setting, analogous considerations lead to a perhaps less-known result, i.e., that the precision matrix of a Gibbs measure of a certain kind differs only by a sparse matrix from the precision matrix of a corresponding Gaussian measure. Our argument for this result mainly involves elementary algebraic manipulations and is in particular non-perturbative. Nonetheless, the proof can be robustly adapted to various settings of interest in physics, including quantum systems (both fermionic and bosonic) at zero and finite temperature, non-equilibrium systems, and superconducting systems.

math-ph↗

arXiv2020

Enhancing robustness and efficiency of density matrix embedding theory via semidefinite programming and local correlation potential fitting

Density matrix embedding theory (DMET) is a powerful quantum embedding method for solving strongly correlated quantum systems. Theoretically, the performance of a quantum embedding method should be limited by the computational cost of the impurity solver. However, the practical performance of DMET is often hindered by the numerical stability and the computational time of the correlation potential fitting procedure, which is defined on a single-particle level. Of particular difficulty are cases in which the effective single-particle system is gapless or nearly gapless. To alleviate these issues, we develop a semidefinite programming (SDP) based approach that can significantly enhance the robustness of the correlation potential fitting procedure compared to the traditional least squares fitting approach. We also develop a local correlation potential fitting approach, which allows one to identify the correlation potential from each fragment independently in each self-consistent field iteration, avoiding any optimization at the global level. We prove that the self-consistent solutions of DMET using this local correlation potential fitting procedure are equivalent to those of the original DMET with global fitting. We find that our combined approach, called L-DMET, in which we solve local fitting problems via semidefinite programming, can significantly improve both the robustness and the efficiency of DMET calculations. We demonstrate the performance of L-DMET on the 2D Hubbard model and the hydrogen chain. We also demonstrate with theoretical and numerical evidence that the use of a large fragment size can be a fundamental source of numerical instability in the DMET procedure.

physics.comp-ph↗

arXiv2020

Efficient hybridization fitting for dynamical mean-field theory via semi-definite relaxation

We introduce a nested optimization procedure using semi-definite relaxation for the fitting step in Hamiltonian-based cluster dynamical mean-field theory (DMFT) methodologies. We show that the proposed method is more efficient and flexible than state-of-the-art fitting schemes, which allows us to treat as large a number of bath sites as the impurity solver at hand allows. We characterize its robustness to initial conditions and symmetry constraints, thus providing conclusive evidence that in the presence of a large bath, our semi-definite relaxation approach can find the correct set of bath parameters without needing to include \emph{a priori} knowledge of the properties that are to be described. We believe this method will be of great use for Hamiltonian-based calculations, simplifying and improving one of the key steps in cluster dynamical mean-field theory calculations.

cond-mat.str-el↗