arXiv subjects

David Yevick

Publications and source records attributed to David Yevick.

18 recordsLinked to original sources

arXiv2025

Grokking in the Ising Model

Delayed generalization, termed grokking, in a machine learning calculation occurs when the increase in test accuracy is delayed relative to the training accuracy. This paper examines grokking in the context of a dense neural network trained to classify 2D Ising model configurations into 4 equally spaced energy regions in the presence of weight decay. Partially with the aid of novel PCA-based network layer analysis techniques, the observed behavior is interpreted as a transition from a connected network to a group of sparse subnetworks in which the number of active weights in each layer decreases monotonically with depth. This architecture reduces classification errors resulting from a multiplicity of paths. The final network layers, as in a convolutional neural network, sequentially identify global features of the input classes, which enables generalization to previously unseen patterns.

cond-mat.dis-nn↗

arXiv2025

Tracing the Path to Grokking: Embeddings, Dropout, and Network Activation

Grokking refers to delayed generalization in which the increase in test accuracy of a neural network occurs appreciably after the improvement in training accuracy This paper introduces several practical metrics including variance under dropout, robustness, embedding similarity, and sparsity measures, that can forecast grokking behavior. Specifically, the resilience of neural networks to noise during inference is estimated from a Dropout Robustness Curve (DRC) obtained from the variation of the accuracy with the dropout rate as the model transitions from memorization to generalization. The variance of the test accuracy under stochastic dropout across training checkpoints further exhibits a local maximum during the grokking. Additionally, the percentage of inactive neurons decreases during generalization, while the embeddings tend to a bimodal distribution independent of initialization that correlates with the observed cosine similarity patterns and dataset symmetries. These metrics additionally provide valuable insight into the origin and behaviour of grokking.

cs.LG↗

arXiv2024

Controlling Grokking with Nonlinearity and Data Symmetry

This paper demonstrates that grokking behavior in modular arithmetic with a modulus P in a neural network can be controlled by modifying the profile of the activation function as well as the depth and width of the model. Plotting the even PCA projections of the weights of the last NN layer against their odd projections further yields patterns which become significantly more uniform when the nonlinearity is increased by incrementing the number of layers. These patterns can be employed to factor P when P is nonprime. Finally, a metric for the generalization ability of the network is inferred from the entropy of the layer weights while the degree of nonlinearity is related to correlations between the local entropy of the weights of the neurons in the final layer.

cs.LG↗

arXiv2024

Nonlinearity Enhanced Adaptive Activation Functions

A general procedure for introducing parametric, learned, nonlinearity into activation functions is found to enhance the accuracy of representative neural networks without requiring significant additional computational resources. Examples are given based on the standard rectified linear unit (ReLU) as well as several other frequently employed activation functions. The associated accuracy improvement is quantified both in the context of the MNIST digit data set and a convolutional neural network (CNN) benchmark example.

cs.LG↗

arXiv2024

Branched Variational Autoencoder Classifiers

This paper introduces a modified variational autoencoder (VAEs) that contains an additional neural network branch. The resulting branched VAE (BVAE) contributes a classification component based on the class labels to the total loss and therefore imparts categorical information to the latent representation. As a result, the latent space distributions of the input classes are separated and ordered, thereby enhancing the classification accuracy. The degree of improvement is quantified by numerical calculations employing the benchmark MNIST dataset for both unrotated and rotated digits. The proposed technique is then compared to and then incorporated into a VAE with fixed output distributions. This procedure is found to yield improved performance for a wide range of output distributions.

cs.LG↗

arXiv2023

Neural Network Characterization and Entropy Regulated Data Balancing through Principal Component Analysis

This paper examines in detail the geometric structure of principal component analysis (PCA) by considering in detail the distributions of both unrotated and rotated MNIST digits in the space defined by the lowest order PCA components. Since digits possessing salient geometric features are mapped to restricted regions far from the origin, they are predicted by neural networks with a greater accuracy than digits that are mapped to broad, diffuse and overlapping volumes of the low order PCA space. Motivated by these results, a new quantity, the local PCA entropy, obtained by dividing the spatial region spanned by the low order principal components into histogram bins and evaluating the entropy associated with the number of occurrences of each input class within a bin, is introduced. The metric locates the input data records that yield the largest confusion in prediction accuracy within reduced coordinate volumes that optimally discriminate among geometric features. As an example of the potential utility of the local PCA entropy, a simple data balancing procedure is realized by oversampling the data records in regions of large local entropy.

cs.LG↗

arXiv2022

Rotated Digit Recognition by Variational Autoencoders with Fixed Output Distributions

This paper demonstrates that a simple modification of the variational autoencoder (VAE) formalism enables the method to identify and classify rotated and distorted digits. In particular, the conventional objective (cost) function employed during the training process of a VAE both quantifies the agreement between the input and output data records and ensures that the latent space representation of the input data record is statistically generated with an appropriate mean and standard deviation. After training, simulated data realizations are generated by decoding appropriate latent space points. Since, however, standard VAE:s trained on randomly rotated MNIST digits cannot reliably distinguish between different digit classes since the rotated input data is effectively compared to a similarly rotated output data record. In contrast, an alternative implementation in which the objective function compares the output associated with each rotated digit to a corresponding fixed unreferenced reference digit is shown here to discriminate accurately among the rotated digits in latent space even when the dimension of the latent space is 2 or 3.

cs.CV↗

arXiv2021

Variational Autoencoder Analysis of Ising Model Statistical Distributions and Phase Transitions

Variational autoencoders employ an encoding neural network to generate a probabilistic representation of a data set within a low-dimensional space of latent variables followed by a decoding stage that maps the latent variables back to the original variable space. Once trained, a statistical ensemble of simulated data realizations can be obtained by randomly assigning values to the latent variables that are subsequently processed by the decoding section of the network. To determine the accuracy of such a procedure when applied to lattice models, an autoencoder is here trained on a thermal equilibrium distribution of Ising spin realizations. When the output of the decoder for synthetic data is interpreted probabilistically, spin realizations can be generated by randomly assigning spin values according to the computed likelihood. The resulting state distribution in energy-magnetization space then qualitatively resembles that of the training samples. However, because correlations between spins are suppressed, the computed energies are unphysically large for low-dimensional latent variable spaces. The features of the learned distributions as a function of temperature, however, provide a qualitative indication of the presence of a phase transition and the distribution of realizations with characteristic cluster sizes.

cond-mat.stat-mech↗

arXiv2020

Conservation Laws and Spin System Modeling through Principal Component Analysis

This paper examines several applications of principal component analysis (PCA) to physical systems. The first of these demonstrates that the principal components in a basis of appropriate system variables can be employed to identify physically conserved quantities. That is, if the general form of a physical symmetry law is known, the PCA can identify an algebraic expression for the symmetry from the observed system trajectories. Secondly, the eigenvalue spectrum of the principal component spectrum for homogeneous periodic spin systems is found to reflect the geometric shape of the boundary. Finally, the PCA is employed to generate synthetic spin realizations with probability distributions in energy-magnetization space that closely resemble that of the input realizations although statistical quantities are inaccurately reproduced.

physics.data-an↗

arXiv2020

The Accuracy of Restricted Boltzmann Machine Models of Ising Systems

Restricted Boltzmann machine (RBM) provide a general framework for modeling physical systems, but their behavior is dependent on hyperparameters such as the learning rate, the number of hidden nodes and the form of the threshold function. This article accordingly examines in detail the influence of these parameters on Ising spin system calculations. A tradeoff is identified between the accuracy of statistical quantities such as the specific heat and that of the joint distribution of energy and magnetization. The optimal structure of the RBM therefore depends intrinsically on the physical problem to which it is applied.

physics.comp-ph↗

arXiv2020

Electric Field Propagation Through Singular Value Decomposition

We demonstrate that the singular value decomposition algorithm in conjunction with the fast Fourier transform or finite difference procedures provides a straightforward and accurate method for rapidly propagating electric fields in the one-way Helmholtz formalism.

cs.CE↗

arXiv2020

Accuracy and Efficiency of Simplified Tensor Network Codes

We examine in detail the accuracy, efficiency and implementation issues that arise when a simplified code structure is employed to evaluate the partition function of the two-dimensional square Ising model on periodic lattices though repeated tensor contractions.

cond-mat.stat-mech↗

arXiv2019

A Cluster Controller for Transition Matrix Calculations

We demonstrate that a temperature schedule for single-spin flip transition matrix calculations can be simply and rapidly generated by monitoring the average size of the Wolff clusters at a set of discrete temperatures. Optimizing this schedule yields a potentially interesting quantity related to the fractal structure of Ising clusters. We also introduce a technique in which the transition matrix is constructed at a sequence of discrete temperatures at which Wolff cluster reversals are alternated with certain series of single-spin flip steps. The single spin-flip transitions are then employed to construct a single transition matrix.

cond-mat.stat-mech↗

arXiv2018

Transition Matrix Cluster Algorithms

We demonstrate that a series of procedures for increasing the efficiency of transition matrix calculations can be realized by integrating the standard single-spin flip transition matrix method with global cluster flipping techniques. Our calculations employ a simple and accurate method based on detailed balance for computing the density of states from the Ising model transition matrix.

cond-mat.stat-mech↗

arXiv2018

A Projected Entropy Controller for Transition Matrix Calculations

We define the projected entropy S(T) at a given temperature T in the context of an Ising model transition matrix calculation as the entropy associated with the distribution of Markov chain realizations in energy-magnetization, E-H, space. An even sampling of states is achieved by accumulating the results from multiple Markov chains while decrementing 1/T at a rate proportional to the inverse of the effective number, exp(S(T)), of accessible projected states. Such a procedure is both highly accurate and far simpler to implement than a previously suggested method based on monitoring the evolution of the E-H distribution at each temperature. [1] We further demonstrate a transition matrix procedure that instead ensures uniform sampling in physical entropy.

cond-mat.stat-mech↗

arXiv2016

Dynamic Canonical and Microcanonical Transition Matrix Analyses of Critical Behavior

By monitoring the sampling of states with different magnetizations in transition matrix procedures a family of accurate and easily implemented techniques are constructed that automatically control the variation of the temperature or energy as the calculation proceeds. The accuracy of the method for a single Markov chain exceeds that of standard transition matrix procedures that accumulate elements from multiple chains.

cond-mat.stat-mech↗

arXiv2015

Renormalized Multicanonical Sampling

For a homogeneous system divisible into identical, weakly interacting subsystems, the muticanonical procedure can be accelerated if it is first applied to determine of the density of states for a single subsystem. This result is then employed to approximate the state density of a subsystem with twice the size that forms the starting point of a new multicanonical iteration. Since this compound subsystem interacts less on average with its environment, iterating this sequence of steps rapidly generates the state density of the full system.

cond-mat.stat-mech↗

arXiv2015

Accelerated rare event sampling

A sampling procedure for the transition matrix Monte Carlo method is introduced that generates the density of states function over a wide parameter range with minimal coding effort.

cond-mat.stat-mech↗