arXiv subjects

Tony Chiang

Publications and source records attributed to Tony Chiang.

16 recordsLinked to original sources

arXiv2026

Ferroelectric dynamic-field-driven nucleation and growth model for predictive materials-to-circuit co-design

Real ferroelectric devices operate under mixed and distorted time-varying voltages, yet the standard nucleation-growth frameworks used to interpret ferroelectric switching - most notably the Kolmogorov-Avrami-Ishibashi (KAI) and nucleation-limited switching models (NLS) - are derived under the critically limiting assumption of a constant electric field. Thus, the prevailing interpretation of ferroelectric switching dynamics fails under real operating conditions. Here we introduce a compact dynamic-field-driven nucleation and growth (DFNG) model that enables quantitative fits to switching transients across multiple ferroelectric materials to extract time-varying domain wall velocity and growth dimensionality, even under arbitrary voltage waveform. This capability then motivates its use in device modeling under complex signals spanning disparate time and frequency scales. Coupling the compact model to application-related waveforms and circuit-level simulation platform facilitates a predictive materials-circuit co-design framework by linking nucleation and growth parameters to memory window, disturb error, speed, and energy dissipation for next-generation ferroelectric technologies.

cond-mat.mtrl-sci↗

arXiv2025

Intertwined polar, chiral, and ferro-rotational orders in a rotation-only insulator

Intertwined orders refer to strongly coupled and mutually dependent orders that coexist in correlated electron systems, often underpinning key physical properties of the host materials. Among them, polar, chiral, and ferro-rotational orders have been theoretically known to form a closed set of intertwined orders. However, experimental investigation into their mutual coupling and physical consequences has remained elusive. In this work, we employ the polar-chiral insulator Ni$_3$TeO$_6$ as a platform and utilize a multimodal optical approach to directly probe and reveal the intertwining among polarity, chirality, and ferro-rotational order. We demonstrate how their coupling governs the formation of domains and dictates the nature of domain walls. Within the domains, we identify spatial inversion symmetry as the operation connecting two domain states of opposite polarity and chirality, with a common ferro-rotational state serving as the prerequisite for these interlocked configurations. At the domain walls, we observe a pronounced enhancement of in-plane polarization accompanied by a suppression of chirality. By combining with Ginzburg-Landau theory within the framework of a pre-existing ferro-rotational background, we uncover the emergence of mixed N\'eel- and Bloch-type domain walls. Our findings highlight the critical role of intertwined orders in defining domain and domain wall characteristics and open pathways for domain switching and domain wall control via intertwined order parameters.

cond-mat.mtrl-sci↗

arXiv2025

Material-Limited Switching in Nanoscale Ferroelectrics

The ferroelectric switching speed has been experimentally obfuscated by the interaction between the measurement circuit and the ferroelectric switching itself. This has prohibited the observation of real material responses at nanosecond timescales and lower. Here, fundamental polarization switching speeds in ferroelectric materials with the perovskite, fluorite, and wurtzite structures are reported. Upon lateral scaling of island capacitors from micron to nanoscales, a clear transition from circuit-limited switching to a material-limited switching regime is observed. In La$_{0.15}$Bi$_{0.85}$FeO$_{3}$ capacitors, switching is as fast as ~150 ps, the fastest switching time reported. For polycrystalline Hf$_{0.5}$Zr$_{0.5}$O$_{2}$ capacitors, a fundamental switching limit of ~210 ps is observed. Switching times for Al$_{0.92}$B$_{0.08}$N are near 20 ns, limited by the coercive and breakdown electric fields. The activation field, instantaneous pseudo-resistivity, and energy-delay are reported in this material-limited regime. Lastly, a criterion for reaching the material-limited regime is provided. This regime enables observation of intrinsic material properties and favorable scaling trends for high-performance computing.

cond-mat.mtrl-sci↗

arXiv2025

Token embeddings violate the manifold hypothesis

A full understanding of the behavior of a large language model (LLM) requires our grasp of its input token space. If this space differs from our assumptions, our comprehension of and conclusions about the LLM will likely be flawed. We elucidate the structure of the token embeddings both empirically and theoretically. We present a novel statistical test assuming that the neighborhood around each token has a relatively flat and smooth structure as the null hypothesis. Failing to reject the null is uninformative, but rejecting it at a specific token $\psi$ implies an irregularity in the token subspace in a $\psi$-neighborhood, $B(\psi)$. The structure assumed in the null is a generalization of a manifold with boundary called a \emph{smooth fiber bundle} (which can be split into two spatial regimes -- small and large radius), so we denote our new hypothesis test as the ``fiber bundle hypothesis.'' By running our test over several open-source LLMs, each with unique token embeddings, we find that the null is frequently rejected, and so the evidence suggests that the token subspace is not a fiber bundle and hence also not a manifold. As a consequence of our findings, when an LLM is presented with two semantically equivalent prompts, if one prompt contains a token implicated by our test, the response to that prompt will likely exhibit less stability than the other.

cs.CL↗

arXiv2025

Assessing Generative Models for Structured Data

Synthetic tabular data generation has emerged as a promising method to address limited data availability and privacy concerns. With the sharp increase in the performance of large language models in recent years, researchers have been interested in applying these models to the generation of tabular data. However, little is known about the quality of the generated tabular data from large language models. The predominant method for assessing the quality of synthetic tabular data is the train-synthetic-test-real approach, where the artificial examples are compared to the original by how well machine learning models, trained separately on the real and synthetic sets, perform in some downstream tasks. This method does not directly measure how closely the distribution of generated data approximates that of the original. This paper introduces rigorous methods for directly assessing synthetic tabular data against real data by looking at inter-column dependencies within the data. We find that large language models (GPT-2), both when queried via few-shot prompting and when fine-tuned, and GAN (CTGAN) models do not produce data with dependencies that mirror the original real data. Results from this study can inform future practice in synthetic data generation to improve data quality.

cs.LG↗

arXiv2024

Understanding Generative AI Content with Embedding Models

Constructing high-quality features is critical to any quantitative data analysis. While feature engineering was historically addressed by carefully hand-crafting data representations based on domain expertise, deep neural networks (DNNs) now offer a radically different approach. DNNs implicitly engineer features by transforming their input data into hidden feature vectors called embeddings. For embedding vectors produced by foundation models -- which are trained to be useful across many contexts -- we demonstrate that simple and well-studied dimensionality-reduction techniques such as Principal Component Analysis uncover inherent heterogeneity in input data concordant with human-understandable explanations. Of the many applications for this framework, we find empirical evidence that there is intrinsic separability between real samples and those generated by artificial intelligence (AI).

cs.LG↗

arXiv2024

Measuring training variability from stochastic optimization using robust nonparametric testing

Deep neural network training often involves stochastic optimization, meaning each run will produce a different model. This implies that hyperparameters of the training process, such as the random seed itself, can potentially have significant influence on the variability in the trained models. Measuring model quality by summary statistics, such as test accuracy, can obscure this dependence. We propose a robust hypothesis testing framework and a novel summary statistic, the $\alpha$-trimming level, to measure model similarity. Applying hypothesis testing directly with the $\alpha$-trimming level is challenging because we cannot accurately describe the distribution under the null hypothesis. Our framework addresses this issue by determining how closely an approximate distribution resembles the expected distribution of a group of individually trained models and using this approximation as our reference. We then use the $\alpha$-trimming level to suggest how many training runs should be sampled to ensure that an ensemble is a reliable representative of the true model performance. We also show how to use the $\alpha$-trimming level to measure model variability and demonstrate experimentally that it is more expressive than performance metrics like validation accuracy, churn, or expected calibration error when taken alone. An application of fine-tuning over random seed in transfer learning illustrates the advantage of our new metric.

stat.ML↗

arXiv2023

Efficient kernel surrogates for neural network-based regression

Despite their immense promise in performing a variety of learning tasks, a theoretical understanding of the limitations of Deep Neural Networks (DNNs) has so far eluded practitioners. This is partly due to the inability to determine the closed forms of the learned functions, making it harder to study their generalization properties on unseen datasets. Recent work has shown that randomly initialized DNNs in the infinite width limit converge to kernel machines relying on a Neural Tangent Kernel (NTK) with known closed form. These results suggest, and experimental evidence corroborates, that empirical kernel machines can also act as surrogates for finite width DNNs. The high computational cost of assembling the full NTK, however, makes this approach infeasible in practice, motivating the need for low-cost approximations. In the current work, we study the performance of the Conjugate Kernel (CK), an efficient approximation to the NTK that has been observed to yield fairly similar results. For the regression problem of smooth functions and logistic regression classification, we show that the CK performance is only marginally worse than that of the NTK and, in certain cases, is shown to be superior. In particular, we establish bounds for the relative test losses, verify them with numerical tests, and identify the regularity of the kernel as the key determinant of performance. In addition to providing a theoretical grounding for using CKs instead of NTKs, our framework suggests a recipe for improving DNN accuracy inexpensively. We present a demonstration of this on the foundation model GPT-2 by comparing its performance on a classification task using a conventional approach and our prescription. We also show how our approach can be used to improve physics-informed operator network training for regression tasks as well as convolutional neural network training for vision classification tasks.

cs.LG↗

arXiv2023

Computational and Systems Biology Advances to Enable Bioagent-Agnostic Signatures

Enumerated threat agent lists have long driven biodefense priorities. The global SARS-CoV-2 pandemic demonstrated the limitations of searching for known threat agents as compared to a more agnostic approach. Recent technological advances are enabling agent-agnostic biodefense, especially through the integration of multi-modal observations of host-pathogen interactions directed by a human immunological model. Although well-developed technical assays exist for many aspects of human-pathogen interaction, the analytic methods and pipelines to combine and holistically interpret the results of such assays are immature and require further investments to exploit new technologies. In this manuscript, we discuss potential immunologically based bioagent-agnostic approaches and the computational tool gaps the community should prioritize filling.

q-bio.OT↗

arXiv2023

Foundation Model's Embedded Representations May Detect Distribution Shift

Sampling biases can cause distribution shifts between train and test datasets for supervised learning tasks, obscuring our ability to understand the generalization capacity of a model. This is especially important considering the wide adoption of pre-trained foundational neural networks -- whose behavior remains poorly understood -- for transfer learning (TL) tasks. We present a case study for TL on the Sentiment140 dataset and show that many pre-trained foundation models encode different representations of Sentiment140's manually curated test set $M$ from the automatically labeled training set $P$, confirming that a distribution shift has occurred. We argue training on $P$ and measuring performance on $M$ is a biased measure of generalization. Experiments on pre-trained GPT-2 show that the features learnable from $P$ do not improve (and in fact hamper) performance on $M$. Linear probes on pre-trained GPT-2's representations are robust and may even outperform overall fine-tuning, implying a fundamental importance for discerning distribution shift in train/test splits for model interpretation.

cs.LG↗

arXiv2023

Robust Nonparametric Hypothesis Testing to Understand Variability in Training Neural Networks

Training a deep neural network (DNN) often involves stochastic optimization, which means each run will produce a different model. Several works suggest this variability is negligible when models have the same performance, which in the case of classification is test accuracy. However, models with similar test accuracy may not be computing the same function. We propose a new measure of closeness between classification models based on the output of the network before thresholding. Our measure is based on a robust hypothesis-testing framework and can be adapted to other quantities derived from trained models.

stat.ML↗

arXiv2023

Exploring Learned Representations of Neural Networks with Principal Component Analysis

Understanding feature representation for deep neural networks (DNNs) remains an open question within the general field of explainable AI. We use principal component analysis (PCA) to study the performance of a k-nearest neighbors classifier (k-NN), nearest class-centers classifier (NCC), and support vector machines on the learned layer-wise representations of a ResNet-18 trained on CIFAR-10. We show that in certain layers, as little as 20% of the intermediate feature-space variance is necessary for high-accuracy classification and that across all layers, the first ~100 PCs completely determine the performance of the k-NN and NCC classifiers. We relate our findings to neural collapse and provide partial evidence for the related phenomenon of intermediate neural collapse. Our preliminary work provides three distinct yet interpretable surrogate models for feature representation with an affine linear model the best performing. We also show that leveraging several surrogate models affords us a clever method to estimate where neural collapse may initially occur within the DNN.

cs.LG↗

arXiv2023

Minibatching Offers Improved Generalization Performance for Second Order Optimizers

Training deep neural networks (DNNs) used in modern machine learning is computationally expensive. Machine learning scientists, therefore, rely on stochastic first-order methods for training, coupled with significant hand-tuning, to obtain good performance. To better understand performance variability of different stochastic algorithms, including second-order methods, we conduct an empirical study that treats performance as a response variable across multiple training sessions of the same model. Using 2-factor Analysis of Variance (ANOVA) with interactions, we show that batch size used during training has a statistically significant effect on the peak accuracy of the methods, and that full batch largely performed the worst. In addition, we found that second-order optimizers (SOOs) generally exhibited significantly lower variance at specific batch sizes, suggesting they may require less hyperparameter tuning, leading to a reduced overall time to solution for model training.

cs.LG↗

arXiv2023

Faithful and Efficient Explanations for Neural Networks via Neural Tangent Kernel Surrogate Models

A recent trend in explainable AI research has focused on surrogate modeling, where neural networks are approximated as simpler ML algorithms such as kernel machines. A second trend has been to utilize kernel functions in various explain-by-example or data attribution tasks. In this work, we combine these two trends to analyze approximate empirical neural tangent kernels (eNTK) for data attribution. Approximation is critical for eNTK analysis due to the high computational cost to compute the eNTK. We define new approximate eNTK and perform novel analysis on how well the resulting kernel machine surrogate models correlate with the underlying neural network. We introduce two new random projection variants of approximate eNTK which allow users to tune the time and memory complexity of their calculation. We conclude that kernel machines using approximate neural tangent kernel as the kernel function are effective surrogate models, with the introduced trace NTK the most consistent performer. Open source software allowing users to efficiently calculate kernel functions in the PyTorch framework is available (https://github.com/pnnl/projection\_ntk).

cs.LG↗

arXiv2022

Spectral Evolution and Invariance in Linear-width Neural Networks

We investigate the spectral properties of linear-width feed-forward neural networks, where the sample size is asymptotically proportional to network width. Empirically, we show that the spectra of weight in this high dimensional regime are invariant when trained by gradient descent for small constant learning rates; we provide a theoretical justification for this observation and prove the invariance of the bulk spectra for both conjugate and neural tangent kernels. We demonstrate similar characteristics when training with stochastic gradient descent with small learning rates. When the learning rate is large, we exhibit the emergence of an outlier whose corresponding eigenvector is aligned with the training data structure. We also show that after adaptive gradient training, where a lower test error and feature learning emerge, both weight and kernel matrices exhibit heavy tail behavior. Simple examples are provided to explain when heavy tails can have better generalizations. We exhibit different spectral properties such as invariant bulk, spike, and heavy-tailed distribution from a two-layer neural network using different training strategies, and then correlate them to the feature learning. Analogous phenomena also appear when we train conventional neural networks with real-world data. We conclude that monitoring the evolution of the spectra during training is an essential step toward understanding the training dynamics and feature learning.

cs.LG↗

arXiv2022

TorchNTK: A Library for Calculation of Neural Tangent Kernels of PyTorch Models

We introduce torchNTK, a python library to calculate the empirical neural tangent kernel (NTK) of neural network models in the PyTorch framework. We provide an efficient method to calculate the NTK of multilayer perceptrons. We compare the explicit differentiation implementation against autodifferentiation implementations, which have the benefit of extending the utility of the library to any architecture supported by PyTorch, such as convolutional networks. A feature of the library is that we expose the user to layerwise NTK components, and show that in some regimes a layerwise calculation is more memory efficient. We conduct preliminary experiments to demonstrate use cases for the software and probe the NTK.

cs.LG↗