arXiv subjects

Andrew Engel

Publications and source records attributed to Andrew Engel.

13 recordsLinked to original sources

arXiv2025

Identifying environmentally induced calibration changes in cryogenic RF axion detector systems using Deep Neural Networks

The axion is a compelling hypothetical particle that could account for the dark matter in our universe, while simultaneously explaining why quark interactions within the neutron do not appear to give rise to an electric dipole moment. The most sensitive axion detection technique in the 1 to 10 GHz frequency range makes use of the axion-photon coupling and is called the axion haloscope. Within a high Q cavity immersed in a strong magnetic field, axions are converted to microwave photons. As searches scan up in axion mass, towards the parameter space favored by theoretical predictions, individual cavity sizes decrease in order to achieve higher frequencies. This shrinking cavity volume translates directly to a loss in signal-to-noise, motivating the plan to replace individual cavity detectors with arrays of cavities. When the transition from one to (N) multiple cavities occurs, haloscope searches are anticipated to become much more complicated to operate: requiring N times as many measurements but also the new requirement that N detectors function in lock step. To offset this anticipated increase in detector complexity, we aim to develop new tools for diagnosing low temperature RF experiments using neural networks for pattern recognition. Current haloscope experiments monitor the scattering parameters of their RF receiver for periodically measuring cavity quality factor and coupling. However off-resonant data remains relatively useless. In this paper, we ask whether the off resonant information contained in these VNA scans could be used to diagnose equipment failures/anomalies and measure physical conditions (e.g., temperatures and ambient magnetic field strengths). We demonstrate a proof-of-concept that AI techniques can help manage the overall complexity of an axion haloscope search for operators.

hep-ex↗

arXiv2025

Mantis Shrimp: Exploring Photometric Band Utilization in Computer Vision Networks for Photometric Redshift Estimation

We present Mantis Shrimp, a multi-survey deep learning model for photometric redshift estimation that fuses ultra-violet (GALEX), optical (PanSTARRS), and infrared (UnWISE) imagery. Machine learning is now an established approach for photometric redshift estimation, with generally acknowledged higher performance in areas with a high density of spectroscopically identified galaxies over template-based methods. Multiple works have shown that image-based convolutional neural networks can outperform tabular-based color/magnitude models. In comparison to tabular models, image models have additional design complexities: it is largely unknown how to fuse inputs from different instruments which have different resolutions or noise properties. The Mantis Shrimp model estimates the conditional density estimate of redshift using cutout images. The density estimates are well calibrated and the point estimates perform well in the distribution of available spectroscopically confirmed galaxies with (bias = 1e-2), scatter (NMAD = 2.44e-2) and catastrophic outlier rate ($\eta$=17.53$\%$). We find that early fusion approaches (e.g., resampling and stacking images from different instruments) match the performance of late fusion approaches (e.g., concatenating latent space representations), so that the design choice ultimately is left to the user. Finally, we study how the models learn to use information across bands, finding evidence that our models successfully incorporates information from all surveys. The applicability of our model to the analysis of large populations of galaxies is limited by the speed of downloading cutouts from external servers; however, our model could be useful in smaller studies such as generating priors over redshift for stellar population synthesis.

astro-ph.IM↗

arXiv2024

Understanding Generative AI Content with Embedding Models

Constructing high-quality features is critical to any quantitative data analysis. While feature engineering was historically addressed by carefully hand-crafting data representations based on domain expertise, deep neural networks (DNNs) now offer a radically different approach. DNNs implicitly engineer features by transforming their input data into hidden feature vectors called embeddings. For embedding vectors produced by foundation models -- which are trained to be useful across many contexts -- we demonstrate that simple and well-studied dimensionality-reduction techniques such as Principal Component Analysis uncover inherent heterogeneity in input data concordant with human-understandable explanations. Of the many applications for this framework, we find empirical evidence that there is intrinsic separability between real samples and those generated by artificial intelligence (AI).

cs.LG↗

arXiv2024

Preliminary Report on Mantis Shrimp: a Multi-Survey Computer Vision Photometric Redshift Model

The availability of large, public, multi-modal astronomical datasets presents an opportunity to execute novel research that straddles the line between science of AI and science of astronomy. Photometric redshift estimation is a well-established subfield of astronomy. Prior works show that computer vision models typically outperform catalog-based models, but these models face additional complexities when incorporating images from more than one instrument or sensor. In this report, we detail our progress creating Mantis Shrimp, a multi-survey computer vision model for photometric redshift estimation that fuses ultra-violet (GALEX), optical (PanSTARRS), and infrared (UnWISE) imagery. We use deep learning interpretability diagnostics to measure how the model leverages information from the different inputs. We reason about the behavior of the CNNs from the interpretability metrics, specifically framing the result in terms of physically-grounded knowledge of galaxy properties.

astro-ph.IM↗

arXiv2023

Efficient kernel surrogates for neural network-based regression

Despite their immense promise in performing a variety of learning tasks, a theoretical understanding of the limitations of Deep Neural Networks (DNNs) has so far eluded practitioners. This is partly due to the inability to determine the closed forms of the learned functions, making it harder to study their generalization properties on unseen datasets. Recent work has shown that randomly initialized DNNs in the infinite width limit converge to kernel machines relying on a Neural Tangent Kernel (NTK) with known closed form. These results suggest, and experimental evidence corroborates, that empirical kernel machines can also act as surrogates for finite width DNNs. The high computational cost of assembling the full NTK, however, makes this approach infeasible in practice, motivating the need for low-cost approximations. In the current work, we study the performance of the Conjugate Kernel (CK), an efficient approximation to the NTK that has been observed to yield fairly similar results. For the regression problem of smooth functions and logistic regression classification, we show that the CK performance is only marginally worse than that of the NTK and, in certain cases, is shown to be superior. In particular, we establish bounds for the relative test losses, verify them with numerical tests, and identify the regularity of the kernel as the key determinant of performance. In addition to providing a theoretical grounding for using CKs instead of NTKs, our framework suggests a recipe for improving DNN accuracy inexpensively. We present a demonstration of this on the foundation model GPT-2 by comparing its performance on a classification task using a conventional approach and our prescription. We also show how our approach can be used to improve physics-informed operator network training for regression tasks as well as convolutional neural network training for vision classification tasks.

cs.LG↗

arXiv2023

Evaluating Physically Motivated Loss Functions for Photometric Redshift Estimation

Physical constraints have been suggested to make neural network models more generalizable, act scientifically plausible, and be more data-efficient over unconstrained baselines. In this report, we present preliminary work on evaluating the effects of adding soft physical constraints to computer vision neural networks trained to estimate the conditional density of redshift on input galaxy images for the Sloan Digital Sky Survey. We introduce physically motivated soft constraint terms that are not implemented with differential or integral operators. We frame this work as a simple ablation study where the effect of including soft physical constraints is compared to an unconstrained baseline. We compare networks using standard point estimate metrics for photometric redshift estimation, as well as metrics to evaluate how faithful our conditional density estimate represents the probability over the ensemble of our test dataset. We find no evidence that the implemented soft physical constraints are more effective regularizers than augmentation.

astro-ph.IM↗

arXiv2023

Foundation Model's Embedded Representations May Detect Distribution Shift

Sampling biases can cause distribution shifts between train and test datasets for supervised learning tasks, obscuring our ability to understand the generalization capacity of a model. This is especially important considering the wide adoption of pre-trained foundational neural networks -- whose behavior remains poorly understood -- for transfer learning (TL) tasks. We present a case study for TL on the Sentiment140 dataset and show that many pre-trained foundation models encode different representations of Sentiment140's manually curated test set $M$ from the automatically labeled training set $P$, confirming that a distribution shift has occurred. We argue training on $P$ and measuring performance on $M$ is a biased measure of generalization. Experiments on pre-trained GPT-2 show that the features learnable from $P$ do not improve (and in fact hamper) performance on $M$. Linear probes on pre-trained GPT-2's representations are robust and may even outperform overall fine-tuning, implying a fundamental importance for discerning distribution shift in train/test splits for model interpretation.

cs.LG↗

arXiv2023

Exploring Learned Representations of Neural Networks with Principal Component Analysis

Understanding feature representation for deep neural networks (DNNs) remains an open question within the general field of explainable AI. We use principal component analysis (PCA) to study the performance of a k-nearest neighbors classifier (k-NN), nearest class-centers classifier (NCC), and support vector machines on the learned layer-wise representations of a ResNet-18 trained on CIFAR-10. We show that in certain layers, as little as 20% of the intermediate feature-space variance is necessary for high-accuracy classification and that across all layers, the first ~100 PCs completely determine the performance of the k-NN and NCC classifiers. We relate our findings to neural collapse and provide partial evidence for the related phenomenon of intermediate neural collapse. Our preliminary work provides three distinct yet interpretable surrogate models for feature representation with an affine linear model the best performing. We also show that leveraging several surrogate models affords us a clever method to estimate where neural collapse may initially occur within the DNN.

cs.LG↗

arXiv2023

Faithful and Efficient Explanations for Neural Networks via Neural Tangent Kernel Surrogate Models

A recent trend in explainable AI research has focused on surrogate modeling, where neural networks are approximated as simpler ML algorithms such as kernel machines. A second trend has been to utilize kernel functions in various explain-by-example or data attribution tasks. In this work, we combine these two trends to analyze approximate empirical neural tangent kernels (eNTK) for data attribution. Approximation is critical for eNTK analysis due to the high computational cost to compute the eNTK. We define new approximate eNTK and perform novel analysis on how well the resulting kernel machine surrogate models correlate with the underlying neural network. We introduce two new random projection variants of approximate eNTK which allow users to tune the time and memory complexity of their calculation. We conclude that kernel machines using approximate neural tangent kernel as the kernel function are effective surrogate models, with the introduced trace NTK the most consistent performer. Open source software allowing users to efficiently calculate kernel functions in the PyTorch framework is available (https://github.com/pnnl/projection\_ntk).

cs.LG↗

arXiv2022

Spectral Evolution and Invariance in Linear-width Neural Networks

We investigate the spectral properties of linear-width feed-forward neural networks, where the sample size is asymptotically proportional to network width. Empirically, we show that the spectra of weight in this high dimensional regime are invariant when trained by gradient descent for small constant learning rates; we provide a theoretical justification for this observation and prove the invariance of the bulk spectra for both conjugate and neural tangent kernels. We demonstrate similar characteristics when training with stochastic gradient descent with small learning rates. When the learning rate is large, we exhibit the emergence of an outlier whose corresponding eigenvector is aligned with the training data structure. We also show that after adaptive gradient training, where a lower test error and feature learning emerge, both weight and kernel matrices exhibit heavy tail behavior. Simple examples are provided to explain when heavy tails can have better generalizations. We exhibit different spectral properties such as invariant bulk, spike, and heavy-tailed distribution from a two-layer neural network using different training strategies, and then correlate them to the feature learning. Analogous phenomena also appear when we train conventional neural networks with real-world data. We conclude that monitoring the evolution of the spectra during training is an essential step toward understanding the training dynamics and feature learning.

cs.LG↗

arXiv2022

TorchNTK: A Library for Calculation of Neural Tangent Kernels of PyTorch Models

We introduce torchNTK, a python library to calculate the empirical neural tangent kernel (NTK) of neural network models in the PyTorch framework. We provide an efficient method to calculate the NTK of multilayer perceptrons. We compare the explicit differentiation implementation against autodifferentiation implementations, which have the benefit of extending the utility of the library to any architecture supported by PyTorch, such as convolutional networks. A feature of the library is that we expose the user to layerwise NTK components, and show that in some regimes a layerwise calculation is more memory efficient. We conduct preliminary experiments to demonstrate use cases for the software and probe the NTK.

cs.LG↗

arXiv2020

GHOST: Using Only Host Galaxy Information to Accurately Associate and Distinguish Supernovae

We present GHOST, a database of 16,175 spectroscopically classified supernovae and the properties of their host galaxies. We have developed a host galaxy association method using image gradients that achieves fewer misassociations for low-z hosts and higher completeness for high-z hosts than previous methods. We use dimensionality reduction to identify the host galaxy properties that distinguish supernova classes. Our results suggest that the hosts of SLSNe, SNe Ia, and core collapse supernovae can be separated using host brightness information and extendedness measures derived from the host's light profile. Next, we train a random forest model with data from GHOST to predict supernova class using exclusively host galaxy information and the radial offset of the supernova. We can distinguish SNe Ia and core collapse supernovae with ~70% accuracy without any photometric data from the event itself. Vera C. Rubin Observatory will usher in a new era of transient population studies, demanding improved photometric tools for rapid identification and classification of transient events. By identifying the host features with high discriminatory power, we will maintain SN sample purities and continue to identify scientifically relevant events as data volumes increase. The GHOST database and our corresponding software for associating transients with host galaxies are both publicly available.

astro-ph.GA↗

arXiv1998

Simulating Electron Transport and Synchrotron Emission in Radio Galaxies: Shock Acceleration and Synchrotron Aging in Axis-Symmetric Flows

We introduce a simple and economical but effective method for including relativistic electron transport in multi-dimensional simulations of radio galaxies. The method is designed to follow explicitly diffusive acceleration at shocks, and, in smooth flows 2nd order Fermi acceleration plus adiabatic and synchrotron cooling. We are able to follow both the spatial and energy distributions of the electrons, so that direct synchrotron emission properties can be modeled in time-dependent flows for the first time. Here we present first results in the form of some axis-symmetric MHD simulations of Mach 20 light jet flows. These show clearly the importance of nonsteady terminal shocks that develop in such flows even when the jet inflow is steady. As a result of this and other consequences of the fundamentally driven character of jets, we find complex patterns of emissivities and synchrotron spectra, including steep spectral gradients in hot spots, islands of distinct spectra electrons within the lobes and spectral gradients coming from the dynamical histories of a given flow element rather than from synchrotron aging of the embedded electrons. In addition, spectral aging in the lobes tends to proceed more slowly than one would estimate from regions of high emissivity.

astro-ph↗