arXiv subjects

Zhiyuan Li

Publications and source records attributed to Zhiyuan Li.

At least 271 records · Page 15Linked to original sources

arXiv2023

A Novel Collaborative Self-Supervised Learning Method for Radiomic Data

The computer-aided disease diagnosis from radiomic data is important in many medical applications. However, developing such a technique relies on annotating radiological images, which is a time-consuming, labor-intensive, and expensive process. In this work, we present the first novel collaborative self-supervised learning method to solve the challenge of insufficient labeled radiomic data, whose characteristics are different from text and image data. To achieve this, we present two collaborative pretext tasks that explore the latent pathological or biological relationships between regions of interest and the similarity and dissimilarity information between subjects. Our method collaboratively learns the robust latent feature representations from radiomic data in a self-supervised manner to reduce human annotation efforts, which benefits the disease diagnosis. We compared our proposed method with other state-of-the-art self-supervised learning methods on a simulation study and two independent datasets. Extensive experimental results demonstrated that our method outperforms other self-supervised learning methods on both classification and regression tasks. With further refinement, our method shows the potential advantage in automatic disease diagnosis with large-scale unlabeled data available.

eess.IV↗

arXiv2023

Error per single-qubit gate below $10^{-4}$ in a superconducting qubit

Implementing arbitrary single-qubit gates with near perfect fidelity is among the most fundamental requirements in gate-based quantum information processing. In this work, we fabric a transmon qubit with long coherence times and demonstrate single-qubit gates with the average gate error below $10^{-4}$, i.e. $(7.42\pm0.04)\times10^{-5}$ by randomized benchmarking (RB). To understand the error sources, we experimentally obtain an error budget, consisting of the decoherence errors lower bounded by $(4.62\pm0.04)\times10^{-5}$ and the leakage rate per gate of $(1.16\pm0.04)\times10^{-5}$. Moreover, we reconstruct the process matrices for the single-qubit gates by the gate set tomography (GST), with which we simulate RB sequences and obtain single-qubit fedlities consistent with experimental results. We also observe non-Markovian behavior in the experiment of long-sequence GST, which may provide guidance for further calibration. The demonstration extends the upper limit that the average fidelity of single-qubit gates can reach in a transmon-qubit system, and thus can be an essential step towards practical and reliable quantum computation in the near future.

quant-ph↗

arXiv2023

Understanding Incremental Learning of Gradient Descent: A Fine-grained Analysis of Matrix Sensing

It is believed that Gradient Descent (GD) induces an implicit bias towards good generalization in training machine learning models. This paper provides a fine-grained analysis of the dynamics of GD for the matrix sensing problem, whose goal is to recover a low-rank ground-truth matrix from near-isotropic linear measurements. It is shown that GD with small initialization behaves similarly to the greedy low-rank learning heuristics (Li et al., 2020) and follows an incremental learning procedure (Gissin et al., 2019): GD sequentially learns solutions with increasing ranks until it recovers the ground truth matrix. Compared to existing works which only analyze the first learning phase for rank-1 solutions, our result provides characterizations for the whole learning process. Moreover, besides the over-parameterized regime that many prior works focused on, our analysis of the incremental learning procedure also applies to the under-parameterized regime. Finally, we conduct numerical experiments to confirm our theoretical findings.

cs.LG↗

arXiv2023

Understanding the Generalization Benefit of Normalization Layers: Sharpness Reduction

Normalization layers (e.g., Batch Normalization, Layer Normalization) were introduced to help with optimization difficulties in very deep nets, but they clearly also help generalization, even in not-so-deep nets. Motivated by the long-held belief that flatter minima lead to better generalization, this paper gives mathematical analysis and supporting experiments suggesting that normalization (together with accompanying weight-decay) encourages GD to reduce the sharpness of loss surface. Here "sharpness" is carefully defined given that the loss is scale-invariant, a known consequence of normalization. Specifically, for a fairly broad class of neural nets with normalization, our theory explains how GD with a finite learning rate enters the so-called Edge of Stability (EoS) regime, and characterizes the trajectory of GD in this regime via a continuous sharpness-reduction flow.

cs.LG↗

arXiv2023

How Does Sharpness-Aware Minimization Minimize Sharpness?

Sharpness-Aware Minimization (SAM) is a highly effective regularization technique for improving the generalization of deep neural networks for various settings. However, the underlying working of SAM remains elusive because of various intriguing approximations in the theoretical characterizations. SAM intends to penalize a notion of sharpness of the model but implements a computationally efficient variant; moreover, a third notion of sharpness was used for proving generalization guarantees. The subtle differences in these notions of sharpness can indeed lead to significantly different empirical results. This paper rigorously nails down the exact sharpness notion that SAM regularizes and clarifies the underlying mechanism. We also show that the two steps of approximations in the original motivation of SAM individually lead to inaccurate local conclusions, but their combination accidentally reveals the correct effect, when full-batch gradients are applied. Furthermore, we also prove that the stochastic version of SAM in fact regularizes the third notion of sharpness mentioned above, which is most likely to be the preferred notion for practical performance. The key mechanism behind this intriguing phenomenon is the alignment between the gradient and the top eigenvector of Hessian when SAM is applied.

cs.LG↗

arXiv2022

Well-posedness of the stochastic time-fractional diffusion and wave equations and inverse random source problems

In this paper, we are concerned with the stochastic time-fractional diffusion-wave equations in a Hilbert space. The main objective of this paper is to establish properties of the stochastic weak solutions of the initial-boundary value problem, such as the existence, uniqueness and regularity estimates. Moreover, we apply the obtained theories to an inverse source problem. The uniqueness of this inverse problem under the boundary measurements is proved.

math.AP↗

arXiv2022

NOEMA Detection of Circumnuclear Molecular Gas in X-ray Weak Dual Active Galactic Nuclei: No Evidence for Heavy Obscuration

Dual active galactic nuclei (AGN), which are the manifestation of two actively accreting supermassive black holes (SMBHs) hosted by a pair of merging galaxies, are a unique laboratory for studying the physics of SMBH feeding and feedback during an indispensable stage of galaxy evolution. In this work, we present NOEMA CO(2-1) observations of seven kpc-scale dual-AGN candidates drawn from a recent Chandra survey of low-redshift, optically classified AGN pairs. These systems are selected because they show unexpectedly low 2-10 keV X-ray luminosities for their small physical separations signifying an intermediate-to-late stage of merger. Circumnuclear molecular gas traced by the CO(2-1) emission is significantly detected in 6 of the 7 pairs and 10 of the 14 nuclei, with an estimated mass ranging between $(0.2 - 21) \times10^9\rm~M_{\odot}$. The primary nuclei, i.e., the ones with the higher stellar velocity dispersion, tend to have a higher molecular gas mass than the secondary. Most CO-detected nuclei show a compact morphology, with a velocity field consistent with a kpc-scale rotating structure. The inferred hydrogen column densities range between $5\times10^{21} - 2\times10^{23}\rm~cm^{-2}$, but mostly at a few times $10^{22}\rm~cm^{-2}$, in broad agreement with those derived from X-ray spectral analysis. Together with the relatively weak mid-infrared emission, the moderate column density argues against the prevalence of heavily obscured, intrinsically luminous AGNs in these seven systems, but favors a feedback scenario in which AGN activity triggered by a recent pericentric passage of the galaxy pair can expel circumnuclear gas and suppress further SMBH accretion.

astro-ph.GA↗

arXiv2022

Interpretable Modeling and Reduction of Unknown Errors in Mechanistic Operators

Prior knowledge about the imaging physics provides a mechanistic forward operator that plays an important role in image reconstruction, although myriad sources of possible errors in the operator could negatively impact the reconstruction solutions. In this work, we propose to embed the traditional mechanistic forward operator inside a neural function, and focus on modeling and correcting its unknown errors in an interpretable manner. This is achieved by a conditional generative model that transforms a given mechanistic operator with unknown errors, arising from a latent space of self-organizing clusters of potential sources of error generation. Once learned, the generative model can be used in place of a fixed forward operator in any traditional optimization-based reconstruction process where, together with the inverse solution, the error in prior mechanistic forward operator can be minimized and the potential source of error uncovered. We apply the presented method to the reconstruction of heart electrical potential from body surface potential. In controlled simulation experiments and in-vivo real data experiments, we demonstrate that the presented method allowed reduction of errors in the physics-based forward operator and thereby delivered inverse reconstruction of heart-surface potential with increased accuracy.

eess.IV↗

arXiv2022

Understanding Gradient Descent on Edge of Stability in Deep Learning

Deep learning experiments by Cohen et al. [2021] using deterministic Gradient Descent (GD) revealed an Edge of Stability (EoS) phase when learning rate (LR) and sharpness (i.e., the largest eigenvalue of Hessian) no longer behave as in traditional optimization. Sharpness stabilizes around $2/$LR and loss goes up and down across iterations, yet still with an overall downward trend. The current paper mathematically analyzes a new mechanism of implicit regularization in the EoS phase, whereby GD updates due to non-smooth loss landscape turn out to evolve along some deterministic flow on the manifold of minimum loss. This is in contrast to many previous results about implicit bias either relying on infinitesimal updates or noise in gradient. Formally, for any smooth function $L$ with certain regularity condition, this effect is demonstrated for (1) Normalized GD, i.e., GD with a varying LR $η_t =\fracη{\| \nabla L(x(t)) \|}$ and loss $L$; (2) GD with constant LR and loss $\sqrt{L- \min_x L(x)}$. Both provably enter the Edge of Stability, with the associated flow on the manifold minimizing $λ_{1}(\nabla^2 L)$. The above theoretical results have been corroborated by an experimental study.

cs.LG↗

arXiv2022

Same Pre-training Loss, Better Downstream: Implicit Bias Matters for Language Models

Language modeling on large-scale datasets leads to impressive performance gains on various downstream language tasks. The validation pre-training loss (or perplexity in autoregressive language modeling) is often used as the evaluation metric when developing language models since the pre-training loss tends to be well-correlated with downstream performance (which is itself difficult to evaluate comprehensively). Contrary to this conventional wisdom, this paper shows that 1) pre-training loss cannot fully explain downstream performance and 2) flatness of the model is well-correlated with downstream performance where pre-training loss is not. On simplified datasets, we identify three ways to produce models with the same (statistically optimal) pre-training loss but different downstream performance: continue pre-training after convergence, increasing the model size, and changing the training algorithm. These experiments demonstrate the existence of implicit bias of pre-training algorithms/optimizers -- among models with the same minimal pre-training loss, they implicitly prefer more transferable ones. Toward understanding this implicit bias, we prove that SGD with standard mini-batch noise implicitly prefers flatter minima in language models, and empirically observe a strong correlation between flatness and downstream performance among models with the same minimal pre-training loss. We also prove in a synthetic language setting that among the models with the minimal pre-training loss, the flattest model transfers to downstream tasks.

cs.LG↗

arXiv2022

A note on Fourier-Mukai partners of abelian varieties over positive characteristic fields

Over complex numbers, the Fourier-Mukai partners of abelian varieties are well-understood. A celebrated result is Orlov's derived Torelli theorem. In this note, we study the FM-partners of abelian varieties in positive characteristic. We notice that, in odd characteristics, two abelian varieties of odd dimension are derived equivalent if their associated Kummer stacks are derived equivalent, which is Krug and Sosna's result over complex numbers. For abelian surfaces in odd characteristic, we show that two abelian surfaces are derived equivalent if and only if their associated Kummer surfaces are isomorphic. This extends the result [doi:10.1215/s0012-7094-03-12036-0] to odd characteristic fields, which solved a classical problem originally from Shioda. Furthermore, we establish the derived Torelli theorem for supersingular abelian varieties and apply it to characterize the quasi-liftable birational models of supersingular generalized Kummer varieties.

math.AG↗

arXiv2022

Few-shot Generation of Personalized Neural Surrogates for Cardiac Simulation via Bayesian Meta-Learning

Clinical adoption of personalized virtual heart simulations faces challenges in model personalization and expensive computation. While an ideal solution is an efficient neural surrogate that at the same time is personalized to an individual subject, the state-of-the-art is either concerned with personalizing an expensive simulation model, or learning an efficient yet generic surrogate. This paper presents a completely new concept to achieve personalized neural surrogates in a single coherent framework of meta-learning (metaPNS). Instead of learning a single neural surrogate, we pursue the process of learning a personalized neural surrogate using a small amount of context data from a subject, in a novel formulation of few-shot generative modeling underpinned by: 1) a set-conditioned neural surrogate for cardiac simulation that, conditioned on subject-specific context data, learns to generate query simulations not included in the context set, and 2) a meta-model of amortized variational inference that learns to condition the neural surrogate via simple feed-forward embedding of context data. As test time, metaPNS delivers a personalized neural surrogate by fast feed-forward embedding of a small and flexible number of data available from an individual, achieving -- for the first time -- personalization and surrogate construction for expensive simulations in one end-to-end learning framework. Synthetic and real-data experiments demonstrated that metaPNS was able to improve personalization and predictive accuracy in comparison to conventionally-optimized cardiac simulation models, at a fraction of computation.

cs.LG↗

arXiv2022

Implicit Regularization and Convergence for Weight Normalization

Normalization methods such as batch [Ioffe and Szegedy, 2015], weight [Salimansand Kingma, 2016], instance [Ulyanov et al., 2016], and layer normalization [Baet al., 2016] have been widely used in modern machine learning. Here, we study the weight normalization (WN) method [Salimans and Kingma, 2016] and a variant called reparametrized projected gradient descent (rPGD) for overparametrized least-squares regression. WN and rPGD reparametrize the weights with a scale g and a unit vector w and thus the objective function becomes non-convex. We show that this non-convex formulation has beneficial regularization effects compared to gradient descent on the original objective. These methods adaptively regularize the weights and converge close to the minimum l2 norm solution, even for initializations far from zero. For certain stepsizes of g and w , we show that they can converge close to the minimum norm solution. This is different from the behavior of gradient descent, which converges to the minimum norm solution only when started at a point in the range space of the feature matrix, and is thus more sensitive to initialization.

cs.LG↗

arXiv2022

A Chandra Survey of Milky Way Globular Clusters. III. Searching for X-ray Signature of Intermediate-mass Black Holes

Globular clusters (GCs) are thought to harbor the long-sought population of intermediate-mass black holes (IMBHs). We present a systematic search for a putative IMBH in 81 Milky Way GCs, based on archival Chandra X-ray observations. We find in only six GCs a significant X-ray source positionally coincident with the cluster center, which have 0.5-8 keV luminosities between $\sim1\times 10^{30}~{\rm erg~s^{-1}}$ to $\sim 4\times10^{33}~{\rm erg~s^{-1}}$. However, the spectral and temporal properties of these six sources can also be explained in terms of binary stars. The remaining 75 GCs do not have a detectable central source, most with $3σ$ upper limits ranging between $10^{29-32}~{\rm erg~s^{-1}}$ over 0.5-8 keV, which are significantly lower than predicted for canonical Bondi accretion. To help understand the feeble X-ray signature, we perform hydrodynamic simulations of stellar wind accretion onto a $1000~{\rm M_\odot}$ IMBH from the most-bound orbiting star, for stellar wind properties consistent with either a main-sequence (MS) star or an asymptotic giant branch (AGB) star. We find that the synthetic X-ray luminosity for the MS case ($\sim 10^{19}\rm~erg~s^{-1}$) is far below the current X-ray limits. The predicted X-ray luminosity for the AGB case ($\sim 10^{34}\rm~erg~s^{-1}$), on the other hand, is compatible with the detected central X-ray sources, in particular the ones in Terzan 5 and NGC 6652. However, the probability of having an AGB star as the most-bound star around the putative IMBH is very low. Our study strongly suggests that it is very challenging to detect the accretion-induced X-ray emission from IMBHs, even if they were prevalent in present-day GCs.

astro-ph.HE↗

arXiv2022

Mukai's program for non-primitive curves on K3 surfaces

Mukai's program seeks to recover a K3 surface $X$ from any curve $C$ on it by exhibiting it as a Fourier-Mukai partner to a Brill-Noether locus of vector bundles on the curve. In the case $X$ has Picard number one and the curve $C\in |H|$ is primitive, this was confirmed by Feyzbakhsh for $g\geq 11$ and $g\neq 12$. More recently, Feyzbakhsh has shown that certain moduli spaces of stable bundles on $X$ are isomorphic to the Brill-Noether locus of curves in $|H|$ if $g$ is sufficiently large. In this paper, we work with irreducible curves in a non-primitive ample linear system $|mH|$ and prove that Mukai's program is valid for any irreducible curve when $g\neq 2$, $mg\geq 11$ and $mg\neq 12$. Furthermore, we introduce the destabilising regions to improve Feyzbakhsh's analysis. We show that there are hyper-Kähler varieties as Brill-Noether loci of curves in every dimension.

math.AG↗

arXiv2022

A Novel Ontology-guided Attribute Partitioning Ensemble Learning Model for Early Prediction of Cognitive Deficits using Quantitative Structural MRI in Very Preterm Infants

Structural magnetic resonance imaging studies have shown that brain anatomical abnormalities are associated with cognitive deficits in preterm infants. Brain maturation and geometric features can be used with machine learning models for predicting later neurodevelopmental deficits. However, traditional machine learning models would suffer from a large feature-to-instance ratio (i.e., a large number of features but a small number of instances/samples). Ensemble learning is a paradigm that strategically generates and integrates a library of machine learning classifiers and has been successfully used on a wide variety of predictive modeling problems to boost model performance. Attribute (i.e., feature) bagging method is the most commonly used feature partitioning scheme, which randomly and repeatedly draws feature subsets from the entire feature set. Although attribute bagging method can effectively reduce feature dimensionality to handle the large feature-to-instance ratio, it lacks consideration of domain knowledge and latent relationship among features. In this study, we proposed a novel Ontology-guided Attribute Partitioning (OAP) method to better draw feature subsets by considering the domain-specific relationship among features. With the better partitioned feature subsets, we developed an ensemble learning framework, which is referred to as OAP-Ensemble Learning (OAP-EL). We applied the OAP-EL to predict cognitive deficits at 2 years of age using quantitative brain maturation and geometric features obtained at term equivalent age in very preterm infants. We demonstrated that the proposed OAP-EL approach significantly outperformed the peer ensemble learning and traditional machine learning approaches.

cs.LG↗

arXiv2022

What Happens after SGD Reaches Zero Loss? --A Mathematical Framework

Understanding the implicit bias of Stochastic Gradient Descent (SGD) is one of the key challenges in deep learning, especially for overparametrized models, where the local minimizers of the loss function $L$ can form a manifold. Intuitively, with a sufficiently small learning rate $η$, SGD tracks Gradient Descent (GD) until it gets close to such manifold, where the gradient noise prevents further convergence. In such a regime, Blanc et al. (2020) proved that SGD with label noise locally decreases a regularizer-like term, the sharpness of loss, $\mathrm{tr}[\nabla^2 L]$. The current paper gives a general framework for such analysis by adapting ideas from Katzenberger (1991). It allows in principle a complete characterization for the regularization effect of SGD around such manifold -- i.e., the "implicit bias" -- using a stochastic differential equation (SDE) describing the limiting dynamics of the parameters, which is determined jointly by the loss function and the noise covariance. This yields some new results: (1) a global analysis of the implicit bias valid for $η^{-2}$ steps, in contrast to the local analysis of Blanc et al. (2020) that is only valid for $η^{-1.6}$ steps and (2) allowing arbitrary noise covariance. As an application, we show with arbitrary large initialization, label noise SGD can always escape the kernel regime and only requires $O(κ\ln d)$ samples for learning an $κ$-sparse overparametrized linear model in $\mathbb{R}^d$ (Woodworth et al., 2020), while GD initialized in the kernel regime requires $Ω(d)$ samples. This upper bound is minimax optimal and improves the previous $\tilde{O}(κ^2)$ upper bound (HaoChen et al., 2020).

cs.LG↗

arXiv2022

Robust Training of Neural Networks Using Scale Invariant Architectures

In contrast to SGD, adaptive gradient methods like Adam allow robust training of modern deep networks, especially large language models. However, the use of adaptivity not only comes at the cost of extra memory but also raises the fundamental question: can non-adaptive methods like SGD enjoy similar benefits? In this paper, we provide an affirmative answer to this question by proposing to achieve both robust and memory-efficient training via the following general recipe: (1) modify the architecture and make it scale invariant, i.e. the scale of parameter doesn't affect the output of the network, (2) train with SGD and weight decay, and optionally (3) clip the global gradient norm proportional to weight norm multiplied by $\sqrt{\tfrac{2λ}η}$, where $η$ is learning rate and $λ$ is weight decay. We show that this general approach is robust to rescaling of parameter and loss by proving that its convergence only depends logarithmically on the scale of initialization and loss, whereas the standard SGD might not even converge for many initializations. Following our recipe, we design a scale invariant version of BERT, called SIBERT, which when trained simply by vanilla SGD achieves performance comparable to BERT trained by adaptive methods like Adam on downstream tasks.

cs.LG↗