arXiv subjects

Yuan Cao

Publications and source records attributed to Yuan Cao.

At least 73 records · Page 4Linked to original sources

arXiv2025

Quantifying the Optimization and Generalization Advantages of Graph Neural Networks Over Multilayer Perceptrons

Graph neural networks (GNNs) have demonstrated remarkable capabilities in learning from graph-structured data, often outperforming traditional Multilayer Perceptrons (MLPs) in numerous graph-based tasks. Although existing works have demonstrated the benefits of graph convolution through Laplacian smoothing, expressivity or separability, there remains a lack of quantitative analysis comparing GNNs and MLPs from an optimization and generalization perspective. This study aims to address this gap by examining the role of graph convolution through feature learning theory. Using a signal-noise data model, we conduct a comparative analysis of the optimization and generalization between two-layer graph convolutional networks (GCNs) and their MLP counterparts. Our approach tracks the trajectory of signal learning and noise memorization in GNNs, characterizing their post-training generalization. We reveal that GNNs significantly prioritize signal learning, thus enhancing the regime of {low test error} over MLPs by $D^{q-2}$ times, where $D$ denotes a node's expected degree and $q$ is the power of ReLU activation function with $q>2$. This finding highlights a substantial and quantitative discrepancy between GNNs and MLPs in terms of optimization and generalization, a conclusion further supported by our empirical simulations on both synthetic and real-world datasets.

cs.LG↗

arXiv2025

Gradient Descent Robustly Learns the Intrinsic Dimension of Data in Training Convolutional Neural Networks

Modern neural networks are usually highly over-parameterized. Behind the wide usage of over-parameterized networks is the belief that, if the data are simple, then the trained network will be automatically equivalent to a simple predictor. Following this intuition, many existing works have studied different notions of "ranks" of neural networks and their relation to the rank of data. In this work, we study the rank of convolutional neural networks (CNNs) trained by gradient descent, with a specific focus on the robustness of the rank to image background noises. Specifically, we point out that, when adding background noises to images, the rank of the CNN trained with gradient descent is affected far less compared with the rank of the data. We support our claim with a theoretical case study, where we consider a particular data model to characterize low-rank clean images with added background noises. We prove that CNNs trained by gradient descent can learn the intrinsic dimension of clean images, despite the presence of relatively large background noises. We also conduct experiments on synthetic and real datasets to further validate our claim.

stat.ML↗

arXiv2025

Transformer Learns Optimal Variable Selection in Group-Sparse Classification

Transformers have demonstrated remarkable success across various applications. However, the success of transformers have not been understood in theory. In this work, we give a case study of how transformers can be trained to learn a classic statistical model with "group sparsity", where the input variables form multiple groups, and the label only depends on the variables from one of the groups. We theoretically demonstrate that, a one-layer transformer trained by gradient descent can correctly leverage the attention mechanism to select variables, disregarding irrelevant ones and focusing on those beneficial for classification. We also demonstrate that a well-pretrained one-layer transformer can be adapted to new downstream tasks to achieve good prediction accuracy with a limited number of samples. Our study sheds light on how transformers effectively learn structured data.

stat.ML↗

arXiv2025

Free-Space Twin-Field Quantum Key Distribution

Twin-field quantum key distribution (TF-QKD) elevates the secure key rate from a linear to a square-root dependence on channel loss while preserving measurement-device-independent security. This protocol is uniquely positioned to enable global-scale quantum networks, even under extreme channel loss. While fiber-based TF-QKD implementations have advanced rapidly since its proposal, free-space realizations have remained elusive due to atmospheric turbulence-induced phase distortions. Here, we report the first experimental demonstration of free-space TF-QKD over 14.2 km urban atmospheric channels, surpassing the effective atmospheric thickness -- a critical threshold for satellite compatibility. We achieve a secret key rate exceeding the repeaterless capacity bound, a milestone for practical quantum communication. Our approach eliminates the need for an auxiliary channel to stabilize a closed interferometer, instead leveraging open-channel time and phase control of optical pulses. This work represents a pivotal advance toward satellite-based global quantum networks, combining high-speed key distribution with inherent resistance to real-world channel fluctuations.

quant-ph↗

arXiv2025

On the Feature Learning in Diffusion Models

The predominant success of diffusion models in generative modeling has spurred significant interest in understanding their theoretical foundations. In this work, we propose a feature learning framework aimed at analyzing and comparing the training dynamics of diffusion models with those of traditional classification models. Our theoretical analysis demonstrates that diffusion models, due to the denoising objective, are encouraged to learn more balanced and comprehensive representations of the data. In contrast, neural networks with a similar architecture trained for classification tend to prioritize learning specific patterns in the data, often focusing on easy-to-learn components. To support these theoretical insights, we conduct several experiments on both synthetic and real-world datasets, which empirically validate our findings and highlight the distinct feature learning dynamics in diffusion models compared to classification.

stat.ML↗

arXiv2025

On the Robustness of Transformers against Context Hijacking for Linear Classification

Transformer-based Large Language Models (LLMs) have demonstrated powerful in-context learning capabilities. However, their predictions can be disrupted by factually correct context, a phenomenon known as context hijacking, revealing a significant robustness issue. To understand this phenomenon theoretically, we explore an in-context linear classification problem based on recent advances in linear transformers. In our setup, context tokens are designed as factually correct query-answer pairs, where the queries are similar to the final query but have opposite labels. Then, we develop a general theoretical analysis on the robustness of the linear transformers, which is formulated as a function of the model depth, training context lengths, and number of hijacking context tokens. A key finding is that a well-trained deeper transformer can achieve higher robustness, which aligns with empirical observations. We show that this improvement arises because deeper layers enable more fine-grained optimization steps, effectively mitigating interference from context hijacking. This is also well supported by our numerical experiments. Our findings provide theoretical insights into the benefits of deeper architectures and contribute to enhancing the understanding of transformer architectures.

cs.CL↗

arXiv2025

An unstructured block-based adaptive mesh refinement approach for explicit discontinuous Galerkin method

In the present paper, we present an adaptive mesh refinement(AMR) approach designed for the discontinuous Galerkin method for conservation laws. The block-based AMR is adopted to ensure the local data structure simplicity and the efficiency, while the unstructured topology of the initial blocks is supported by the forest concept such that the complex geometry of the computational domain can be easily treated. The inter-block communication through guardcells is introduced to avoid the direct treatment of flux computing between cells at different refinement levels. The sharp corners and creases generated during direct refinement can be avoided by projecting the boundary nodes to either the user-defined boundary surface function or the auto-generated NURBs. High-level MPI parallelization is implemented with dynamic load balancing through a space curve filling procedure. Some test cases are presented. As a result, ideal accuracy order and versatility in tracing and controlling the dynamic refinement are observed. Also, good parallelization efficiency is demonstrated.

physics.flu-dyn↗

arXiv2025

Beyond Pass or Fail: Multi-Dimensional Benchmarking of Foundation Models for Goal-based Mobile UI Navigation

Recent advances of foundation models (FMs) have made navigating mobile applications (apps) based on high-level goal instructions within reach, with significant industrial applications such as UI testing. While existing benchmarks evaluate FM-based UI navigation using the binary pass/fail metric, they have two major limitations: they cannot reflect the complex nature of mobile UI navigation where FMs may fail for various reasons (e.g., misunderstanding instructions and failed planning), and they lack industrial relevance due to oversimplified tasks that poorly represent real-world scenarios. To address the preceding limitations, we propose Sphinx, a comprehensive benchmark for multi-dimensional evaluation of FMs in industrial settings of UI navigation. Sphinx introduces a specialized toolkit that evaluates five essential FM capabilities, providing detailed insights into failure modes such as insufficient app knowledge or planning issues. Using both popular Google Play applications and WeChat's internal UI test cases, we evaluate 8 FMs with 20 different configurations. Our results show that existing FMs universally struggle with goal-based testing tasks, primarily due to insufficient UI-specific capabilities. We summarize seven lessons learned from benchmarking FMs with Sphinx, providing clear directions for improving FM-based mobile UI navigation.

cs.SE↗

arXiv2025

Transformers versus the EM Algorithm in Multi-class Clustering

LLMs demonstrate significant inference capacities in complicated machine learning tasks, using the Transformer model as its backbone. Motivated by the limited understanding of such models on the unsupervised learning problems, we study the learning guarantees of Transformers in performing multi-class clustering of the Gaussian Mixture Models. We develop a theory drawing strong connections between the Softmax Attention layers and the workflow of the EM algorithm on clustering the mixture of Gaussians. Our theory provides approximation bounds for the Expectation and Maximization steps by proving the universal approximation abilities of multivariate mappings by Softmax functions. In addition to the approximation guarantees, we also show that with a sufficient number of pre-training samples and an initialization, Transformers can achieve the minimax optimal rate for the problem considered. Our extensive simulations empirically verified our theory by revealing the strong learning capacities of Transformers even beyond the assumptions in the theory, shedding light on the powerful inference capacities of LLMs.

stat.ML↗

arXiv2025

Transformers and Their Roles as Time Series Foundation Models

We give a comprehensive analysis of transformers as time series foundation models, focusing on their approximation and generalization capabilities. First, we demonstrate that there exist transformers that fit an autoregressive model on input univariate time series via gradient descent. We then analyze MOIRAI, a multivariate time series foundation model capable of handling an arbitrary number of covariates. We prove that it is capable of automatically fitting autoregressive models with an arbitrary number of covariates, offering insights into its design and empirical success. For generalization, we establish bounds for pretraining when the data satisfies Dobrushin's condition. Experiments support our theoretical findings, highlighting the efficacy of transformers as time series foundation models.

cs.LG↗

arXiv2025

Learning Spectral Methods by Transformers

Transformers demonstrate significant advantages as the building block of modern LLMs. In this work, we study the capacities of Transformers in performing unsupervised learning. We show that multi-layered Transformers, given a sufficiently large set of pre-training instances, are able to learn the algorithms themselves and perform statistical estimation tasks given new instances. This learning paradigm is distinct from the in-context learning setup and is similar to the learning procedure of human brains where skills are learned through past experience. Theoretically, we prove that pre-trained Transformers can learn the spectral methods and use the classification of bi-class Gaussian mixture model as an example. Our proof is constructive using algorithmic design techniques. Our results are built upon the similarities of multi-layered Transformer architecture with the iterative recovery algorithms used in practice. Empirically, we verify the strong capacity of the multi-layered (pre-trained) Transformer on unsupervised learning through the lens of both the PCA and the Clustering tasks performed on the synthetic and real-world datasets.

stat.ML↗

arXiv2025

An Infrastructure Software Perspective Toward Computation Offloading between Executable Specifications and Foundation Models

Foundation Models (FMs) have become essential components in modern software systems, excelling in tasks such as pattern recognition and unstructured data processing. However, their capabilities are complemented by the precision, verifiability, and deterministic nature of executable specifications, such as symbolic programs. This paper explores a new perspective on computation offloading, proposing a framework that strategically distributes computational tasks between FMs and executable specifications based on their respective strengths. We discuss the potential design of an infrastructure software framework to enable this offloading, focusing on key mechanisms such as task decomposition, resource allocation, and adaptive optimization. Furthermore, we identify critical technical challenges, including semantic-gap resolution, reliability, and scalability, that must be addressed to realize this approach. By leveraging the complementary strengths of FMs and symbolic programs, this perspective lays the groundwork for advancing hybrid software systems that are both efficient and reliable.

cs.SE↗

arXiv2024

Geometrical Distances of Extragalactic Binaries through Spectroastrometry

The growing ``Hubble tension'' has prompted the need for precise measurements of cosmological distances. This paper demonstrates a purely geometric approach for determining the distance to extragalactic binaries through a joint analysis of spectroastrometry (SA), radial velocity (RV), and light curve (LC) observations. A parameterized model for the binary system is outlined, and simulated SA, RV, and LC data are computed to infer the probability distribution of model parameters based on the mock data. The impact of data quality and binary parameters on distance uncertainties is comprehensively analyzed, showcasing the method's potential for high-precision distance measurements. For a typical eclipsing binary in the Large Magellanic Cloud (LMC), the distance uncertainty is approximately 6% under reasonable observational conditions. Within a specific range of data quality and input parameters, the distance measurement precision of individual binary star systems is generally better than 10%. As a geometric method based on the simplest dynamics, it is independent of empirical calibration and the systematics caused by model selections can be tested using nearby binaries with known distances. By measuring multiple binary star systems or monitoring one binary system repeatedly, geometric distance measurements of nearby galaxies can be achieved, providing valuable insights into the Hubble tension and advancing our understanding of the universe's structure and evolution.

astro-ph.CO↗

arXiv2024

One-Layer Transformer Provably Learns One-Nearest Neighbor In Context

Transformers have achieved great success in recent years. Interestingly, transformers have shown particularly strong in-context learning capability -- even without fine-tuning, they are still able to solve unseen tasks well purely based on task-specific prompts. In this paper, we study the capability of one-layer transformers in learning one of the most classical nonparametric estimators, the one-nearest neighbor prediction rule. Under a theoretical framework where the prompt contains a sequence of labeled training data and unlabeled test data, we show that, although the loss function is nonconvex when trained with gradient descent, a single softmax attention layer can successfully learn to behave like a one-nearest neighbor classifier. Our result gives a concrete example of how transformers can be trained to implement nonparametric machine learning algorithms, and sheds light on the role of softmax attention in transformer models.

cs.LG↗

arXiv2024

On the Comparison between Multi-modal and Single-modal Contrastive Learning

Multi-modal contrastive learning with language supervision has presented a paradigm shift in modern machine learning. By pre-training on a web-scale dataset, multi-modal contrastive learning can learn high-quality representations that exhibit impressive robustness and transferability. Despite its empirical success, the theoretical understanding is still in its infancy, especially regarding its comparison with single-modal contrastive learning. In this work, we introduce a feature learning theory framework that provides a theoretical foundation for understanding the differences between multi-modal and single-modal contrastive learning. Based on a data generation model consisting of signal and noise, our analysis is performed on a ReLU network trained with the InfoMax objective function. Through a trajectory-based optimization analysis and generalization characterization on downstream tasks, we identify the critical factor, which is the signal-to-noise ratio (SNR), that impacts the generalizability in downstream tasks of both multi-modal and single-modal contrastive learning. Through the cooperation between the two modalities, multi-modal learning can achieve better feature learning, leading to improvements in performance in downstream tasks compared to single-modal learning. Our analysis provides a unified framework that can characterize the optimization and generalization of both single-modal and multi-modal contrastive learning. Empirical experiments on both synthetic and real-world datasets further consolidate our theoretical findings.

cs.LG↗

arXiv2024

Global Convergence in Training Large-Scale Transformers

Despite the widespread success of Transformers across various domains, their optimization guarantees in large-scale model settings are not well-understood. This paper rigorously analyzes the convergence properties of gradient flow in training Transformers with weight decay regularization. First, we construct the mean-field limit of large-scale Transformers, showing that as the model width and depth go to infinity, gradient flow converges to the Wasserstein gradient flow, which is represented by a partial differential equation. Then, we demonstrate that the gradient flow reaches a global minimum consistent with the PDE solution when the weight decay regularization parameter is sufficiently small. Our analysis is based on a series of novel mean-field techniques that adapt to Transformers. Compared with existing tools for deep networks (Lu et al., 2020) that demand homogeneity and global Lipschitz smoothness, we utilize a refined analysis assuming only $\textit{partial homogeneity}$ and $\textit{local Lipschitz smoothness}$. These new techniques may be of independent interest.

stat.ML↗

arXiv2024

Initialization Matters: On the Benign Overfitting of Two-Layer ReLU CNN with Fully Trainable Layers

Benign overfitting refers to how over-parameterized neural networks can fit training data perfectly and generalize well to unseen data. While this has been widely investigated theoretically, existing works are limited to two-layer networks with fixed output layers, where only the hidden weights are trained. We extend the analysis to two-layer ReLU convolutional neural networks (CNNs) with fully trainable layers, which is closer to the practice. Our results show that the initialization scaling of the output layer is crucial to the training dynamics: large scales make the model training behave similarly to that with the fixed output, the hidden layer grows rapidly while the output layer remains largely unchanged; in contrast, small scales result in more complex layer interactions, the hidden layer initially grows to a specific ratio relative to the output layer, after which both layers jointly grow and maintain that ratio throughout training. Furthermore, in both settings, we provide nearly matching upper and lower bounds on the test errors, identifying the sharp conditions on the initialization scaling and signal-to-noise ratio (SNR) in which the benign overfitting can be achieved or not. Numerical experiments back up the theoretical results.

cs.LG↗

arXiv2024

Control of chirality and directionality of nonlinear metasurface light source via twisting

Metasurfaces have revolutionized nonlinear and quantum light manipulation in the past decade, enabling the design of materials that can tune polarization, frequency, and direction of light simultaneously. However, tuning of metasurfaces is traditionally achieved by changing their microscopic structure, which does not allow \emph{in situ} tuning and dynamic optimization of the metasurfaces. In this Letter, we explore the concept of twisted bilayer and tetralayer nonlinear metasurfaces, which offer rich tunability in its effective nonlinear susceptibilities. Using gold-based metasurfaces, we demonstrate that a number of different singularities of nonlinear susceptibilities can exist in the parameter space of twist angle and interlayer gap between different twisted layers. At the singularities, reflected/transmitted light from the nonlinear process (such as second-harmonic generation) can either become circularly polarized (for C points), or entirely vanish (for V points). By further breaking symmetries of the system, we can independently tune all aspects of the reflected and transmitted nonlinear emission, achieving unidirectional emission with full-Poincaré polarization tunability, a dark state (V-V point), or any other bidirectional output. Our work paves the way for multidimensional control of polarization and directionality in nonlinear light sources, opening new avenues in ultrafast optics, optical communication, sensing, and quantum optics.

physics.optics↗