arXiv subjects

Yongqiang Cai

Publications and source records attributed to Yongqiang Cai.

15 recordsLinked to original sources

arXiv2026

FlashPDE: A Drop-In Fused Triton Operator Library for Neural PDE Solvers

Physics-Informed Neural Networks (PINNs) solve PDEs by incorporating physical constraints into neural-network training, but large-scale problems are limited by automatic-differentiation memory overhead and inefficient execution of grid-based PDE operators. We present FlashPDE, a drop-in fused operator library for grid-based scientific machine learning. FlashPDE replaces fragmented PyTorch finite-difference execution with differentiable Triton kernels. Each operator integrates fused stencil evaluation, an analytic discrete-adjoint backward pass, and boundary-gradient correction within a unified PyTorch autograd Function interface. The library provides 14 differentiable PDE operators covering 17 configurations across 1D--3D elliptic, parabolic, and Navier--Stokes systems, while remaining independent of neural architectures and training strategies. Experiments on an NVIDIA A100 GPU show that FlashPDE reduces peak memory usage by up to 37.0x compared with coordinate-based automatic differentiation and reduces CUDA kernel launches by up to 3.5x compared with eager PyTorch finite-difference implementations. Across six representative PDE benchmarks, FlashPDE achieves up to 2.30x end-to-end time-to-solution speedup and up to 19.2x kernel-level acceleration while maintaining numerical agreement with PyTorch finite-difference references. FlashPDE provides a hardware-efficient execution layer that bridges differentiable PDE solvers and GPU-optimized numerical computation within the PyTorch ecosystem.

cs.LG↗

arXiv2025

Vocabulary In-Context Learning in Transformers: Benefits of Positional Encoding

Numerous studies have demonstrated that the Transformer architecture possesses the capability for in-context learning (ICL). In scenarios involving function approximation, context can serve as a control parameter for the model, endowing it with the universal approximation property (UAP). In practice, context is represented by tokens from a finite set, referred to as a vocabulary, which is the case considered in this paper, \emph{i.e.}, vocabulary in-context learning (VICL). We demonstrate that VICL in single-layer Transformers, without positional encoding, does not possess the UAP; however, it is possible to achieve the UAP when positional encoding is included. Several sufficient conditions for the positional encoding are provided. Our findings reveal the benefits of positional encoding from an approximation theory perspective in the context of ICL.

cs.LG↗

arXiv2025

Achieving Universal Approximation and Universal Interpolation via Nonlinearity of Control Families

A significant connection exists between the controllability of dynamical systems and the approximation capabilities of neural networks, where residual networks and vanilla feedforward neural networks can both be regarded as numerical discretizations of the flow maps of dynamical systems. Leveraging the expressive power of neural networks, prior works have explored various control families $\mathcal{F}$ that enable the flow maps of dynamical systems to achieve either the universal approximation property (UAP) or the universal interpolation property (UIP). For example, the control family $\mathcal{F}_\text{ass}({\mathrm{ReLU}})$, consisting of affine maps together with a specific nonlinear function ReLU, achieves UAP; while the affine-invariant nonlinear control family $\mathcal{F}_{\mathrm{aff}}(f)$ containing a nonlinear function $f$ achieves UIP. However, UAP and UIP are generally not equivalent, and thus typically need to be studied separately with different techniques. In this paper, we investigate more general control families, including $\mathcal{F}_\text{ass}(f)$ with nonlinear functions $f$ beyond ReLU, the diagonal affine-invariant family $\mathcal{F}_{\mathrm{diag}}(f)$, and UAP for orientation-preserving diffeomorphisms under the uniform norm. We show that in certain special cases, UAP and UIP are indeed equivalent, whereas in the general case, we introduce the notion of local UIP (a substantially weaker version of UIP) and prove that the combination of UAP and local UIP implies UIP. In particular, the control family $\mathcal{F}_\text{ass}({\mathrm{ReLU}})$ achieves the UIP.

math.OC↗

arXiv2025

Characterization of phospholipid-cholesterol bilayers as self-assembled amphiphile block polymers that contain headgroups

Cholesterol is known to modulate the structure and function of biological membranes. In this study, we use self-consistent field theory (SCFT) to investigate phospholipid/cholesterol bilayer membranes modeled with two types of diblock copolymers. These copolymer-based bilayers serve as biomimetic platforms with applications in areas such as drug delivery. Our simulations identify a minimum free energy configuration characterized by phospholipid tails tilted relative to the membrane normal. The model quantitatively captures the well-known area condensation effect as cholesterol concentration increases, along with membrane thickening and reduced tilt angle. Thermodynamically, we observe a linear dependence between cholesterol's chemical potential and its concentration within the 37-50% range, consistent with experimental results. Additionally, we analyze the effects of block copolymer length and headgroup interactions on bilayer structure. Interactions between phospholipid headgroups and the solvent emerge as the most influential. This work provides a theoretical framework for understanding cholesterol's regulatory role in membrane structure and mechanics.

cond-mat.soft↗

arXiv2025

A Minimal Control Family of Dynamical Systems for Universal Approximation

The universal approximation property (UAP) holds a fundamental position in deep learning, as it provides a theoretical foundation for the expressive power of neural networks. It is widely recognized that a composition of linear and nonlinear functions, such as the rectified linear unit (ReLU) activation function, can approximate continuous functions on compact domains. In this paper, we extend this efficacy to a scenario containing dynamical systems with controls. We prove that the control family $\mathcal{F}_1$ containing all affine maps and the nonlinear ReLU map is sufficient for generating flow maps that can approximate orientation-preserving (OP) diffeomorphisms on any compact domain. Since $\mathcal{F}_1$ contains only one nonlinear function and the UAP does not hold if we remove the nonlinear function, we call $\mathcal{F}_1$ a minimal control family for the UAP. On this basis, several mild sufficient conditions, such as affine invariance, are established for the control family and discussed. Our results reveal an underlying connection between the approximation power of neural networks and control systems and could provide theoretical guidance for examining the approximation power of flow-based models.

eess.SY↗

arXiv2025

Retrieval Backward Attention without Additional Training: Enhance Embeddings of Large Language Models via Repetition

Language models can be viewed as functions that embed text into Euclidean space, where the quality of the embedding vectors directly determines model performance, training such neural networks involves various uncertainties. This paper focuses on improving the performance of pre-trained language models in zero-shot settings through a simple and easily implementable method. We propose a novel backward attention mechanism to enhance contextual information encoding. Evaluated on the Chinese Massive Text Embedding Benchmark (C-MTEB), our approach achieves significant improvements across multiple tasks, providing valuable insights for advancing zero-shot learning capabilities.

cs.CL↗

arXiv2025

Neural Networks Trained by Weight Permutation are Universal Approximators

The universal approximation property is fundamental to the success of neural networks, and has traditionally been achieved by training networks without any constraints on their parameters. However, recent experimental research proposed a novel permutation-based training method, which exhibited a desired classification performance without modifying the exact weight values. In this paper, we provide a theoretical guarantee of this permutation training method by proving its ability to guide a ReLU network to approximate one-dimensional continuous functions. Our numerical results further validate this method's efficiency in regression tasks with various initializations. The notable observations during weight permutation suggest that permutation training can provide an innovative tool for describing network learning behavior.

cs.LG↗

arXiv2024

Vanilla Feedforward Neural Networks as a Discretization of Dynamical Systems

Deep learning has made significant applications in the field of data science and natural science. Some studies have linked deep neural networks to dynamic systems, but the network structure is restricted to the residual network. It is known that residual networks can be regarded as a numerical discretization of dynamic systems. In this paper, we back to the classical network structure and prove that the vanilla feedforward networks could also be a numerical discretization of dynamic systems, where the width of the network is equal to the dimension of the input and output. Our proof is based on the properties of the leaky-ReLU function and the numerical technique of splitting method to solve differential equations. Our results could provide a new perspective for understanding the approximation properties of feedforward neural networks.

cs.LG↗

arXiv2024

Vocabulary for Universal Approximation: A Linguistic Perspective of Mapping Compositions

In recent years, deep learning-based sequence modelings, such as language models, have received much attention and success, which pushes researchers to explore the possibility of transforming non-sequential problems into a sequential form. Following this thought, deep neural networks can be represented as composite functions of a sequence of mappings, linear or nonlinear, where each composition can be viewed as a \emph{word}. However, the weights of linear mappings are undetermined and hence require an infinite number of words. In this article, we investigate the finite case and constructively prove the existence of a finite \emph{vocabulary} $V=\{ϕ_i: \mathbb{R}^d \to \mathbb{R}^d | i=1,...,n\}$ with $n=O(d^2)$ for the universal approximation. That is, for any continuous mapping $f: \mathbb{R}^d \to \mathbb{R}^d$, compact domain $Ω$ and $\varepsilon>0$, there is a sequence of mappings $ϕ_{i_1}, ..., ϕ_{i_m} \in V, m \in \mathbb{Z}_+$, such that the composition $ϕ_{i_m} \circ ... \circ ϕ_{i_1} $ approximates $f$ on $Ω$ with an error less than $\varepsilon$. Our results demonstrate an unusual approximation power of mapping compositions and motivate a novel compositional model for regular languages.

cs.LG↗

arXiv2024

Periodic Cylindrical Bilayers Self-Assembled from Diblock Polymers

Amphiphilic polymers in aqueous solutions can self-assemble to form bilayer membranes, and their elastic properties can be captured by the well-known Helfrich model involving several elastic constants. In this paper, we employ the self-consistent field model to simulate sinusoidal bilayers self-assembled from diblock copolymers where a proper constraint term is introduced to stabilize periodic bilayers with prescribed amplitudes. Then, we devise several methods to extract the shape of these bilayers and examine the accuracy of the free energy predicted by the Helfrich model. Numerical results show that when the bilayer curvature is small, the Helfrich model predicts the excess free energy more accurately. However, when the curvature is large, the accuracy heavily depends on the method used to determine the shape of the bilayer. In addition, the dependence of free energy on interaction strength, constraint amplitude, and constraint period are systematically studied. Moreover, we obtain certain periodic cylindrical bilayers that are equilibrium states of the self-consistent field model, which agree with the theoretical predictions made by the shape equations.

cond-mat.soft↗

arXiv2024

Minimum Width of Leaky-ReLU Neural Networks for Uniform Universal Approximation

The study of universal approximation properties (UAP) for neural networks (NN) has a long history. When the network width is unlimited, only a single hidden layer is sufficient for UAP. In contrast, when the depth is unlimited, the width for UAP needs to be not less than the critical width $w^*_{\min}=\max(d_x,d_y)$, where $d_x$ and $d_y$ are the dimensions of the input and output, respectively. Recently, \cite{cai2022achieve} shows that a leaky-ReLU NN with this critical width can achieve UAP for $L^p$ functions on a compact domain ${K}$, \emph{i.e.,} the UAP for $L^p({K},\mathbb{R}^{d_y})$. This paper examines a uniform UAP for the function class $C({K},\mathbb{R}^{d_y})$ and gives the exact minimum width of the leaky-ReLU NN as $w_{\min}=\max(d_x,d_y)+Δ(d_x, d_y)$, where $Δ(d_x, d_y)$ is the additional dimensions for approximating continuous functions with diffeomorphisms via embedding. To obtain this result, we propose a novel lift-flow-discretization approach that shows that the uniform UAP has a deep connection with topological theory.

cs.LG↗

arXiv2023

Achieve the Minimum Width of Neural Networks for Universal Approximation

The universal approximation property (UAP) of neural networks is fundamental for deep learning, and it is well known that wide neural networks are universal approximators of continuous functions within both the $L^p$ norm and the continuous/uniform norm. However, the exact minimum width, $w_{\min}$, for the UAP has not been studied thoroughly. Recently, using a decoder-memorizer-encoder scheme, \citet{Park2021Minimum} found that $w_{\min} = \max(d_x+1,d_y)$ for both the $L^p$-UAP of ReLU networks and the $C$-UAP of ReLU+STEP networks, where $d_x,d_y$ are the input and output dimensions, respectively. In this paper, we consider neural networks with an arbitrary set of activation functions. We prove that both $C$-UAP and $L^p$-UAP for functions on compact domains share a universal lower bound of the minimal width; that is, $w^*_{\min} = \max(d_x,d_y)$. In particular, the critical width, $w^*_{\min}$, for $L^p$-UAP can be achieved by leaky-ReLU networks, provided that the input or output dimension is larger than one. Our construction is based on the approximation power of neural ordinary differential equations and the ability to approximate flow maps by neural networks. The nonmonotone or discontinuous activation functions case and the one-dimensional case are also discussed.

cs.LG↗

arXiv2020

Machine learning and high-throughput robust design of P3HT-CNT composite thin films for high electrical conductivity

Combining high-throughput experiments with machine learning allows quick optimization of parameter spaces towards achieving target properties. In this study, we demonstrate that machine learning, combined with multi-labeled datasets, can additionally be used for scientific understanding and hypothesis testing. We introduce an automated flow system with high-throughput drop-casting for thin film preparation, followed by fast characterization of optical and electrical properties, with the capability to complete one cycle of learning of fully labeled ~160 samples in a single day. We combine regio-regular poly-3-hexylthiophene with various carbon nanotubes to achieve electrical conductivities as high as 1200 S/cm. Interestingly, a non-intuitive local optimum emerges when 10% of double-walled carbon nanotubes are added with long single wall carbon nanotubes, where the conductivity is seen to be as high as 700 S/cm, which we subsequently explain with high fidelity optical characterization. Employing dataset resampling strategies and graph-based regressions allows us to account for experimental cost and uncertainty estimation of correlated multi-outputs, and supports the proving of the hypothesis linking charge delocalization to electrical conductivity. We therefore present a robust machine-learning driven high-throughput experimental scheme that can be applied to optimize and understand properties of composites, or hybrid organic-inorganic materials.

physics.app-ph↗

arXiv2020

Optimization in Machine Learning: A Distribution Space Approach

We present the viewpoint that optimization problems encountered in machine learning can often be interpreted as minimizing a convex functional over a function space, but with a non-convex constraint set introduced by model parameterization. This observation allows us to repose such problems via a suitable relaxation as convex optimization problems in the space of distributions over the training parameters. We derive some simple relationships between the distribution-space problem and the original problem, e.g. a distribution-space solution is at least as good as a solution in the original space. Moreover, we develop a numerical algorithm based on mixture distributions to perform approximate optimization directly in distribution space. Consistency of this approximation is established and the numerical efficacy of the proposed algorithm is illustrated on simple examples. In both theory and practice, this formulation provides an alternative approach to large-scale optimization in machine learning.

cs.LG↗

arXiv2019

A Quantitative Analysis of the Effect of Batch Normalization on Gradient Descent

Despite its empirical success and recent theoretical progress, there generally lacks a quantitative analysis of the effect of batch normalization (BN) on the convergence and stability of gradient descent. In this paper, we provide such an analysis on the simple problem of ordinary least squares (OLS). Since precise dynamical properties of gradient descent (GD) is completely known for the OLS problem, it allows us to isolate and compare the additional effects of BN. More precisely, we show that unlike GD, gradient descent with BN (BNGD) converges for arbitrary learning rates for the weights, and the convergence remains linear under mild conditions. Moreover, we quantify two different sources of acceleration of BNGD over GD -- one due to over-parameterization which improves the effective condition number and another due having a large range of learning rates giving rise to fast descent. These phenomena set BNGD apart from GD and could account for much of its robustness properties. These findings are confirmed quantitatively by numerical experiments, which further show that many of the uncovered properties of BNGD in OLS are also observed qualitatively in more complex supervised learning problems.

cs.LG↗