arXiv subjects

Chaoyue Liu

Publications and source records attributed to Chaoyue Liu.

At least 19 recordsLinked to original sources

arXiv2026

Label-NTK Alignments and A Tighter Convergence Bound in the NTK Regime

The Neural Tangent Kernel (NTK) framework explains optimization in over-parameterized neural networks via approximately linearized dynamics, yielding exponential convergence guarantees. However, existing results are often overly pessimistic and do not match the fast training in practice, as they depend on the smallest NTK eigenvalue, which is typically extremely small in practice. In this work, we develop sharper convergence guarantees by characterizing the interaction between data labels and the NTK eigen-spectrum. We identify two key phenomena, Label-NTK alignment and Residual-NTK alignment, showing that projections of labels and residuals onto NTK eigenvectors scale with the corresponding eigenvalues. We provide empirical evidence and theoretical justification under mild data assumptions. Exploiting these alignment properties, we derive a refined convergence bound that depends on the full spectrum and closely matches practical training dynamics, significantly improving over classical worst-case results. We further obtain improved generalization bounds. Experiments on MLPs and CNNs across multiple datasets validate our theory.

cs.LG↗

arXiv2025

Better NTK Conditioning: A Free Lunch from (ReLU) Nonlinear Activation in Wide Neural Networks

Nonlinear activation functions are widely recognized for enhancing the expressivity of neural networks, which is the primary reason for their widespread implementation. In this work, we focus on ReLU activation and reveal a novel and intriguing property of nonlinear activations. By comparing enabling and disabling the nonlinear activations in the neural network, we demonstrate their specific effects on wide neural networks: (a) better feature separation, i.e., a larger angle separation for similar data in the feature space of model gradient, and (b) better NTK conditioning, i.e., a smaller condition number of neural tangent kernel (NTK). Furthermore, we show that the network depth (i.e., with more nonlinear activation operations) further amplifies these effects; in addition, in the infinite-width-then-depth limit, all data are equally separated with a fixed angle in the model gradient feature space, regardless of how similar they are originally in the input space. Note that, without the nonlinear activation, i.e., in a linear neural network, the data separation remains the same as for the original inputs and NTK condition number is equivalent to the Gram matrix, regardless of the network depth. Due to the close connection between NTK condition number and convergence theories, our results imply that nonlinear activation helps to improve the worst-case convergence rates of gradient based methods.

cs.LG↗

arXiv2025

MEMS chip-based single proof-mass triaxial fiber-optic accelerometer with ultra-low noise level

High-precision triaxial acceleration detection holds critical applications in seismic wave detection, geological resource exploration, and aerospace systems. Fabry-Perot (FP) optical sensors have gained widespread adoption in these domains due to their compact footprint and immunity to electromagnetic interference. Nevertheless, conventional three-axis measurements predominantly rely on assembling multiple single-axis transducers, introducing limitations such as increased device volume and misalignment errors. In this paper, we demonstrate a MEMS based monolithically integrated triaxial optical accelerometer that integrates a compact size with minimal noise and low crosstalk. The triaxial sensing structure employs a shared proof mass, achieving significant miniaturization compared to conventional multi-chip assembled triaxial optical accelerometers. In-plane sensing is realized through folded spring beams, while out-of-plane detection utilizes U-shaped suspension beams with widened central segments to suppress cross-axis sensitivity and enhance mechanical responsivity. Experimental results demonstrate that an operational bandwidth of 1\sim35 Hz, a minimum detectable acceleration of 4.12 ng/\sqrt{Hz}, and crosstalk below 0.023\%. The compact sensor footprint measures 16 mm \times 16 mm \times 0.5 mm. This optical accelerometer achieves nano-g resolution in the three-axis direction, demonstrating strong potential for applications in seismic wave detection and other precision vibration monitoring fields.

physics.optics↗

arXiv2025

Gradient Correction in Federated Learning with Adaptive Optimization

In federated learning (FL), model training performance is strongly impacted by data heterogeneity across clients. Client-drift compensation methods have recently emerged as a solution to this issue, introducing correction terms into local model updates. To date, these methods have only been considered under stochastic gradient descent (SGD)-based model training, while modern FL frameworks also employ adaptive optimizers (e.g., Adam) for improved convergence. However, due to the complex interplay between first and second moments found in most adaptive optimization methods, naively injecting correction terms can lead to performance degradation in heterogeneous settings. In this work, we propose {\tt FAdamGC}, the first algorithm to integrate drift compensation into adaptive federated optimization. The key idea of {\tt FAdamGC} is injecting a pre-estimation correction term that aligns with the moment structure of adaptive methods. We provide a rigorous convergence analysis of our algorithm under non-convex settings, showing that {\tt FAdamGC} results in better rate and milder assumptions than naively porting SGD-based correction algorithms into adaptive optimizers. Our experimental results demonstrate that {\tt FAdamGC} consistently outperform existing methods in total communication and computation cost across varying levels of data heterogeneity, showing the efficacy of correcting gradient information in federated adaptive optimization.

cs.LG↗

arXiv2025

Hubs and Spokes Learning: Efficient and Scalable Collaborative Machine Learning

We introduce the Hubs and Spokes Learning (HSL) framework, a novel paradigm for collaborative machine learning that combines the strengths of Federated Learning (FL) and Decentralized Learning (P2PL). HSL employs a two-tier communication structure that avoids the single point of failure inherent in FL and outperforms the state-of-the-art P2PL framework, Epidemic Learning Local (ELL). At equal communication budgets (total edges), HSL achieves higher performance than ELL, while at significantly lower communication budgets, it can match ELL's performance. For instance, with only 400 edges, HSL reaches the same test accuracy that ELL achieves with 1000 edges for 100 peers (spokes) on CIFAR-10, demonstrating its suitability for resource-constrained systems. HSL also achieves stronger consensus among nodes after mixing, resulting in improved performance with fewer training rounds. We substantiate these claims through rigorous theoretical analyses and extensive experimental results, showcasing HSL's practicality for large-scale collaborative learning.

cs.LG↗

arXiv2025

Communication-Efficient and Differentially Private Vertical Federated Learning with Zeroth-Order Optimization

Vertical Federated Learning (VFL) enables collaborative model training across feature-partitioned devices, yet its reliance on device-server information exchange introduces significant communication overhead and privacy risks. Downlink communication from the server to devices in VFL exposes gradient-related signals of the global loss that can be leveraged in inference attacks. Existing privacy-preserving VFL approaches that inject differential privacy (DP) noise on the downlink have the natural repercussion of degraded gradient quality, slowed convergence, and excessive communication rounds. In this work, we propose DPZV, a communication-efficient and differentially private ZO-VFL framework with tunable privacy guarantees. Based on zeroth-order (ZO) optimization, DPZV injects calibrated scalar-valued DP noise on the downlink, significantly reducing variance amplification while providing equivalent protection against targeted inference attacks. Through rigorous theoretical analysis, we establish convergence guarantees comparable to first-order DP-SGD, despite relying solely on ZO estimators, and prove that DPZV satisfies $(\epsilon, \delta)$-DP. Extensive experiments demonstrate that DPZV consistently achieves a superior privacy-utility tradeoff and requires fewer communication rounds than existing DP-VFL baselines under strict privacy constraints ($\epsilon \leq 10$).

cs.LG↗

arXiv2024

Catapults in SGD: spikes in the training loss and their impact on generalization through feature learning

In this paper, we first present an explanation regarding the common occurrence of spikes in the training loss when neural networks are trained with stochastic gradient descent (SGD). We provide evidence that the spikes in the training loss of SGD are "catapults", an optimization phenomenon originally observed in GD with large learning rates in [Lewkowycz et al. 2020]. We empirically show that these catapults occur in a low-dimensional subspace spanned by the top eigenvectors of the tangent kernel, for both GD and SGD. Second, we posit an explanation for how catapults lead to better generalization by demonstrating that catapults promote feature learning by increasing alignment with the Average Gradient Outer Product (AGOP) of the true predictor. Furthermore, we demonstrate that a smaller batch size in SGD induces a larger number of catapults, thereby improving AGOP alignment and test performance.

cs.LG↗

arXiv2024

Quadratic models for understanding catapult dynamics of neural networks

While neural networks can be approximated by linear models as their width increases, certain properties of wide neural networks cannot be captured by linear models. In this work we show that recently proposed Neural Quadratic Models can exhibit the "catapult phase" [Lewkowycz et al. 2020] that arises when training such models with large learning rates. We then empirically show that the behaviour of neural quadratic models parallels that of neural networks in generalization, especially in the catapult phase regime. Our analysis further demonstrates that quadratic models can be an effective tool for analysis of neural networks.

cs.LG↗

arXiv2023

Toward High-Performance Energy and Power Battery Cells with Machine Learning-based Optimization of Electrode Manufacturing

The optimization of the electrode manufacturing process is important for upscaling the application of Lithium Ion Batteries (LIBs) to cater for growing energy demand. In particular, LIB manufacturing is very important to be optimized because it determines the practical performance of the cells when the latter are being used in applications such as electric vehicles. In this study, we tackled the issue of high-performance electrodes for desired battery application conditions by proposing a powerful data-driven approach supported by a deterministic machine learning (ML)-assisted pipeline for bi-objective optimization of the electrochemical performance. This ML pipeline allows the inverse design of the process parameters to adopt in order to manufacture electrodes for energy or power applications. The latter work is an analogy to our previous work that supported the optimization of the electrode microstructures for kinetic, ionic, and electronic transport properties improvement. An electrochemical pseudo-two-dimensional model is fed with the electrode properties characterizing the electrode microstructures generated by manufacturing simulations and used to simulate the electrochemical performances. Secondly, the resulting dataset was used to train a deterministic ML model to implement fast bi-objective optimizations to identify optimal electrodes. Our results suggested a high amount of active material, combined with intermediate values of solid content in the slurry and calendering degree, to achieve the optimal electrodes.

cs.LG↗

arXiv2023

Transition to Linearity of General Neural Networks with Directed Acyclic Graph Architecture

In this paper we show that feedforward neural networks corresponding to arbitrary directed acyclic graphs undergo transition to linearity as their "width" approaches infinity. The width of these general networks is characterized by the minimum in-degree of their neurons, except for the input and first layers. Our results identify the mathematical structure underlying transition to linearity and generalize a number of recent works aimed at characterizing transition to linearity or constancy of the Neural Tangent Kernel for standard architectures.

cs.LG↗

arXiv2023

On Emergence of Clean-Priority Learning in Early Stopped Neural Networks

When random label noise is added to a training dataset, the prediction error of a neural network on a label-noise-free test dataset initially improves during early training but eventually deteriorates, following a U-shaped dependence on training time. This behaviour is believed to be a result of neural networks learning the pattern of clean data first and fitting the noise later in the training, a phenomenon that we refer to as clean-priority learning. In this study, we aim to explore the learning dynamics underlying this phenomenon. We theoretically demonstrate that, in the early stage of training, the update direction of gradient descent is determined by the clean subset of training data, leaving the noisy subset has minimal to no impact, resulting in a prioritization of clean learning. Moreover, we show both theoretically and experimentally, as the clean-priority learning goes on, the dominance of the gradients of clean samples over those of noisy samples diminishes, and finally results in a termination of the clean-priority learning and fitting of the noisy samples.

cs.LG↗

arXiv2023

Aiming towards the minimizers: fast convergence of SGD for overparametrized problems

Modern machine learning paradigms, such as deep learning, occur in or close to the interpolation regime, wherein the number of model parameters is much larger than the number of data samples. In this work, we propose a regularity condition within the interpolation regime which endows the stochastic gradient method with the same worst-case iteration complexity as the deterministic gradient method, while using only a single sampled gradient (or a minibatch) in each iteration. In contrast, all existing guarantees require the stochastic gradient method to take small steps, thereby resulting in a much slower linear rate of convergence. Finally, we demonstrate that our condition holds when training sufficiently wide feedforward neural networks with a linear output layer.

cs.LG↗

arXiv2022

Realization of advanced passive silicon photonic devices with subwavelength-grating structures developed by efficient inverse design

The realization of ultra-compact passive silicon photonic devices is becoming more and more important for the future large-scale photonic integration as desired for many systems. Although some compact silicon photonic devices have been demonstrated by using inverse design, the device performance is still insufficient for real applications. Here, we propose and realize several representative ultra-compact advanced passive silicon photonic devices with decent performances by introducing subwavelength-grating (SWG) structures developed by our high-efficiency inverse design method. These devices are designed by optimally manipulating the multimode excitation and the multimode interference in a region defined with SWG structures. These SWG structures with excellent feature-size uniformity are more fabrication-friendly than those random nano-structures used in previous inverse-designed photonic devices. The high-efficiency of our inverse design method is attributed to a novel search-space-dimension control strategy and the efficient problem-oriented electromagnetic-field solvers available for SWG structures. Specifically, we present the realization of a 6-channel mode (de)multiplexer, a broadband 90°-hybrid, and a two-channel flat-top wavelength demultiplexer as some examples, which can hardly be realized by previously reported inverse design approaches. These devices exhibit ultra-compact footprints as well as decent performances when compared to the counterparts developed by the classical theory.

physics.optics↗

arXiv2022

An Experimentally-Validated 3D Electrochemical Model Revealing Electrode Manufacturing Parameters Effects on Battery Performance

Electrode manufacturing is at the core of the lithium ion battery (LIB) fabrication process. The electrode microstructure and the electrochemical performance are determined by the adopted manufacturing parameters. However, in view of the strong interdependencies between these parameters, evaluating their influence on the performance is not a trivial task. In this work we present an experimentally validated 3D-resolved electrochemical model of a NMC111-based electrode which reveals how slurry formulation and calendering degree affect the electrode performance. A series of electrodes with different formulations and calendering degrees were fabricated at the experimental level. Corresponding three-dimensional manufacturing models were built based on the same experimental manufacturing parameters to generate the digital counterparts of the experimental electrodes that were then used in the electrochemical model. The results of simulations and experiments were compared individually. Among the manufacturing parameters analyzed, we found that the major factors linking manufacturing parameters and electrode performance are the carbon and binder domain (CBD) distribution within the electrode volume, and the electrostatic potential difference between the electrode and the current collector. A well-connected electronic conductive network throughout the electrode is vital for ensuring full utilization of active material, and it was found that increasing calendering degree is effective in reducing interfacial impedance. This work uncovers, based on a dual modeling/experimental approach, the essence of how electrode manufacturing process takes effect on electrode performance by influencing its microstructure.

physics.chem-ph↗

arXiv2022

Lithium Ion Battery Electrode Manufacturing Model Accounting for 3D Realistic Shapes of Active Material Particles

The demand for lithium ion batteries (LIBs) on the market has gradually risen, with production increasing every year. To meet industrial needs, the development of digital twins designed to optimize LIB manufacturing processes is essential. Here, by using LiNi0.33Co0.33Mn0.33O2 (NMC111) material as an example, we introduce the realistic particles shapes of the active material obtained from X-ray micro-computed tomography into a Coarse-Grained Molecular Dynamic physical model to simulate the slurry and its drying, and into a Discrete Element Method model able to simulate the calendering of the resulting electrode. This model enables to link the manufacturing parameters with the microstructure of the electrodes and to better observe the effect of the former on the heterogeneity of the electrodes. The results of the simulations allow us, among others, to observe the alteration of the electrode heterogeneity during the manufacturing process and the slight deformation of the secondary particles of active material.

physics.chem-ph↗

arXiv2022

High-speed graphene-silicon-graphene waveguide PDs with high photo-to-dark-current ratio and large linear dynamic range

Two-dimensional materials (2DMs) meet the demand of broadband and low-cost photodetection on silicon for many applications. Currently, it is still very challenging to realize excellent silicon-2DM PDs. Here we demonstrate graphene-silicon-graphene waveguide PDs operating at the wavelength-bands of 1.55 μm and 2 μm, showing the potential for large-scale integration. For the fabricated PDs, the measured responsivities are respectively ~0.15 mA/W and ~0.015 mA/W for the wavelengths of 1.55 μm and 1.96μm. In particular, the PDs exhibit a high bandwidth of ~33 GHz, an ultra-low dark current of tens of pico-amperes, a high normalized photo-to-dark-current ratio (NPDR) of 1.63x10^6 W^-1, as well as a high linear dynamic range of 3 μW-1.86 mW (and beyond) at 1.55 μm. According to the measurement results for the wavelength-bands of 1.55/2.0 μm and the theoretical modeling for the silicon-graphene heterostructure, it is revealed that internal photo-emission and photo-assisted thermionic field emission dominantly contribute to the photoresponse in the graphene-silicon Schottky junctions, which helps the future work to further improve the performance.

physics.optics↗

arXiv2022

Machine Learning-Assisted Multi-Objective Optimization of Battery Manufacturing from Synthetic Data Generated by Physics-Based Simulations

The optimization of the electrodes manufacturing process constitutes one of the most critical steps to ensure high-quality Lithium-Ion Battery (LIB) cells, in particular for automotive applications. Because LIB electrode manufacturing is a complex process involving multiple steps and interdependent parameters, we have shown in our previous works that 3D-resolved physics-based models constitute very useful tools to provide insights about the impact of the manufacturing process parameters on the textural and performance properties of the electrodes. However, their high-throughput application for electrode properties optimization and inverse design of manufacturing parameters is limited due to the high computational cost associated with this kind of model. In this work, we tackle this issue by proposing an innovative approach, supported by a deterministic machine learning (ML)-assisted pipeline for multi-objective optimization of LIB electrode properties and inverse design of its manufacturing process. Firstly, the pipeline generates a synthetic dataset from physics-based simulations with low discrepancy sequences, that allow to sufficiently represent the manufacturing parameters space. Secondly, the generated dataset is used to train deterministic ML models for the implementation of a fast multi-objective optimization, to identify an optimal electrode and the manufacturing parameters to adopt in order to fabricate it. Lastly, this electrode was successfully fabricated experimentally, proving that our modeling pipeline prediction is physical-relevant. Here, we demonstrate our pipeline for the simultaneous minimization of the electrode tortuosity factor and maximization of the effective electronic conductivity, the active surface area, and the density, all being parameters that affect the Li$^+$ (de-)intercalation kinetics, ionic, and electronic transport properties of the electrode.

physics.app-ph↗

arXiv2022

Transition to Linearity of Wide Neural Networks is an Emerging Property of Assembling Weak Models

Wide neural networks with linear output layer have been shown to be near-linear, and to have near-constant neural tangent kernel (NTK), in a region containing the optimization path of gradient descent. These findings seem counter-intuitive since in general neural networks are highly complex models. Why does a linear structure emerge when the networks become wide? In this work, we provide a new perspective on this "transition to linearity" by considering a neural network as an assembly model recursively built from a set of sub-models corresponding to individual neurons. In this view, we show that the linearity of wide neural networks is, in fact, an emerging property of assembling a large number of diverse "weak" sub-models, none of which dominate the assembly.

cs.LG↗