arXiv subjects

Shuchen Zhu

Publications and source records attributed to Shuchen Zhu.

At least 19 recordsLinked to original sources

arXiv2026

Negligible in Size, Significant in Effect: On Scale Vectors in Large Language Models

Normalization layers in modern large language models (LLMs) consist of a deterministic normalization operation and a learnable scale vector. While the normalization operation has been extensively studied, the scale vector remains poorly understood despite its ubiquitous use. In this work, we present a systematic study of scale vectors in LLMs from the perspectives of expressivity, optimization, and architectural structure. First, we show empirically that although scale vectors constitute only a negligible fraction of model parameters, removing them substantially degrades LLM pre-training. Our theory further shows that, in Pre-Norm architectures, scale vectors do not increase expressivity; instead, they improve optimization through a self-amplifying preconditioning effect on subsequent linear mappings. Second, we investigate the role of weight decay for scale vectors. By distinguishing Input-Norm and Output-Norm layers, we theoretically show that weight decay is beneficial for the former but harmful for the latter, due to their distinct roles in optimization and expressivity. Third, motivated by this understanding, we propose three lightweight and complementary improvements to scale vectors: branch-specific heterogeneity, improved placement around linear mappings, and magnitude-direction reparameterization. Both theory and experiments show that each improvement yields consistent gains. Finally, we combine these improvements into a unified scale-vector strategy and evaluate it through extensive LLM pre-training experiments on dense and mixture-of-experts models ranging from 0.12B to 2B parameters, across multiple optimizers and learning rate schedules, under industrial-scale token budgets. The unified strategy consistently achieves lower terminal loss than well-tuned baselines and exhibits more favorable scaling behavior, while adding negligible parameter and computational overhead.

cs.LG↗

arXiv2026

Curvature-Conditioned Multiscale Momentum with Sphere Constraints for LLM Pretraining

Pretraining accounts for a large fraction of the total computational cost in LLM training. However, noise-dominant gradients and the highly ill-conditioned loss landscape bring severe challenges. Although modern adaptive optimizers such as AdamW and Muon have achieved great success in large-scale pretraining, their reliance on gradient normalization offers limited mitigation of the ill-conditioned curvature. The progress along flat directions (eigen-directions of small eigenvalues), which dominates the final loss reduction, remains relatively slow. To enhance training dynamics along flat directions, we propose a curvature-conditioned multiscale momentum method with sphere constraints, delivering steady acceleration in LLM pretraining. This multiscale momentum, applied only along flat directions, pairs a slow-decay component for noise reduction with a fast-decay component for rapid curvature adaptation, harnessing their complementary strengths. Crucially, we employ a sphere constraint technique to prevent parameter inflation and excessively rapid effective learning rate decay that would otherwise arise from a naive combination. Extensive experiments show that the proposed method significantly accelerates Muon across diverse architectures (dense, MoE) and model sizes (0.12B--2.3B parameters). Theoretically, we verify the acceleration effect and provide insight into the design principles underlying the flat-direction multiscale momentum.

cs.LG↗

arXiv2026

ReasFlow: Assisting Reasoning-Centric Scientific Discovery in Applied Mathematics via a Knowledge-Based Multi-Agent System

Recent advances in Large Language Models have fueled autonomous AI agents capable of tackling complex scientific tasks, yet existing automated research systems remain predominantly focused on empirically driven domains with quantitative benchmarks, leaving theory-driven discovery, particularly in mathematically grounded disciplines requiring rigorous proofs and synthesis of domain knowledge, largely underexplored. Key challenges include the difficulty of verifying theoretical reasoning at scale, insufficient reasoning ability for autonomous frontier exploration, and a scarcity of procedural heuristics in the literature. We introduce ReasFlow, an end-to-end autonomous agent system for reasoning-centric scientific discovery that operationalizes a collaborative paradigm where the human expert acts as Principal Investigator while the agent executes rigorous derivations as a capable graduate student. ReasFlow incorporates (i) a robust internal verification loop that audits logical coherence and corrects fundamental errors prior to human inspection, and (ii) an automated knowledge retrieval and self-improvement mechanism that proactively surfaces both declarative facts and overlooked procedural heuristics, substantially reducing expert intervention. The system unifies literature synthesis, algorithm design, theorem proving, experimentation, and manuscript preparation in a single system. Deployed to autonomously generate five complete research papers with rigorous theoretical and empirical content from minimal prompts, ReasFlow consistently achieves the highest evaluation scores among state-of-the-art open-access baselines under a curated LLM-based review rubric. ReasFlow is publicly accessible via the ReasLab platform, providing a collaborative workspace for AI-assisted theoretical research. Github repo: https://github.com/reaslab/ReasFlow.git.

cs.AI↗

arXiv2026

Logical Resource Estimation for Quantum State Preparation with Compilation

Quantum state preparation is a fundamental primitive in quantum algorithms for encoding classical data into quantum amplitudes. We compare the cost of preparing general $n$-qubit states with real amplitudes using two common paradigms: rotation-based methods, based on controlled rotations, and sampling-based methods, based on a structured representation of the target state. Although these approaches are often theoretically compared using CNOT count and $T$-count, their relative performance in total gate count remains less well understood practically. We compare representative rotation-based and sampling-based methods using $T$-count and total gate count, and analyze how compilation overhead affects their relative performance. We also develop a software package for compiling state preparation circuits, designed as a practical subroutine for more general quantum computations. Numerical experiments on resource states and quantum states related to quantum chemistry, condensed matter physics, and simulation via Magnus expansion over a range of target accuracies $ε$ support the analysis. Our results show that sampling-based methods achieve asymptotically lower $T$-count and retain an overall advantage after accounting for total gate count and compilation overhead.

quant-ph↗

arXiv2026

Subspace Optimization for Efficient Federated Learning under Heterogeneous Data

Federated learning increasingly operates in a large-model regime where communication, memory, and computation are all scarce. Typically, non-IID client data induce drift that degrades the stability and performance of local training. Existing remedies such as SCAFFOLD introduce heterogeneity-correction mechanisms to address this challenge, but they incur substantial extra communication and memory overhead. This paper proposes a subspace optimization method for federated learning (SSF), which performs heterogeneity-corrected optimization in a low-dimensional subspace using only projected quantities, while preserving full-dimensional control information through a backfill-style update that retains residual components whenever the active subspace changes. Under standard smoothness and bounded-variance assumptions, SSF attains a non-asymptotic rate of order $\widetilde{\mathcal{O}}(1/T+1/\sqrt{NKT})$. Experiments show favorable accuracy--efficiency trade-offs under heterogeneous data.

cs.LG↗

arXiv2026

Accelerating LLM Pre-Training through Flat-Direction Dynamics Enhancement

Pre-training Large Language Models requires immense computational resources, making optimizer efficiency essential. The optimization landscape is highly anisotropic, with loss reduction driven predominantly by progress along flat directions. While matrix-based optimizers such as Muon and SOAP leverage fine-grained curvature information to outperform AdamW, their updates tend toward isotropy -- relatively conservative along flat directions yet potentially aggressive along sharp ones. To address this limitation, we first establish a unified Riemannian Ordinary Differential Equation (ODE) framework that elucidates how common adaptive algorithms operate synergistically: the preconditioner induces a Riemannian geometry that mitigates ill-conditioning, while momentum serves as a Riemannian damping term that promotes convergence. Guided by these insights, we propose LITE, a generalized acceleration strategy that enhances training dynamics by applying larger Hessian damping coefficients and learning rates along flat trajectories. Extensive experiments demonstrate that LITE significantly accelerates both Muon and SOAP across diverse architectures (Dense, MoE), parameter scales (130M--1.3B), datasets (C4, Pile), and learning-rate schedules (cosine, warmup-stable-decay). Theoretical analysis confirms that LITE facilitates faster convergence along flat directions in anisotropic landscapes, providing a principled approach to efficient LLM pre-training. The code is available at https://github.com/SHUCHENZHU/LITE.

cs.LG↗

arXiv2026

Classical Simulation of Noiseless Quantum Dynamics without Randomness

Simulating noiseless quantum dynamics classically faces a fundamental dilemma: tensor-network methods become inefficient as entanglement saturates, while Pauli-truncation approaches typically rely on noise or randomness. To close the gap, we propose the Low-weight Pauli Dynamics (LPD) algorithm that efficiently approximates local observables for short-time dynamics in the absence of noise. We prove that the truncation error admits an average-case bound without assuming randomness, provided that the state is sufficiently entangled. Counterintuitively, entanglement--usually an obstacle for classical simulation--alleviates classical simulation error. We further show that such entangled states can be generated either by tensor-network classical simulation or near-term quantum devices. Our results establish a rigorous synergy between existing classical simulation methods and provide a complementary route to quantum simulation that reduces circuit depth for long-time dynamics, thereby extending the accessible regime of quantum dynamics.

quant-ph↗

arXiv2026

Learning Hamiltonians in the Heisenberg limit with static single-qubit fields

Learning the Hamiltonian governing a quantum system is a central task in quantum metrology, sensing, and device characterization. Existing Heisenberg-limited Hamiltonian learning protocols either require multi-qubit operations that are prone to noise, or single-qubit operations whose frequency or strength increases with the desired precision. These two requirements limit the applicability of Hamiltonian learning on near-term quantum platforms. We present a protocol that learns a quantum Hamiltonian with the optimal Heisenberg-limited scaling using only single-qubit control in the form of static fields with strengths that are independent of the target precision. Our protocol is robust against the state preparation and measurement (SPAM) error. By overcoming these limitations, our protocol provides new tools for device characterization and quantum sensing. We demonstrate that our method achieves the Heisenberg-limited scaling through rigorous mathematical proof and numerical experiments. We also prove an information-theoretic lower bound showing that a non-vanishing static field strength is necessary for achieving the Heisenberg limit unless one employs an extensive number of discrete control operations.

quant-ph↗

arXiv2025

Magic State Distillation using Asymptotically Good Codes on Qudits

Qudits offer the potential for low-overhead magic state distillation, although previous results for asymptotically good codes have required qudit dimension $q\gg 100$ or code length $\mathcal{N}\gg 100$. These parameters far exceed experimental demonstrations of qudit platforms, and thus motivate the search for better codes. Using a novel lifting procedure, we construct the first family of good triorthogonal codes on the $\mathbb{F}_{2^{2m}}$ alphabet with $m \geq 3$ that lies above the Tsfasman-Vladut-Zink bound. These codes yield a family of asymptotically good quantum codes with transversal CCZ gates, enabling constant space overhead magic state distillation with qudit dimension as small as $q=64$. Further, we identify a promising code with parameters $[[42,14,6]]_{64}$. Finally, we show that a distilled $|{CCZ}\rangle_{2^{2m}}$ can be reduced to a $|{CCZ}\rangle_{2^n}$ state for arbitrary $n$ with a constant-depth Clifford circuit of at most 9 computational basis measurements, 12 single-qudit and 9 two-qudit Clifford gates.

quant-ph↗

arXiv2025

Unified Architecture for Quantum Lookup Tables

Quantum access to arbitrary classical data encoded in unitary black-box oracles underlies interesting data-intensive quantum algorithms, such as machine learning or electronic structure simulation. The feasibility of these applications depends crucially on gate-efficient implementations of these oracles, which are commonly some reversible versions of the boolean circuit for a classical lookup table. We present a general parameterized architecture for quantum circuits implementing a lookup table that encompasses all prior work in realizing a continuum of optimal tradeoffs between qubits, non-Clifford gates, and error resilience, up to logarithmic factors. Our architecture assumes only local 2D connectivity, yet recovers results that previously required all-to-all connectivity, particularly, with the appropriate parameters, poly-logarithmic error scaling. We also identify novel regimes, such as simultaneous sublinear scaling in all parameters. These results enable tailoring implementations of the commonly used lookup table primitive to any given quantum device with constrained resources.

quant-ph↗

arXiv2025

Generalized Short Path Algorithms: Towards Super-Quadratic Speedup over Markov Chain Search for Combinatorial Optimization

We analyze generalizations of quantum algorithms based on the short path framework first proposed by Hastings~[\textit{Quantum} 2, 78 (2018)], which has been extended and shown by Dalzell~et~al.~[STOC~'23] to achieve super-Grover speedups for certain binary optimization problems. We demonstrate that, under some commonly satisfied technical conditions, an appropriate generalization can achieve super-quadratic speedups not only over unstructured search but also over a classical optimization algorithm that searches for the optimum by drawing samples from the stationary distribution of a Markov chain. We employ this framework to obtain algorithms for problems including variants of Max Bisection, Max Independent Set, and finding the ground states of the Antiferromagnetic Ising Model and the Sherrington-Kirkpatrick Model, whose runtimes are asymptotically faster than those obtainable with previous short path techniques. In certain cases, our algorithms achieve super-quadratic speedups compared to the best known classical algorithms with rigorously established runtimes.

quant-ph↗

arXiv2025

Decentralized Bilevel Optimization: A Perspective from Transient Iteration Complexity

Stochastic bilevel optimization (SBO) is becoming increasingly essential in machine learning due to its versatility in handling nested structures. To address large-scale SBO, decentralized approaches have emerged as effective paradigms in which nodes communicate with immediate neighbors without a central server, thereby improving communication efficiency and enhancing algorithmic robustness. However, most decentralized SBO algorithms focus solely on asymptotic convergence rates, overlooking transient iteration complexity-the number of iterations required before asymptotic rates dominate, which results in limited understanding of the influence of network topology, data heterogeneity, and the nested bilevel algorithmic structures. To address this issue, this paper introduces D-SOBA, a Decentralized Stochastic One-loop Bilevel Algorithm framework. D-SOBA comprises two variants: D-SOBA-SO, which incorporates second-order Hessian and Jacobian matrices, and D-SOBA-FO, which relies entirely on first-order gradients. We provide a comprehensive non-asymptotic convergence analysis and establish the transient iteration complexity of D-SOBA. This provides the first theoretical understanding of how network topology, data heterogeneity, and nested bilevel structures influence decentralized SBO. Extensive experimental results demonstrate the efficiency and theoretical advantages of D-SOBA.

math.OC↗

arXiv2025

Block encoding with low gate count for second-quantized Hamiltonians

Efficient block encoding of many-body Hamiltonians is a central requirement for quantum algorithms in scientific computing, particularly in the early fault-tolerant era. In this work, we introduce new explicit constructions for block encoding second-quantized Hamiltonians that substantially reduce Clifford+T gate complexity and ancilla overhead. By utilizing a data lookup strategy based on the SWAP architecture for the sparsity oracle $O_C$, and a direct sampling method for the amplitude oracle $O_A$ with SELECT-SWAP architecture, we achieve a T count that scales as $\mathcal{\tilde{O}}(\sqrt{L})$ with respect to the number of interaction terms $L$ in general second-quantized Hamiltonians. We also achieve an improved constant factor in the Clifford gate count of our oracle. Furthermore, we design a block encoding that directly targets the $\eta$-particle subspace, thereby reducing the subnormalization factor from $\mathcal{O}(L)$ to $\mathcal{O}(\sqrt{L})$, and improving fault-tolerant efficiency when simulating systems with fixed particle numbers. Building on the block encoding framework developed for general many-body Hamiltonians, we extend our approach to electronic Hamiltonians whose coefficient tensors exhibit translation invariance or possess decaying structures. Our results provide a practical path toward early fault-tolerant quantum simulation of many-body systems, substantially lowering resource overheads compared to previous methods.

quant-ph↗

arXiv2025

High-order Magnus Expansion for Hamiltonian Simulation

Efficient simulation of quantum dynamics with time-dependent Hamiltonians is important not only for time-varying systems but also for time-independent Hamiltonians in the interaction picture. Such simulations are more challenging than their time-independent counterparts due to the complexity introduced by time ordering. Existing algorithms that aim to capture commutator-based scaling either exhibit polynomial cost dependence on the Hamiltonian's time derivatives or are limited to low-order accuracy. In this work, we establish the general commutator-scaling error bounds for the truncated Magnus expansion at arbitrary order, where only Hamiltonian terms appear in the nested commutators, with no time derivatives involved. Building on this analysis, we design a high-order quantum algorithm with explicit circuit constructions. The algorithm achieves cost scaling with the commutator structure in the high-precision regime and depends only logarithmically on the Hamiltonian's time variation, making it efficient for general time-dependent settings, including the interaction picture.

quant-ph↗

arXiv2025

State-space gradient descent and metastability in quantum systems

We propose a quantum algorithm, inspired by ADAPT-VQE, to variationally prepare the ground state of a quantum Hamiltonian, with the desirable property that if it fails to find the ground state, it still yields a physically meaningful local-minimum state that oftentimes corresponds to a metastable state of the quantum system. At each iteration, our algorithm reduces the energy using a set of local physical operations. The operations to perform are chosen using gradient and Hessian information that can be efficiently extracted from experiments. We show that our algorithm does not suffer from the barren plateau problem, which is a significant issue in many variational quantum algorithms. We use numerical simulation to demonstrate that our method reliably produces either the true ground state or a physically meaningful metastable state in typical physical systems with such states.

quant-ph↗

arXiv2025

Accelerated Natural Gradient Method for Parametric Manifold Optimization

Parametric manifold optimization problems frequently arise in various machine learning tasks, where state functions are defined on infinite-dimensional manifolds. We propose a unified accelerated natural gradient descent (ANGD) framework to address these problems. By incorporating a Hessian-driven damping term into the manifold update, we derive an accelerated Riemannian gradient (ARG) flow that mitigates oscillations. An equivalent first-order system is further presented for the ARG flow, enabling a unified discretization scheme that leads to the ANGD method. In our discrete update, our framework considers various advanced techniques, including least squares approximation of the update direction, projected momentum to accelerate convergence, and efficient approximation methods through the Kronecker product. It accommodates various metrics, including $H^s$, Fisher-Rao, and Wasserstein-2 metrics, providing a computationally efficient solution for large-scale parameter spaces. We establish a convergence rate for the ARG flow under geodesic convexity assumptions. Numerical experiments demonstrate that ANGD outperforms standard NGD, underscoring its effectiveness across diverse deep learning tasks.

math.OC↗

arXiv2025

Synthesis of Single Qutrit Circuits from Clifford+R

We present two deterministic algorithms to approximate single-qutrit gates. These algorithms utilize the Clifford + $\mathbf{R}$ group to find the best approximation of diagonal rotations. The first algorithm exhaustively searches over the group; while the second algorithm searches only for Householder reflections. The exhaustive search algorithm yields an average $\mathbf{R}$ count of $2.193(11) + 8.621(7) \log_{10}(1 / \varepsilon)$, albeit with a time complexity of $\mathcal{O}(\varepsilon^{-4.4})$. The Householder search algorithm results in a larger average $\mathbf{R}$ count of $3.20(13) + 10.77(3) \log_{10}(1 / \varepsilon)$ at a reduced time complexity of $\mathcal{O}(\varepsilon^{-0.42})$, greatly extending the reach in $\varepsilon$. These costs correspond asymptotically to 35% and 69% more non-Clifford gates compared to synthesizing the same unitary with two qubits. Such initial results are encouraging for using the $\mathbf{R}$ gate as the non-transversal gate for qutrit-based computation.

quant-ph↗

arXiv2024

SPARKLE: A Unified Single-Loop Primal-Dual Framework for Decentralized Bilevel Optimization

This paper studies decentralized bilevel optimization, in which multiple agents collaborate to solve problems involving nested optimization structures with neighborhood communications. Most existing literature primarily utilizes gradient tracking to mitigate the influence of data heterogeneity, without exploring other well-known heterogeneity-correction techniques such as EXTRA or Exact Diffusion. Additionally, these studies often employ identical decentralized strategies for both upper- and lower-level problems, neglecting to leverage distinct mechanisms across different levels. To address these limitations, this paper proposes SPARKLE, a unified Single-loop Primal-dual AlgoRithm frameworK for decentraLized bilEvel optimization. SPARKLE offers the flexibility to incorporate various heterogeneitycorrection strategies into the algorithm. Moreover, SPARKLE allows for different strategies to solve upper- and lower-level problems. We present a unified convergence analysis for SPARKLE, applicable to all its variants, with state-of-the-art convergence rates compared to existing decentralized bilevel algorithms. Our results further reveal that EXTRA and Exact Diffusion are more suitable for decentralized bilevel optimization, and using mixed strategies in bilevel algorithms brings more benefits than relying solely on gradient tracking.

math.OC↗