arXiv subjects

Yanxia Deng

Publications and source records attributed to Yanxia Deng.

9 recordsLinked to original sources

arXiv2025

DiaBlo: Diagonal Blocks Are Sufficient For Finetuning

Fine-tuning is a critical step for adapting large language models (LLMs) to domain-specific downstream tasks. To mitigate the substantial computational and memory costs of full-model fine-tuning, Parameter-Efficient Fine-Tuning (PEFT) methods have been proposed to update only a small subset of model parameters. However, performance gaps between PEFT approaches and full-model fine-tuning still exist. In this work, we present DiaBlo, a simple yet effective PEFT approach that updates only the diagonal blocks of selected model weight matrices. Unlike Low-Rank Adaptation (LoRA) and its variants, DiaBlo eliminates the need for low-rank matrix products, thereby avoiding the reliance on auxiliary initialization schemes or customized optimization strategies to improve convergence. This design leads to stable and robust convergence while maintaining comparable memory efficiency and training speed to LoRA. Moreover, we provide theoretical guarantees showing that, under mild low-rank conditions, DiaBlo is more expressive than LoRA in the linear problem and converges to a stationary point of the general nonlinear full fine-tuning. Through extensive experiments across a range of tasks, including commonsense reasoning, arithmetic reasoning, code generation, and safety alignment, we show that fine-tuning only diagonal blocks is sufficient for strong and consistent performance. DiaBlo not only achieves competitive accuracy but also preserves high memory efficiency and fast fine-tuning speed. Codes are available at https://github.com/ziyangjoy/DiaBlo.

cs.LG↗

arXiv2025

CLoQ: Enhancing Fine-Tuning of Quantized LLMs via Calibrated LoRA Initialization

Fine-tuning large language models (LLMs) using low-rank adaptation (LoRA) has become a highly efficient approach for downstream tasks, particularly in scenarios with limited computational resources. However, applying LoRA techniques to quantized LLMs poses unique challenges due to the reduced representational precision of quantized weights. In this paper, we introduce CLoQ (Calibrated LoRA initialization for Quantized LLMs), a simplistic initialization strategy designed to overcome these challenges. Our approach focuses on minimizing the layer-wise discrepancy between the original LLM and its quantized counterpart with LoRA components during initialization. By leveraging a small calibration dataset, CLoQ quantizes a pre-trained LLM and determines the optimal LoRA components for each layer, ensuring a strong foundation for subsequent fine-tuning. A key contribution of this work is a novel theoretical result that enables the accurate and closed-form construction of these optimal LoRA components. We validate the efficacy of CLoQ across multiple tasks such as language generation, arithmetic reasoning, and commonsense reasoning, demonstrating that it consistently outperforms existing LoRA fine-tuning methods for quantized LLMs, especially at ultra low-bit widths.

cs.LG↗

arXiv2024

MagR: Weight Magnitude Reduction for Enhancing Post-Training Quantization

In this paper, we present a simple optimization-based preprocessing technique called Weight Magnitude Reduction (MagR) to improve the performance of post-training quantization. For each linear layer, we adjust the pre-trained floating-point weights by solving an $\ell_\infty$-regularized optimization problem. This process greatly diminishes the maximum magnitude of the weights and smooths out outliers, while preserving the layer's output. The preprocessed weights are centered more towards zero, which facilitates the subsequent quantization process. To implement MagR, we address the $\ell_\infty$-regularization by employing an efficient proximal gradient descent algorithm. Unlike existing preprocessing methods that involve linear transformations and subsequent post-processing steps, which can introduce significant overhead at inference time, MagR functions as a non-linear transformation, eliminating the need for any additional post-processing. This ensures that MagR introduces no overhead whatsoever during inference. Our experiments demonstrate that MagR achieves state-of-the-art performance on the Llama family of models. For example, we achieve a Wikitext2 perplexity of 5.95 on the LLaMA2-70B model for per-channel INT2 weight quantization without incurring any inference overhead.

cs.LG↗

arXiv2021

Action and periodic orbits on annulus

We consider the classical problem of area-preserving maps on annulus $\mathbb{A} = S^1 \times [0, 1]$ . Let $\mathcal{M}_f$ be the set of all invariant probability measures of an area-preserving, orientation preserving diffeomorphism $f$ on $\mathbb{A}$. Given any $\mu_1$ and $\mu_2$ in $\mathcal{M}_f$, Franks \cite{Franks1988}\cite{Franks1992}, generalizing Poincar\'e's last geometric theorem (Birkhoff \cite{Birkhoff1913}), showed that if their rotation numbers are different, then $f$ has infinitely many periodic orbits. In this paper, we show that if $\mu_1$ and $\mu_2$ have different actions, even if they have the same rotation number, then $f$ has infinitely many periodic orbits. In particular, if the action difference is larger than one, then $f$ has at least two fixed points. The same result is also true for area-preserving diffeomorphisms on unit disk, where no rotation number is available.

math.DS↗

arXiv2021

Odd index of the amended potential implies linear instability

For a relative equilibrium of a symmetric simple mechanical system, if the Morse index of the corresponding amended potential is odd, whether the nullity is zero or not, it is linearly unstable. We also provide a sufficient condition for spectral instability.

math.DS↗

arXiv2020

Global existence and singularity of the Hill's type lunar problem with strong potential

We characterize the fate of the solutions of Hill's type lunar problem using the ideas of ground states from PDE. In particular, the relative equilibrium will be defined as the ground state, which satisfies some crucial energetic variational properties in our analysis. We study the dynamics of the solutions below, at, and (slightly) above the ground state energy threshold.

math.DS↗

arXiv2019

Numerical investigation on the Hill's type lunar problem with homogeneous potential

We consider the planar Hill's lunar problem with a homogeneous gravitational potential. The investigation of the system is twofold. First, the starting conditions of the trajectories are classified into three classes, that is bounded, escaping, and collisional. Second, we study the no-return property of the Lagrange point $L_2$ and we observe that the escaping trajectories are scattered exponentially. Moreover, it is seen that in the supercritical case, with $\alpha \geq 2$, the basin boundaries are smooth. On the other hand, in the subcritical case, with $\alpha < 2$ the boundaries between the different types of basins exhibit fractal properties.

nlin.CD↗

arXiv2019

Global Existence and Singularity of the N-body Problem with Strong Force

We use the idea of ground states and excited states in nonlinear dispersive equations (e.g. Klein-Gordon and Schr\"odinger equations) to characterize solutions in the N-body problem with strong force under some energy constraints. Indeed, relative equilibria of the N-body problem play a similar role as solitons in PDE. We introduce the ground state and excited energy for the N-body problem. {We are able to give a conditional dichotomy of the global existence and singularity below the excited energy in Theorem \ref{thm:dichotomy}, the proof of which seems original and simple. This dichotomy is given by the sign of a threshold function $K_\omega$}. The characterization for the two-body problem in this new perspective is non-conditional and it resembles the results in PDE nicely. For $N\geq3$, we will give some refinements of the characterization, in particular, we examine the situation where there are infinitely transitions for the sign of $K_\omega$.

math.CA↗

arXiv2018

Stability of Periodic Orbits by Conley-Zehnder index theory

We give a necessary and sufficient condition for strong stability of low dimensional Hamiltonian systems, in terms of the iterates of a closed orbit and the Conley-Zehnder index. Applications to Mathieu equation and stable harmonic oscillations for forced pendulum type equations are considered as applications of the main result.

math.DS↗