arXiv subjects

Yuqi Luo

Publications and source records attributed to Yuqi Luo.

3 recordsLinked to original sources

arXiv2026

Heavy-quark transport across the QCD crossover driven by a lattice-constrained in-medium potential

We present a self-consistent framework for heavy-quark transport in the quark-gluon plasma across the QCD crossover region. By synthesizing perturbative and nonperturbative interactions into a unified interaction kernel, we circumvent the traditional reliance on arbitrary soft-hard momentum separation scales. The interaction is governed by an in-medium effective potential, incorporating short-range Yukawa screening and long-range confining string contributions, both rigorously constrained by the latest lattice QCD data. Our results reveal that the nonperturbative string tension is indispensable for capturing the extreme opacity of the medium near the critical temperature $T_c$. Specifically, our model predicts a spatial diffusion coefficient of $2\pi T D_s \approx 0.5 \sim 1.7$, demonstrating a striking quantitative agreement with the recent lattice QCD extractions. Ultimately, our results provide a robust dynamical interpretation of the strong heavy-quark coupling near the QCD crossover and offer a unified framework for describing heavy-flavor transport in hot and dense QCD matter.

hep-ph↗

arXiv2024

Sparsing Law: Towards Large Language Models with Greater Activation Sparsity

Activation sparsity denotes the existence of substantial weakly-contributed elements within activation outputs that can be eliminated, benefiting many important applications concerned with large language models (LLMs). Although promoting greater activation sparsity within LLMs deserves deep studies, existing works lack comprehensive and quantitative research on the correlation between activation sparsity and potentially influential factors. In this paper, we present a comprehensive study on the quantitative scaling properties and influential factors of the activation sparsity within decoder-only Transformer-based LLMs. Specifically, we propose PPL-$p\%$ sparsity, a precise and performance-aware activation sparsity metric that is applicable to any activation function. Through extensive experiments, we find several important phenomena. Firstly, different activation functions exhibit comparable performance but opposite training-time sparsity trends. The activation ratio (i.e., $1-\mathrm{sparsity\ ratio}$) evolves as a convergent increasing power-law and decreasing logspace power-law with the amount of training data for SiLU-activated and ReLU-activated LLMs, respectively. These demonstrate that ReLU is more efficient as the activation function than SiLU and can leverage more training data to improve activation sparsity. Secondly, the activation ratio linearly increases with the width-depth ratio below a certain bottleneck point, indicating the potential advantage of a deeper architecture at a fixed parameter scale. Finally, at similar width-depth ratios, we surprisingly find that the limit value of activation sparsity varies weakly with the parameter scale, i.e., the activation patterns within LLMs are insensitive to the parameter scale. These empirical laws towards LLMs with greater activation sparsity have important implications for making LLMs more efficient and interpretable.

cs.LG↗

arXiv2023

Variator: Accelerating Pre-trained Models with Plug-and-Play Compression Modules

Pre-trained language models (PLMs) have achieved remarkable results on NLP tasks but at the expense of huge parameter sizes and the consequent computational costs. In this paper, we propose Variator, a parameter-efficient acceleration method that enhances computational efficiency through plug-and-play compression plugins. Compression plugins are designed to reduce the sequence length via compressing multiple hidden vectors into one and trained with original PLMs frozen. Different from traditional model acceleration methods, which compress PLMs to smaller sizes, Variator offers two distinct advantages: (1) In real-world applications, the plug-and-play nature of our compression plugins enables dynamic selection of different compression plugins with varying acceleration ratios based on the current workload. (2) The compression plugin comprises a few compact neural network layers with minimal parameters, significantly saving storage and memory overhead, particularly in scenarios with a growing number of tasks. We validate the effectiveness of Variator on seven datasets. Experimental results show that Variator can save 53% computational costs using only 0.9% additional parameters with a performance drop of less than 2%. Moreover, when the model scales to billions of parameters, Variator matches the strong performance of uncompressed PLMs.

cs.CL↗