arXiv subjects

Shigeng Wang

Publications and source records attributed to Shigeng Wang.

6 recordsLinked to original sources

arXiv2026

Attend to Your Own Thoughts: Breaking the Barrier for Post-Training Quantization of Reasoning LLMs through the Lens of 1.58-Bit Quantization

We propose ScaleQ-1.58, a scalable ternary post-training quantization (PTQ) framework for reasoning LLMs. Its core insight stems from an empirical finding: although modern LLMs are typically trained to exhibit chain-of-thought reasoning capabilities, in the PTQ regime, even the latest CAT-Q method based on learning-based differentiable ternarization still leads to performance collapse on challenging mathematics and coding tasks when using conventional calibration schemes that ignore the model's reasoning process. Driven by this finding, we introduce a simple calibration approach, Attend to Your Own Thoughts (AYOT), where reasoning traces and final answers generated by the pre-trained high-precision target LLM on a proper set of calibration samples are used as the context input during the ternarization process, along with the corresponding questions. ScaleQ-1.58 is formed by simply integrating AYOT with CAT-Q, which demonstrates several scaling properties: (1) with only 4M calibration tokens, Qwen3-1.7B ternarized by ScaleQ-1.58 reaches over 90.52% of the performance of the prior best BitNet b1.58 2B4T averaged over 4 mathematics and coding tasks, and our ternary Qwen3-4B shows an absolute gain of 8.97%, while requiring 1,000,000x fewer calibration tokens for quantization; (2) ScaleQ-1.58 generalizes well to both dense and MoE architectures, with performance improving as model scale increases (up to 235B parameters); (3) ScaleQ-1.58 demonstrates strong generalization across tasks of varying difficulty levels, including mathematics, coding and scientific logic reasoning, as well as commonsense reasoning and basic language generation; (4) its performance continues to improve as the number of calibration tokens increases. Notably, AYOT also exhibits strong generalization ability across other quantization bit-widths. Code will be available at https://github.com/IntelChina-AI/BitTern.

cs.CL↗

arXiv2026

CAT-Q: Cost-efficient and Accurate Ternary Quantization for LLMs

In this paper, we present CAT-Q, Cost-efficient and Accurate Ternary Quantization, for compressing and accelerating LLMs. Unlike existing state-of-the-art ternary quantization methods that rely on data-intensive and costly quantization-aware training to mitigate severe performance degradation, CAT-Q is a simple yet effective post-training quantization scheme that is readily applicable to LLMs with diverse architectures and model sizes. It has two key components, learnable modulation (LM) and softened ternarization (ST), which are coupled from an optimization perspective. LM leverages a composition of learnable factors to modulate the distribution of pre-trained high-precision weights and the ternary threshold, making them less sensitive to ternarization. ST further introduces a differentiable transition function to guide the ternarization process toward stable convergence. We show that, for pre-trained LLMs with 1.7B to 8B parameters, CAT-Q can efficiently quantize them into ternary models using only 512 calibration samples, while achieving superior performance than the seminal BitNet 1.58-bit v1 and v2 families (with 1.3B to 7B parameters) trained with 100B tokens, yielding about a 100,000X reduction in training tokens. Moreover, we show for the first time that CAT-Q can quantize much larger pre-trained LLMs having 14B to 235B parameters into leading ternary models within just 8 to 60 hours on 8 A100-80GB GPUs. Code is available at https://github.com/IntelChina-AI/BitTern.

cs.CL↗

arXiv2026

Decentralized Stability-Constrained Optimal Power Flow for Inverter-Based Power Systems

Future inverter-dominated power systems feature higher variability and more stressed operating conditions, which motivates the consideration of stability in operational settings. Existing approaches to stability-constrained OPF often rely on eigenvalue calculation, global model information, or dynamic evaluation inside optimization formulation, which are computationally intensive and difficult to scale. This paper proposes the first decentralized stability-constrained OPF framework for inverter-based power systems. The key novelty lies in the incorporation of a class of algebraic decentralized small-signal stability criteria that admits tractable representations in steady-state variables and is therefore suitable for optimization. The decentralized stability condition is based on local voltage differences and enables clear theoretical and practical economic interpretation of the stability contribution from each inverter. We define a Nodal Stability Shadow Price (NSSP) for each inverter, and characterize the role of these stability constraints through their associated shadow prices, enabling a nodal interpretation of their economic impacts. It is proved that under active-power-only objectives in lossless networks, binding stability constraints may occur but will admit zero shadow prices if all other operational constraints are inactive. Most importantly, we reveal the importance of considering the opportunity cost of reactive power for inverter-based resources (IBRs) that have limited capacity. When reactive power costs are considered, stability constraints can carry strictly positive shadow prices and admit meaningful economic impacts.

math.OC↗

arXiv2026

Chain-of-Models Pre-Training: Rethinking Training Acceleration of Vision Foundation Models

In this paper, we present Chain-of-Models Pre-Training (CoM-PT), a novel performance-lossless training acceleration method for vision foundation models (VFMs). This approach fundamentally differs from existing acceleration methods in its core motivation: rather than optimizing each model individually, CoM-PT is designed to accelerate the training pipeline at the model family level, scaling efficiently as the model family expands. Specifically, CoM-PT establishes a pre-training sequence for the model family, arranged in ascending order of model size, called model chain. In this chain, only the smallest model undergoes standard individual pre-training, while the other models are efficiently trained through sequential inverse knowledge transfer from their smaller predecessors by jointly reusing the knowledge in the parameter space and the feature space. As a result, CoM-PT enables all models to achieve performance that is mostly superior to standard individual training while significantly reducing training cost, and this is extensively validated across 45 datasets spanning zero-shot and fine-tuning tasks. Notably, its efficient scaling property yields a remarkable phenomenon: training more models even results in higher efficiency. For instance, when pre-training on CC3M: i) given ViT-L as the largest model, progressively prepending smaller models to the model chain reduces computational complexity by up to 72%; ii) within a fixed model size range, as the VFM family scales across 3, 4, and 7 models, the acceleration ratio of CoM-PT exhibits a striking leap: from 4.13X to 5.68X and 7.09X. Since CoM-PT is naturally agnostic to specific pre-training paradigms, we open-source the code to spur further extensions in more computationally intensive scenarios, such as large language model pre-training.

cs.CV↗

arXiv2026

SliderQuant: Accurate Post-Training Quantization for LLMs

In this paper, we address post-training quantization (PTQ) for large language models (LLMs) from an overlooked perspective: given a pre-trained high-precision LLM, the predominant sequential quantization framework treats different layers equally, but this may be not optimal in challenging bit-width settings. We empirically study the quantization impact of different layers on model accuracy, and observe that: (1) shallow/deep layers are usually more sensitive to quantization than intermediate layers; (2) among shallow/deep layers, the most sensitive one is the first/last layer, which exhibits significantly larger quantization error than others. These empirical observations imply that the quantization design for different layers of LLMs is required on multiple levels instead of a single level shared to all layers. Motivated by this, we propose a new PTQ framework termed Sliding-layer Quantization (SliderQuant) that relies on a simple adaptive sliding quantization concept facilitated by few learnable parameters. The base component of SliderQuant is called inter-layer sliding quantization, which incorporates three types of novel sliding window designs tailored for addressing the varying quantization sensitivity of shallow, intermediate and deep layers. The other component is called intra-layer sliding quantization that leverages an incremental strategy to quantize each window. As a result, SliderQuant has a strong ability to reduce quantization errors across layers. Extensive experiments on basic language generation, zero-shot commonsense reasoning and challenging math and code tasks with various LLMs, including Llama/Llama2/Llama3/Qwen2.5 model families, DeepSeek-R1 distilled models and large MoE models, show that our method outperforms existing PTQ methods (including the latest PTQ methods using rotation transformations) for both weight-only quantization and weight-activation quantization.

cs.AI↗

arXiv2024

OIDM: An Observability-based Intelligent Distributed Edge Sensing Method for Industrial Cyber-Physical Systems

Industrial cyber-physical systems (ICPS) integrate physical processes with computational and communication technologies in industrial settings. With the support of edge computing technology, it is feasible to schedule large-scale sensors for efficient distributed sensing. In the sensing process, observability is the key to obtaining complete system states, and stochastic scheduling is more suitable considering uncertain factors in wireless communication. However, existing works have limited research on observability in stochastic scheduling. Targeting this issue, we propose an observability-based intelligent distributed edge sensing method (OIDM). Deep reinforcement learning (DRL) methods are adopted to optimize sensing accuracy and power efficiency. Based on the system's ability to achieve observability, we establish a bridge between observability and the number of successful sensor transmissions. Novel linear approximations of observability criteria are provided, and probabilistic bounds on observability are derived. Furthermore, these bounds guide the design of action space to achieve a probabilistic observability guarantee in stochastic scheduling. Finally, our proposed method is applied to the estimation of slab temperature in industrial hot rolling process, and simulation results validate its effectiveness.

eess.SY↗