arXiv subjects

Tianyang Cai

Publications and source records attributed to Tianyang Cai.

5 recordsLinked to original sources

arXiv2025

Efficiently Training A Flat Neural Network Before It has been Quantizated

Post-training quantization (PTQ) for vision transformers (ViTs) has garnered significant attention due to its efficiency in compressing models. However, existing methods typically overlook the relationship between a well-trained NN and the quantized model, leading to considerable quantization error for PTQ. However, it is unclear how to efficiently train a model-agnostic neural network which is tailored for a predefined precision low-bit model. In this paper, we firstly discover that a flat full precision neural network is crucial for low-bit quantization. To achieve this, we propose a framework that proactively pre-conditions the model by measuring and disentangling the error sources. Specifically, both the Activation Quantization Error (AQE) and the Weight Quantization Error (WQE) are statistically modeled as independent Gaussian noises. We study several noise injection optimization methods to obtain a flat minimum. Experimental results attest to the effectiveness of our approach. These results open novel pathways for obtaining low-bit PTQ models.

cs.CV↗

arXiv2025

MEC-Quant: Maximum Entropy Coding for Extremely Low Bit Quantization-Aware Training

Quantization-Aware Training (QAT) has driven much attention to produce efficient neural networks. Current QAT still obtains inferior performances compared with the Full Precision (FP) counterpart. In this work, we argue that quantization inevitably introduce biases into the learned representation, especially under the extremely low-bit setting. To cope with this issue, we propose Maximum Entropy Coding Quantization (MEC-Quant), a more principled objective that explicitly optimizes on the structure of the representation, so that the learned representation is less biased and thus generalizes better to unseen in-distribution samples. To make the objective end-to-end trainable, we propose to leverage the minimal coding length in lossy data coding as a computationally tractable surrogate for the entropy, and further derive a scalable reformulation of the objective based on Mixture Of Experts (MOE) that not only allows fast computation but also handles the long-tailed distribution for weights or activation values. Extensive experiments on various tasks on computer vision tasks prove its superiority. With MEC-Qaunt, the limit of QAT is pushed to the x-bit activation for the first time and the accuracy of MEC-Quant is comparable to or even surpass the FP counterpart. Without bells and whistles, MEC-Qaunt establishes a new state of the art for QAT.

cs.CV↗

arXiv2025

Stabilizing Quantization-Aware Training by Implicit-Regularization on Hessian Matrix

Quantization-Aware Training (QAT) is one of the prevailing neural network compression solutions. However, its stability has been challenged for yielding deteriorating performances as the quantization error is inevitable. We find that the sharp landscape of loss, which leads to a dramatic performance drop, is an essential factor that causes instability. Theoretically, we have discovered that the perturbations in the feature would bring a flat local minima. However, simply adding perturbations into either weight or feature empirically deteriorates the performance of the Full Precision (FP) model. In this paper, we propose Feature-Perturbed Quantization (FPQ) to stochastically perturb the feature and employ the feature distillation method to the quantized model. Our method generalizes well to different network architectures and various QAT methods. Furthermore, we mathematically show that FPQ implicitly regularizes the Hessian norm, which calibrates the smoothness of a loss landscape. Extensive experiments demonstrate that our approach significantly outperforms the current State-Of-The-Art (SOTA) QAT methods and even the FP counterparts.

cs.CV↗

arXiv2025

In-Distribution Consistency Regularization Improves the Generalization of Quantization-Aware Training

Although existing Quantization-Aware Training (QAT) methods intensively depend on knowledge distillation to guarantee performance, QAT still suffers from severe performance drop. The experiments have shown that vanilla quantization is sensitive to the perturbation from both the input and weights. Therefore, we assume that the generalization ability of QAT is predominantly caused by both the intrinsic instability (training time) and the limited generalization ability (testing time). In this paper, we address both issues from a new perspective by leveraging Consistency Regularization (CR) to improve the generalization ability of QAT. Empirical results and theoretical analysis verify that CR would bring a good generalization ability to different network architectures and various QAT methods. Extensive experiments demonstrate that our approach significantly outperforms current state-of-the-art QAT methods and even the FP counterparts. On CIFAR-10, the proposed method improves by 3.79% compared to the baseline method using ResNet18, and improves by 3.84% compared to the baseline method using the lightweight model MobileNet.

cs.CV↗

arXiv2023

Catch Me If You Can: A New Low-Rate DDoS Attack Strategy Disguised by Feint

While collaborative systems provide convenience to our lives, they also face many security threats. One of them is the Low-rate Distributed Denial-of-Service (LDDoS) attack, which is a worthy concern. Unlike volumetric DDoS attacks that continuously send large volumes of traffic, LDDoS attacks are more stealthy and difficult to be detected owing to their low-volume feature. Due to its stealthiness and harmfulness, LDDoS has become one of the most destructive attacks in cloud computing. Although a few LDDoS attack detection and defense methods have been proposed, we observe that sophisticated LDDoS attacks (being more stealthy) can bypass some of the existing LDDoS defense methods. To verify our security observation, we proposed a new Feint-based LDDoS (F-LDDoS) attack strategy. In this strategy, we divide a Pulse Interval into a Feinting Interval and an Attack Interval. Unlike the previous LDDoS attacks, the bots also send traffic randomly in the Feinting Interval, thus disguise themselves as benign users during the F-LDDoS attack. In this way, although the victim detects that it is under an LDDoS attack, it is difficult to locate the attack sources and apply mitigation solutions. Experimental results show that F-LDDoS attack can degrade TCP bandwidth 6.7%-14% more than the baseline LDDoS attack. Besides, F-LDDoS also reduces the similarities between bot traffic and aggregated attack traffic, and increases the uncertainty of packet arrival. These results mean that the proposed F-LDDoS is more effective and more stealthy than normal LDDoS attacks. Finally, we discuss the countermeasures of F-LDDoS to draw the attention of defenders and improve the defense methods.

cs.CR↗