arXiv · 2611.18587
Quantize, Qualify, Rerank: A Recipe for Compressing LLMs Without Losing Quality
Abstract
Compressing large language models lowers serving cost, but a single automatic metric may miss quality loss. For the WMT26 Model Compression constrained track, we compress google/gemma-3-12b-it (24.4 GB in bf16) for three translation directions on one H100. Paired bootstrap analysis exposes metric disagreement on 4-bit quantization; guided by WMT24 human meta-evaluation, we use MetricX24-XXL as primary and CometKiwi-XXL as a cross-check. Removing the unused vision stack and non-task vocabulary and using int8 weights or fp8 activations causes no resolved loss, whereas int4 leaves a small residual. QLoRA healing overfits its calibration domain. A Cometoid best-of-4 reranker at $T=0.3$ improves both the 8-bit and 4-bit systems under WMT25 test set. On post-submission WMT26, both remain competitive with bf16 under reference-free metrics; the small difference between the reranked variants is treated as noise.
Explore related subjects
Keep this discovery
Explore connections, maps & timelines
Thamme Gowda. 2026-08-12. Quantize, Qualify, Rerank: A Recipe for Compressing LLMs Without Losing Quality. https://searcharxiv.com/abs/2611.18587
Citations retain the original authors and DOI and include this site’s record URL. If this discovery service contributed to your research, you can also cite Searcharxiv by Indiciae: BibTeX · RIS.