arXiv subjects

Zhang Cao

Publications and source records attributed to Zhang Cao.

3 recordsLinked to original sources

arXiv2026

LUMEN: Coordinated Failure Recovery for Distributed LLM Serving

Modern large language model (LLM) serving clusters distribute inference requests across multiple worker processes on different GPUs, but failures are prevalent at scale. When a worker fails, the cluster simultaneously loses the failed worker's GPU-resident key-value (KV) caches and serving capacity, leaving surviving workers to absorb the redirected traffic while re-running interrupted requests from scratch. Existing fault-tolerant systems either restart interrupted requests from scratch or restore KV caches from checkpoints stored on a fixed neighboring worker, but both approaches route recovery work without considering current cluster load and leave the recovering worker idle during model reload. We present LUMEN, a fault-tolerant LLM serving system that treats recovery as a load-aware coordination problem across three decision points: checkpoint placement before failures, interrupted-request distribution at failure time, and serving capacity restoration during model reload. We evaluate LUMEN using both prototype experiments and large-scale simulations and demonstrate significant improvements in serving and recovery times.

cs.DC↗

arXiv2024

Harnessing Your DRAM and SSD for Sustainable and Accessible LLM Inference with Mixed-Precision and Multi-level Caching

Although Large Language Models (LLMs) have demonstrated remarkable capabilities, their massive parameter counts and associated extensive computing make LLMs' deployment the main part of carbon emission from nowadays AI applications. Compared to modern GPUs like H$100$, it would be significantly carbon-sustainable if we could leverage old-fashioned GPUs such as M$40$ (as shown in Figure 1, M$40$ only has one third carbon emission of H$100$'s) for LLM servings. However, the limited High Bandwidth Memory (HBM) available on such GPU often cannot support the loading of LLMs due to the gigantic model size and intermediate activation data, making their serving challenging. For instance, a LLaMA2 model with $70$B parameters typically requires $128$GB for inference, which substantially surpasses $24$GB HBM in a $3090$ GPU and remains infeasible even considering the additional $64$GB DRAM. To address this challenge, this paper proposes a mixed-precision with a model modularization algorithm to enable LLM inference on outdated hardware with resource constraints. (The precision denotes the numerical precision like FP16, INT8, INT4) and multi-level caching (M2Cache).) Specifically, our M2Cache first modulizes neurons in LLM and creates their importance ranking. Then, it adopts a dynamic sparse mixed-precision quantization mechanism in weight space to reduce computational demands and communication overhead at each decoding step. It collectively lowers the operational carbon emissions associated with LLM inference. Moreover, M2Cache introduces a three-level cache management system with HBM, DRAM, and SSDs that complements the dynamic sparse mixed-precision inference. To enhance communication efficiency, M2Cache maintains a neuron-level mixed-precision LRU cache in HBM, a larger layer-aware cache in DRAM, and a full model in SSD.

cs.LG↗

arXiv2020

Relative Entropy Regularised TDLAS Tomography for Robust Temperature Imaging

Tunable Diode Laser Absorption Spectroscopy (TDLAS) tomography has been widely used for in situ combustion diagnostics, yielding images of both species concentration and temperature. The temperature image is generally obtained from the reconstructed absorbance distributions for two spectral transitions, i.e. two-line thermometry. However, the inherently ill-posed nature of tomographic data inversion leads to noise in each of the reconstructed absorbance distributions. These noise effects propagate into the absorbance ratio and generate artefacts in the retrieved temperature image. To address this problem, we have developed a novel algorithm, which we call Relative Entropy Tomographic RecOnstruction (RETRO), for TDLAS tomography. A relative entropy regularisation is introduced for high-fidelity temperature image retrieval from jointly reconstructed two-line absorbance distributions. We have carried out numerical simulations and proof-of-concept experiments to validate the proposed algorithm. Compared with the well-established Simultaneous Algebraic Reconstruction Technique (SART), the RETRO algorithm significantly improves the quality of the tomographic temperature images, exhibiting excellent robustness against TDLAS tomographic measurement noise. RETRO offers great potential for industrial field applications of TDLAS tomography, where it is common for measurements to be performed in very harsh environments.

eess.IV↗