arXiv subjects

Junhan Liao

Publications and source records attributed to Junhan Liao.

3 recordsLinked to original sources

arXiv2025

DOPD: A Dynamic PD-Disaggregation Architecture for Maximizing Goodput in LLM Inference Serving

To meet strict Service-Level Objectives (SLOs),contemporary Large Language Models (LLMs) decouple the prefill and decoding stages and place them on separate GPUs to mitigate the distinct bottlenecks inherent to each phase. However, the heterogeneity of LLM workloads causes producerconsumer imbalance between the two instance types in such disaggregated architecture. To address this problem, we propose DOPD (Dynamic Optimal Prefill/Decoding), a dynamic LLM inference system that adjusts instance allocations to achieve an optimal prefill-to-decoding (P/D) ratio based on real-time load monitoring. Combined with an appropriate request-scheduling policy, DOPD effectively resolves imbalances between prefill and decoding instances and mitigates resource allocation mismatches due to mixed-length requests under high concurrency. Experimental evaluations show that, compared with vLLM and DistServe (representative aggregation-based and disaggregationbased approaches), DOPD improves overall system goodput by up to 1.5X, decreases P90 time-to-first-token (TTFT) by up to 67.5%, and decreases P90 time-per-output-token (TPOT) by up to 22.8%. Furthermore, our dynamic P/D adjustment technique performs proactive reconfiguration based on historical load, achieving over 99% SLOs attainment while using less additional resources.

cs.DC↗

arXiv2025

Cloud Native System for LLM Inference Serving

Large Language Models (LLMs) are revolutionizing numerous industries, but their substantial computational demands create challenges for efficient deployment, particularly in cloud environments. Traditional approaches to inference serving often struggle with resource inefficiencies, leading to high operational costs, latency issues, and limited scalability. This article explores how Cloud Native technologies, such as containerization, microservices, and dynamic scheduling, can fundamentally improve LLM inference serving. By leveraging these technologies, we demonstrate how a Cloud Native system enables more efficient resource allocation, reduces latency, and enhances throughput in high-demand scenarios. Through real-world evaluations using Kubernetes-based autoscaling, we show that Cloud Native architectures can dynamically adapt to workload fluctuations, mitigating performance bottlenecks while optimizing LLM inference serving performance. This discussion provides a broader perspective on how Cloud Native frameworks could reshape the future of scalable LLM inference serving, offering key insights for researchers, practitioners, and industry leaders in cloud computing and artificial intelligence.

cs.DC↗

arXiv2025

Auto-scaling Approaches for Microservice Applications: A Survey and Taxonomy

Microservice applications are created as loosely coupled application components and they leverage cloud elasticity to reduce costs and increase development speed. However, microservice applications exhibit complex interactions among dynamically evolving services and highly variable workloads, posing significant challenges to auto-scaling mechanisms. Key issues include service dependency management, performance profiling, anomaly detection, workload characterization, and fine-grained resource allocation. To address these challenges, recent auto-scaling approaches leverage historical and runtime data to adapt resource provisioning and optimize system efficiency. Since 2018, marked by the graduation of Kubernetes as the first Cloud Native Computing Foundation (CNCF) project, microservice applications have been widely deployed on standardized orchestration platforms, fundamentally shifting auto-scaling from coarse-grained to service-level, dependency-aware strategies. Accordingly, this paper surveys state-of-the-art auto-scaling approaches for microservice applications since 2018 and presents a taxonomy along five dimensions: infrastructure, architecture, scaling methods, optimization objectives, and behavior modeling. These perspectives collectively target key objectives, including resource efficiency, cost efficiency, and Service Level Agreement (SLA) assurance, aiming to balance system optimization with SLA compliance. We further present a comprehensive comparison and in-depth analysis of representative approaches, examining their core features, strengths, limitations, and applicable scenarios, as well as their performance across diverse environments and workload conditions.

cs.DC↗