arXiv subjects

Ming-Hung Chen

Publications and source records attributed to Ming-Hung Chen.

4 recordsLinked to original sources

arXiv2025

An Online Fragmentation-Aware Scheduler for Managing GPU-Sharing Workloads on Multi-Instance GPUs

Modern GPU workloads increasingly demand efficient resource sharing, as many jobs do not require the full capacity of a GPU. Among sharing techniques, NVIDIA's Multi-Instance GPU (MIG) offers strong resource isolation by enabling hardware-level GPU partitioning. However, leveraging MIG effectively introduces new challenges. First, resource contention persists due to shared components such as PCIe bandwidth. Second, GPU fragmentation becomes a critical issue, which is different from prior fine-grained GPU sharing work due to MIG's limited number of valid MIG configurations. Fragmentation arises not only from spatial discontinuity but also from rigid profile placement constraints, especially after job arrivals and terminations. To address these issues, we propose an online scheduling framework that integrates conditional load balancing, dynamic partitioning, and job migration. Our approach dynamically adapts job placement to minimize contention and reorganizes GPU allocations to combat both internal and external fragmentation. Experimental results show that our method significantly improves system efficiency. When all techniques are applied, the makespan improves by up to 35%.

cs.DC↗

arXiv2024

The infrastructure powering IBM's Gen AI model development

AI Infrastructure plays a key role in the speed and cost-competitiveness of developing and deploying advanced AI models. The current demand for powerful AI infrastructure for model training is driven by the emergence of generative AI and foundational models, where on occasion thousands of GPUs must cooperate on a single training job for the model to be trained in a reasonable time. Delivering efficient and high-performing AI training requires an end-to-end solution that combines hardware, software and holistic telemetry to cater for multiple types of AI workloads. In this report, we describe IBM's hybrid cloud infrastructure that powers our generative AI model development. This infrastructure includes (1) Vela: an AI-optimized supercomputing capability directly integrated into the IBM Cloud, delivering scalable, dynamic, multi-tenant and geographically distributed infrastructure for large-scale model training and other AI workflow steps and (2) Blue Vela: a large-scale, purpose-built, on-premises hosting environment that is optimized to support our largest and most ambitious AI model training tasks. Vela provides IBM with the dual benefit of high performance for internal use along with the flexibility to adapt to an evolving commercial landscape. Blue Vela provides us with the benefits of rapid development of our largest and most ambitious models, as well as future-proofing against the evolving model landscape in the industry. Taken together, they provide IBM with the ability to rapidly innovate in the development of both AI models and commercial offerings.

cs.DC↗

arXiv2022

EFloat: Entropy-coded Floating Point Format for Compressing Vector Embedding Models

In a large class of deep learning models, including vector embedding models such as word and database embeddings, we observe that floating point exponent values cluster around a few unique values, permitting entropy based data compression. Entropy coding compresses fixed-length values with variable-length codes, encoding most probable values with fewer bits. We propose the EFloat compressed floating point number format that uses a variable field boundary between the exponent and significand fields. EFloat uses entropy coding on exponent values and signs to minimize the average width of the exponent and sign fields, while preserving the original FP32 exponent range unchanged. Saved bits become part of the significand field increasing the EFloat numeric precision by 4.3 bits on average compared to other reduced-precision floating point formats. EFloat makes 8-bit and even smaller floats practical without sacrificing the exponent range of a 32-bit floating point representation. We currently use the EFloat format for saving memory capacity and bandwidth consumption of large vector embedding models such as those used for database embeddings. Using the RMS error as metric, we demonstrate that EFloat provides higher accuracy than other floating point formats with equal bit budget. The EF12 format with 12-bit budget has less end-to-end application error than the 16-bit BFloat16. EF16 with 16-bit budget has an RMS-error 17 to 35 times less than BF16 RMS-error for a diverse set of embedding models. When making similarity and dissimilarity queries, using the NDCG ranking metric, EFloat matches the result quality of prior floating point representations with larger bit budgets.

cs.LG↗

arXiv2013

Deadline is not Enough: How to Achieve Importance-aware Server-centric Data Centers via a Cross Layer Approach

Today's datacenters face important challenges for providing low-latency high-quality interactive services to meet user's expectation. For improving the application throughput, recent research works have embedded application deadline information into design of network flow schedule to meet the latency requirement. Here, arises a critical question: does application-level throughput mean providing better quality service? We note that there are usually a set of semantic related responses (or flows) for answering a query; and, some responses are highly correlative with the query while others do not. Thus, this observation motivates us to associate the importance of the contents with the application flows (or responses) in order to enhance the service quality. We first model the application importance maximization problem in a generic network and in a server-centric network. Since both of them are too complicated to be deployed in the real world, we propose the importance-aware delivery protocol, which is a distributed event-driven rate-based delivery control protocol, for server-centric datacenter networks. The proposed protocol is able to make use of the multiple disjoin paths of server-centric network, and jointly consider flow importance, flow size, and deadline to maximize the goodput of most-related semantic data of a query. Through real-data-based or synthetic simulations, the results show that our proposed protocol significantly outperforms D3 and MPTCP in terms of the precision at K and the sum of application-level importance.

cs.DC↗