arXiv subjects

Roozbeh Bostandoost

Publications and source records attributed to Roozbeh Bostandoost.

6 recordsLinked to original sources

arXiv2026

MetaKV: Adaptive KV Cache Compression for Constrained LLM Inference

Key--value (KV) cache compression is an effective way to reduce the memory overhead of large language model (LLM) inference, particularly for long-context workloads. However, existing compression methods make different trade-offs among accuracy, inference latency, and peak KV cache memory utilization, making a single fixed configuration unsuitable across different prompts and resource constraints. We introduce MetaKV, an adaptive framework that selects a KV cache compression configuration for each input prompt based on user-specified latency and peak memory budgets. MetaKV uses lightweight prediction models to estimate the end-to-end latency, peak memory, and probability of a correct response for each candidate configuration, and selects the configuration that best satisfies the latency-memory constraints while preserving accuracy. We evaluate MetaKV across ten configurations from three representative KV cache compression methods, KVQuant, H$_2$O, and RocketKV, together with an uncompressed FP16 configuration, on four datasets covering mathematics, science, commonsense reasoning, and reading comprehension. Across a wide range of latency and peak memory constraints, MetaKV consistently outperforms the best static configuration, improving constrained success rate (CSR), the fraction of prompts answered correctly while satisfying both constraints, by approximately 0.07 on average and up to 0.135. These results demonstrate the benefit of adapting KV cache compression to individual prompts and latency-memory constraints. Code is available at https://github.com/MichaelWang0505/MetaKV.git

cs.LG↗

arXiv2025

A Performance Analyzer for a Public Cloud's ML-Augmented VM Allocator

Cloud operators increasingly deploy multiple ML models in their VM allocation pipelines. In such settings, individually benign predictions can shift and compound, severely degrading performance. In a cloud provider's VM placement pipeline, CPU, memory, and lifetime prediction models jointly determine server count, live migration frequency, and network utilization; yet no existing approach can systematically stress-test how these models adversely interact. Deterministic adversarial analyzers cannot capture probabilistic ML behavior, so operators miss failures that arise only from correlated distributional shifts across models In SANJESH, we formulate a bi-level optimization that captures how the ML models behave statistically and uncovers how they adversely interact. The outer level searches over what predictions the ML models could produce under distributional uncertainty to find adversarial conditions; the inner level evaluates how the VM allocator behaves given those predictions. When we applied it to the operator's production traces, SANJESH uncovered scenarios that cause $4\times$ worse performance than the operators' evaluator detected.

cs.DC↗

arXiv2025

Quantifying the Carbon Reduction of DAG Workloads: A Job Shop Scheduling Perspective

Carbon-aware schedulers aim to reduce the operational carbon footprint of data centers by running flexible workloads during periods of low carbon intensity. Most schedulers treat workloads as single monolithic tasks, ignoring that many jobs, like video encoding or offline inference, consist of smaller tasks with specific dependencies and resource needs; however, knowledge of this structure enables opportunities for greater carbon efficiency. We quantify the maximum benefit of a dependency-aware approach for batch workloads. We model the problem as a flexible job-shop scheduling variant and use an offline solver to compute upper bounds on carbon and energy savings. Results show up to $25\%$ lower carbon emissions on average without increasing the optimal makespan (total job completion time) compared to a makespan-only baseline. Although in heterogeneous server setup, these schedules may use more energy than energy-optimal ones. Our results also show that allowing twice the optimal makespan nearly doubles the carbon savings, underscoring the tension between carbon, energy, and makespan. We also highlight key factors such as job structure and server count influence the achievable carbon reductions.

cs.DC↗

arXiv2024

LACS: Learning-Augmented Algorithms for Carbon-Aware Resource Scaling with Uncertain Demand

Motivated by an imperative to reduce the carbon emissions of cloud data centers, this paper studies the online carbon-aware resource scaling problem with unknown job lengths (OCSU) and applies it to carbon-aware resource scaling for executing computing workloads. The task is to dynamically scale resources (e.g., the number of servers) assigned to a job of unknown length such that it is completed before a deadline, with the objective of reducing the carbon emissions of executing the workload. The total carbon emissions of executing a job originate from the emissions of running the job and excess carbon emitted while switching between different scales (e.g., due to checkpoint and resume). Prior work on carbon-aware resource scaling has assumed accurate job length information, while other approaches have ignored switching losses and require carbon intensity forecasts. These assumptions prohibit the practical deployment of prior work for online carbon-aware execution of scalable computing workload. We propose LACS, a theoretically robust learning-augmented algorithm that solves OCSU. To achieve improved practical average-case performance, LACS integrates machine-learned predictions of job length. To achieve solid theoretical performance, LACS extends the recent theoretical advances on online conversion with switching costs to handle a scenario where the job length is unknown. Our experimental evaluations demonstrate that, on average, the carbon footprint of LACS lies within 1.2% of the online baseline that assumes perfect job length information and within 16% of the offline baseline that, in addition to the job length, also requires accurate carbon intensity forecasts. Furthermore, LACS achieves a 32% reduction in carbon footprint compared to the deadline-aware carbon-agnostic execution of the job.

cs.DC↗

arXiv2023

The War of the Efficiencies: Understanding the Tension between Carbon and Energy Optimization

Major innovations in computing have been driven by scaling up computing infrastructure, while aggressively optimizing operating costs. The result is a network of worldwide datacenters that consume a large amount of energy, mostly in an energy-efficient manner. Since the electric grid powering these datacenters provided a simple and opaque abstraction of an unlimited and reliable power supply, the computing industry remained largely oblivious to the carbon intensity of the electricity it uses. Much like the rest of the society, it generally treated the carbon intensity of the electricity as constant, which was mostly true for a fossil fuel-driven grid. As a result, the cost-driven objective of increasing energy-efficiency -- by doing more work per unit of energy -- has generally been viewed as the most carbon-efficient approach. However, as the electric grid is increasingly powered by clean energy and is exposing its time-varying carbon intensity, the most energy-efficient operation is no longer necessarily the most carbon-efficient operation. There has been a recent focus on exploiting the flexibility of computing's workloads -- along temporal, spatial, and resource dimensions -- to reduce carbon emissions, which comes at the cost of either performance or energy efficiency. In this paper, we discuss the trade-offs between energy efficiency and carbon efficiency in exploiting computing's flexibility and show that blindly optimizing for energy efficiency is not always the right approach.

cs.DC↗

arXiv2023

Near-optimal Online Algorithms for Joint Pricing and Scheduling in EV Charging Networks

With the rapid acceleration of transportation electrification, public charging stations are becoming vital infrastructure in a smart sustainable city to provide on-demand electric vehicle (EV) charging services. As more consumers seek to utilize public charging services, the pricing and scheduling of such services will become vital, complementary tools to mediate competition for charging resources. However, determining the right prices to charge is difficult due to the online nature of EV arrivals. This paper studies a joint pricing and scheduling problem for the operator of EV charging networks with limited charging capacity and time-varying energy cost. Upon receiving a charging request, the operator offers a price, and the EV decides whether to admit the offer based on its own value and the posted price. The operator then schedules the real-time charging process to satisfy the charging request if the EV admits the offer. We propose an online pricing algorithm that can determine the posted price and EV charging schedule to maximize social welfare, i.e., the total value of EVs minus the energy cost of charging stations. Theoretically, we prove the devised algorithm can achieve the order-optimal competitive ratio under the competitive analysis framework. Practically, we show the empirical performance of our algorithm outperforms other benchmark algorithms in experiments using real EV charging data.

cs.GT↗