John P. Shen | Searcharxiv

arXiv2026

A-Graph: A Unified Graph Representation for Cross-Stack Cost Modeling

As computer systems continue to diversify across technologies, architectures, applications, and beyond, the relevant design space has become larger and more complex. Given such trends, design space exploration (DSE) at early stages is critical to ensure agile development towards optimal performance and cost. Industry-grade EDA tools directly take in RTL code and report accurate results, but do not perform DSE. Recent works have attempted to explore the design space via simulation. However, most of these works are domain-specific and constrain the space that users are allowed to explore, offering limited flexibility between technologies, architecture, and applications. Moreover, they often demand high domain expertise to ensure high accuracy. To enable simulation that is agnostic to technology, architecture, and application at any granularity, we introduce Architecture-Graph (Agraph), a graph that unifies the system representation surrounding any arbitrary application, software, architecture, and circuit. Such a unified representation distinguishes Agraph from prior works, which focus on a single stack, allowing users to freely explore the design space across system stacks. To fully unleash the potential of Agraph, we further present Archx, a framework that implements Agraph. Archx is user-friendly in two ways. First, Archx has an easy-to-use programming interface to automatically generate and sweep design points under user constraints, boosting the programmability. Second, Archx adopts scope-based metric retrieval to analyze and understand each design point at any user-preferred hierarchy, enhancing the explainability. We conduct case studies that demonstrate Agraph's generalization across technologies, architecture, and applications with high simulation accuracy. Overall, we argue that Agraph and Archx serve as a foundation to simulate both performance and cost at will.

cs.PF↗

arXiv2026

TaxBreak: Unmasking the Hidden Costs of LLM Inference Through Overhead Decomposition

Large Language Model (LLM) inference is widely used in interactive assistants and agentic systems. In latency-sensitive deployments, inference time can become dominated by host-side overheads. Existing approaches typically expose this cost only as an aggregate residual or a launch/queue metric, which is often insufficient to identify which execution layer should be optimized. This work presents TaxBreak, a trace-driven methodology for decomposing host-visible orchestration overhead into three components: framework translation time, CUDA library translation time, and kernel launch-path time. We validate TaxBreak on NVIDIA H100 and H200 systems and use it to derive our proposed Host-Device Balance Index (HDBI), a boundedness summary index that relates device-active execution to host-visible orchestration. Across representative dense and mixture-of-experts workloads in both prefill and decode, we show that aggregate latency, GPU inactivity, or boundedness ratios alone can obscure the dominant optimization target. TaxBreak instead distinguishes cases where optimization should reduce software-stack overhead from cases where the primary win comes from reducing device-side work. We further show that MoE models dispatch 8-11x more kernels per output token than dense models, and that for such host-bound workloads, CPU single-thread performance is a first-order parameter: a faster host CPU reduces orchestration overhead by 10-29% and improves end-to-end latency by up to 14%, even when paired with a slower-clocked GPU. These results position TaxBreak as a diagnostic tool for assessing whether optimization effort should target the software stack or the device-side workload execution.

cs.DC↗