arXiv · 2611.12737
Inference Validation for Regulated AI: A Governed Decision Pipeline for LLM Serving
Abstract
Deploying large language models in regulated industries requires more than picking the fastest inference engine: it requires evidence that a configuration meets contractual and legal obligations before it serves a single customer request. We describe Runway, an internal validation harness built at MasterControl that builds a governed decision pipeline around NVIDIA AIPerf. The key insight is separating concerns: AIPerf provides the trusted measurement, while Runway provides the experiment design that comes before it and the compliance, cost, and audit decisions that come after. We describe the architecture and the storage model that makes caching and audit one mechanism. We report three findings from Runway sweeps: TensorRT-LLM and vLLM rankings invert with concurrency; a newer, pricier GPU instance can be the cheaper one per request; and speculative decoding behaves differently across engines under load, so engine configuration can outweigh engine choice.
Explore related subjects
Keep this discovery
Explore connections, maps & timelines
Dev Akre, Roman Bobalo, Manoj Dobbali. 2026-11-11. Inference Validation for Regulated AI: A Governed Decision Pipeline for LLM Serving. https://searcharxiv.com/abs/2611.12737
Citations retain the original authors and DOI and include this site’s record URL. If this discovery service contributed to your research, you can also cite Searcharxiv by Indiciae: BibTeX · RIS.