Ravi Satya Durga Prasad Yenugula

arXiv2026

Wiring Beats Blending: Structure-Aware Compensation for Transformer Downscaling

Model families are trained size by size. Can a pretrained large model instead be converted into a smaller sibling? We study the 1.4B->410M conversion in Pythia end to end. Representations align strongly across sizes (ridge R^2=0.84); parameters align weakly. Dense weight projection is destructive; a bit-exact control places the fault in basis mixing, which breaks rotary, per-head, GELU, and LayerNorm structure. Residuals after the best-fit linear operator carry no learnable or transferable signal under shuffle controls, so conversion value lives in initialization. Matched-budget continued pre-training separates two independent levers: least-squares compensation (function lever, best zero-shot) and variance-preserving rescale (dynamics lever, best endpoints). Placement follows the architecture: compensation is well-posed exactly where no normalization sits between cut and read; norm-fronted paths take rescale. Compensation is a low-budget, token-efficiency win, not a universal one. At 30M tokens it beats the best subcloning variant on a width-reduced pair (84.0+-1.8 vs. 89.7+-3.7, 3/3 seeds) and a held-out depth-reduced pair (109.3 vs. 117.9, 3/3 seeds). Selection given the same activation statistics recovers under half of that gap (3/3 seeds): the gain is the re-fit, not the information. At 33x the budget the two reach parity (40.3+-0.3 vs. 40.3+-0.5, 3 seeds), both far ahead of from-scratch, which transfer always beats (up to 18x at low budget, narrowing at convergence and at the largest scale). At ~5x the donor scale (6.9B->1.4B) stacking both levers over-corrects, consistent with an ill-conditioned compensation solve at large width, pointing to dimension-aware regularization as a fix. The init also beats structured pruning plus distillation, the standard pipeline, at matched budget, and improves further combined with it. Code, checkpoints, and the frozen eval corpus are released.

cs.LG↗

arXiv2026

A Scalable Pipeline for LLM-Teacher Distillation Labeling: Work-Stealing Job Scheduling and Memory-Aware GPU Concurrency

Labeling large text corpora with LLM teachers has become a practical route to training data at scale. At millions of items, hand-labeling every batch is not feasible, and two questions dominate: what label quality a teacher buys per dollar, and how to keep a fleet of GPU workers busy under skewed, failure-prone workloads. We present a simple, reproducible pipeline that addresses both. First, a work-stealing ring pool: each worker owns a queue, drains it first, and then steals from ring successors, with exactly-once task claims via atomic conditional writes and crash tolerance via stale-claim sweeping. The claim protocol requires only a compare-and-set primitive from its storage layer; we implement it on a single SQLite file, which makes the reference implementation dependency-free and the experiments reproducible on one machine. Second, a memory-aware concurrency rule that sizes per-node parallelism by how many model copies fit on the GPU, so the same code runs safely across device sizes. Third, a relabeling benchmark methodology in which the teacher relabels a public dataset that already has gold labels, so quality reduces to an agreement measurement and cost follows from measured throughput. Under skewed load the pool sustains up to 3.4 times the throughput of static sharding while matching it at zero skew, loses 0 of 2,000 tasks when half the workers are killed mid-run (static sharding loses 953), and yields measured quality and cost points for an instruction-tuned teacher on irony and sentiment tasks. All experiments run on public data and commodity hardware; code, tests, and run logs are released.

cs.DC↗