Searcharxiv⌕ Search
Home/Archive/arXiv

arXiv · 2610.16890

FORGE: Procedural Weight Generation for Sparse MoE Inference

Chanyoung Moon·Travis Pence·Daisuke Yamada·Jiaqi Mo·Yanxin Zhang·Vikas Singh·Karthikeyan Sankaralingam
View original source ↗Read source document ↗Export metadata ↓

Abstract

We introduce FORGE (Fast On-chip Rotor Generation for Experts), a procedural weight generation system for Sparse Mixture-of-Experts (MoE) inference. Rather than storing and loading every expert weight tensor from HBM, we show how a compact accelerator inside each Streaming Multiprocessor can regenerate them on-the-fly from $O(\log^2 d)$ rotor parameters per projection. Per replaced layer, gigabytes of stored expert weights are approximated with under a megabyte of seeds. This addresses the dual memory bottleneck of autoregressive Sparse MoE decode, where per-expert FFN operations are HBM-bandwidth-bound at small routed-token counts and the aggregate expert footprint forces multi-node Expert Parallelism (EP) with all-to-all traffic. We realize the approach as the Rotor Generation Unit (RGU), a fixed-function datapath that generates FFN weight tiles without competing with Tensor Core execution. With 30% of the MoE FFN layers replaced across five Sparse MoE models, we estimate 1.3-2.8x higher system throughput at matched GPU count and 1.10-1.15x lower per-user decode latency at short context. In exchange, downstream accuracy on three model families falls below that of the same model fine-tuned without rotor replacement, and the replacement ratio controls this tradeoff. FORGE also expands KV cache capacity by 1.2-2.6x. It is agnostic to the bit width at which weights are stored, so it can compose with quantization. The RGU is fully implemented/verified in SystemVerilog at ~0.9% SM area overhead (0.032 mm^2/SM, 7nm).

Explore related subjects

cs.ARcs.DCcs.LG

Keep this discovery

Explore connections, maps & timelines

BibTeXRIS

Chanyoung Moon, Travis Pence, Daisuke Yamada, Jiaqi Mo, Yanxin Zhang, Vikas Singh, Karthikeyan Sankaralingam. 2026-10-12. FORGE: Procedural Weight Generation for Sparse MoE Inference. https://doi.org/10.1145/3845814.3850033

Cite the original work for its findings. Save a collection to share your selection of sources.

KEEP EXPLORING

Related papers

arXiv2026

MFSA: A Multi-Format Systolic Array with Native Micro Scaling Support

AI workloads increasingly employ low-precision formats such as BF16 / FP16, FP8 variants, FP4, Micro Scaling (MX), and INT modes across models, layers, and execution stages to balance accuracy, memory footprint, and throughput, which in turn requires hardware to support these formats in a flexible and efficient way. We present MFSA, a weight-stationary multi-format systolic array that achieves area efficiency through two architectural choices. At the PE level, MFSA uses dot-product-accumulator-based processing elements extended from an existing open-source trans-precision FPU to support reduced-precision formats (FP32/FP16/BF16/CFP8/FP4/INT16/INT8/INT4) with a shared datapath and proportional throughput scaling. At the array level, MFSA employs Online Block-Scale Rebasing (OBSR) for block-scaled formats, which folds each block scale into the running FP32 partial sum at block boundaries without adding a second cross-block accumulator, thereby reducing the additional overhead of block-scaling support. In TSMC 28 nm at 1 GHz, matched-throughput BF16/FP8/FP4 cascaded FMA-PE baselines are 1.5x/2.5x/4.5x larger than the single-format DPA-PE, the gap growing with subword parallelism. A routed 16x16 MFSA reaches a peak 0.5/1/2/4 TOPS of FP32/BF16/FP8/FP4, and OBSR cuts the incremental MX overhead by 64.2% in area and 67.5% in energy versus a naive MX implementation baseline, without hurting the numerical stability on real LLM traces. Adding support for all other formats based on a BF16+CFP8 systolic array costs 25.7% additional area. OBSR also extends to non-power-of-two block scales: folding the fractional part of an E4M3 scale into the MFPE dot product supports NVFP4 alongside OCP MX for 11.8% area and 7.3% power over a CFP8+FP4 base, less than half the cost of a naive implementation.

cs.AR↗
arXiv2026

ESL-former: Event Sparse Linear Transformer for FPGA-Accelerated Malware Detection

Transformer based intrusion detection has shown strong potential for identifying unknown malware variants, but its deployment on resource constrained edge devices remains difficult due to the high computational and memory cost of self attention. This paper presents an event sparse Linear Transformer framework for malware detection on FPGA edge platforms. Instead of relying on an explicit attention matrix, the proposed method derives token and channel importance directly from the accumulated key value state of Linear Transformer. A contribution aware scoring function is introduced to measure the perturbation of each token or kernel channel to the linear attention state, allowing redundant tokens and low contribution channels to be removed without constructing quadratic attention maps. To translate sparsity into practical hardware acceleration, the proposed design further replaces conventional binary mask based execution with a Top K active token buffer and a hardware compaction unit. The selected active tokens and channels are reorganized into dense fixed length data streams, enabling downstream processing elements to execute regular multiply accumulate operations with reduced redundant memory access and control overhead. This design is suitable for low power FPGA based malware detection, where both latency and on chip resource usage are critical. Experimental evaluation demonstrates that the proposed sparse Linear Transformer can reduce hardware cost while preserving malware classification accuracy.

cs.AR↗
arXiv2026

CAFE: Conducting-Cell-Aware Adaptive Differential Encoding for Robust DNN Inference in FeFET-Based Compute-in-Memory

Ferroelectric field-effect transistor (FeFET)-based compute-in-memory (CIM) promises energy-efficient deep neural network (DNN) inference, but device variation can severely degrade bit-sliced CIM accuracy. Prior works either rely on hardware modifications or treat such variation as generic noise, overlooking a key FeFET asymmetry: negligible off-state leakage makes errors dominated by conducting on-state cells. This asymmetry is amplified by two's-complement sign extension, which maps small negative values to high-significance bit slices and magnifies their errors during shift-and-add reconstruction. In this paper, we propose CAFE, a conducting-cell-aware adaptive differential encoding framework built upon this FeFET-specific asymmetry. CAFE introduces C$^2$AM, a bit-significance-weighted conducting-cell activity metric that predicts output deviation and inference accuracy degradation under FeFET device variation. Guided by C$^2$AM, CAFE applies differential weight and activation encoding to eliminate sign-extension-induced conducting-cell amplification. To balance robustness and cost, CAFE performs type-wise encoding enumeration, layer-wise NSGA-II search, and post-GA pruning to allocate robust encodings where the robustness-cost trade-off is favorable. Across representative vision and language workloads, CAFE reduces C$^2$AM by up to $28.6\times$; in the main deployment study, it improves accuracy over two's-complement CIM by more than 87 percentage points and approaches full-differential robustness at $1.86$-$2.78\times$ normalized cost, compared with $3.06\times$ for uniform full-differential encoding.

cs.AR↗