VersaQ-3D: Architecture Support for Visual Geometry Grounded Transformers via Versatile Quantization

Abstract

3D reconstruction and view synthesis are fundamental to AR/VR, robotics, and digital twins. The Visual Geometry Grounded Transformer (VGGT) enables strong feed-forward 3D reconstruction while its billion-parameter scale limits on-device deployment. LLM-oriented quantization methods fail on VGGT due to saturated activation channels that resist low-bit quantization and diverse 3D semantics that impede calibration. VGGT further poses hardware challenges from multi-precision architecture support and long-sequence global attention with excessive memory demands. We propose VersaQ-3D, an algorithm-architecture co-design framework for efficient VGGT inference. At the algorithm level, we present the first calibration-free, input-agnostic quantization method for VGGT, leveraging transform coding to suppress outliers and preserve structural weight features, enabling robust low-bit inference down to 4 bits. At the architecture level, we design a reconfigurable accelerator with a hierarchical multi-precision compute unit (BF16/INT8/INT4) that executes both linear and non-linear operators within a shared systolic datapath, reducing end-to-end latency by 77%. A two-stage recomputation-based tiling strategy further cuts runtime by 7% by alleviating on-chip memory pressure for long-sequence attention. Evaluations across various datasets show that VersaQ-3D incurs negligible accuracy loss at W4A8 and consistently achieves leading accuracy at W4A4 over prior quantization methods across diverse scenes. The co-designed accelerator delivers 5.4$\times$-22.0$\times$ speedup over edge GPUs and 2.2$\times$-3.0$\times$ over prior quantization-based accelerators under iso-PE-area comparison, enabling instant and energy-efficient feed-forward 3D reconstruction on edge devices.