SAGE: Saliency-Aware Generalist Encoder with Multi-level Feature Enhancement for Visual Recognition
Pretrained vision transformers (ViTs) have demonstrated strong transferability across visual recognition tasks. However, adapting a shared representation to diverse downstream tasks remains challenging, as different tasks demand varying levels of semantic abstraction and spatial precision. We present SAGE (Saliency-Aware Generalist Encoder), an encoder-side architecture that enhances pretrained ViT representations by integrating multi-depth features, register-token context, saliency-guided modulation, and aligned local views into a unified multi-scale feature pyramid. Evaluated with ViT-B and ViT-L backbones on six visual tasks across seven benchmarks, SAGE achieves 62.8 box AP, 55.3 mask AP, and 78.5 keypoint AP on COCO; 60.3 semantic segmentation mIoU and 44.9 instance segmentation AP on ADE20K; 94.76\% classification top-1 accuracy on TinyImageNet; and 81.10 oriented detection AP$_{50}$ on DIOR-R. Combined with a progressive two-stage training strategy, SAGE-L achieves the best performance among the selected reference methods in six of the seven benchmark settings, while remaining competitive on keypoint estimation against substantially larger models. Importantly, these tasks share the same SAGE encoder and multi-scale feature interface, while employing task-specific prediction heads. To the best of our knowledge, SAGE is the first publicly reported encoder to achieve highly competitive results on six visual tasks across seven benchmarks, demonstrating its generality and competitiveness.