Diagnosing Zero-Shot Forgetting in CLIP Fine-Tuning: Attention Drift Characterizes Adaptation, Representation Drift Predicts Transfer Degradation

Abstract

Fine-tuning CLIP on a downstream task typically reduces its zero-shot accuracy on other tasks, and quantifying this loss ordinarily requires a labeled transfer benchmark. We study whether the loss can be anticipated from unlabeled target images that are already available during fine-tuning. Four signals are compared: the entropy of CLS token attention, linear centered kernel alignment (CKA) with the pretrained model, embedding drift, and relative weight change. ViT-B/32 is adapted by full fine-tuning (Full FT) and by LoRA on EuroSAT and Oxford-IIIT Pets over four learning rates and three seeds, with an additional ViT-B/16 grid; DTD and GTSRB are held out as confirmation targets, and forgetting is measured as zero-shot retention on CIFAR-100, Food-101, and ImageNet-R. Within a fixed configuration, the change in attention entropy is strongly associated with forgetting (mean $|ρ|=0.860$). Its sign, however, depends on the dataset, the adaptation method, and the backbone, and the pooled correlation falls to 0.361. A block decomposition shows that this sign is determined almost entirely by blocks 7 to 12. CKA ranks runs by retention at $ρ=0.960$, and the ranking is already established after the first of six epochs ($ρ=0.959$). On the held-out targets, CKA reaches $ρ=0.95$, and the block account correctly predicts the sign of the attention change for both datasets. These results support a simple decision rule: CKA is suited to comparisons across configurations, whereas attention entropy is informative only within a single configuration. The evidence is limited to one model family and short training schedules.