Merge Now, Regret Later: The Hidden Cost of Model Merging Is Adversarial Transferability

Abstract

Model Merging (MM) has proven to be an effective alternative to multi-task learning, where several fine-tuned models are merged, without access to the tasks' training data, into one model that retains performance across different tasks. Recent works have explored the security of MM, showing how MM can confer robustness against various adversarial attacks. However, none of them has sufficiently explored its impact on transfer attacks using transferable adversarial examples. In this work, we study the effect of MM on the transferability of adversarial examples. We perform comprehensive evaluations and statistical analysis consisting of eight MM methods, seven datasets, and six attack methods, sweeping over 336 distinct attack settings. Through it, we first challenge the prevailing notion of MM conferring free adversarial robustness, and show that MM cannot reliably defend against transfer attacks, with over 80% transfer rate. Moreover, we reveal two key insights for machine-learning practitioners regarding MM and transferability for a robust system design: (1) stronger MM methods increase vulnerability to transfer attacks and (2) mitigating representation bias increases vulnerability to transfer attacks. We also find weight averaging as an exception to (1), i.e., it is found to be the most vulnerable method to transfer attacks, despite being the weakest MM method. Finally, we analyze the underlying reasons for these findings, show how an adversary with limited information could launch an attack, and provide potential solutions. These findings offer actionable insights for deploying MM in security-sensitive machine-learning systems.