FedSEAM: Federated Continual Learning of LLMs via Subspace Expert Agreement Merging

Abstract

Federated continual learning trains a model on a sequence of tasks across several clients without sharing their data while retaining knowledge of earlier tasks. Sparse mixture-of-experts models support this setting by adapting selected parameters on a frozen backbone. However, clients learn different expert content as tasks evolve, making it difficult to share knowledge across clients without disrupting task-specific and client-specific specialization. Matching tile coordinates alone does not establish whether their learned content is compatible. We present FedSEAM, a federated continual learning framework that organizes task knowledge into sparse experts and selectively shares their content across tasks and clients. On each client, task-aware tile selection allocates capacity to a new task, while a subspace-based compatibility test determines which existing expert tiles to reuse and where to introduce fresh capacity. On the server, Subspace Expert Agreement Merging (SEAM) compares client updates as subspaces and moves each client towards consensus only along agreed-upon directions, leaving the rest of its update unchanged. We identify two remaining sources of forgetting, routing drift and merge erasure, and address them with three protections, one of which exactly preserves tile components in a selected prior-overlap subspace at every merge. We evaluate FedSEAM on TRACE using Qwen3-4B and Llama-3.2-3B, with component-wise ablations and studies of data size to assess its balance between learning new tasks and retaining prior knowledge.