The pretraining domain outweighs the training objective in setting the privacy-utility trade-off of differentially private medical image analysis
Differential privacy protects the patients whose images train medical imaging models, but it lowers diagnostic accuracy, and the initialization is the strongest known remedy. Practice increasingly favors large generic self-supervised encoders. Yet the pretraining objective and the pretraining domain are confounded in existing comparisons, so which one preserves utility under privacy is unknown, and the pretraining corpus is treated as public even when it holds patient images. We trained ConvNeXt classifiers with differentially private stochastic gradient descent from five initializations that vary the objective and the domain independently, at four privacy budgets and without privacy, and evaluated them locally on more than 590,000 chest radiographs from five external datasets in four countries. Supervised pretraining on chest radiographs ranked first in 24 of 25 dataset and budget combinations. Its lead over ImageNet grew from 2.5 to 14.6 points of macro-averaged area under the receiver operating characteristic curve as the budget tightened, and the domain effect exceeded the objective effect by a factor of 2.2 to 3.4. Pretraining that corpus privately cost about 5 points and, under privacy, still beat every public initialization. Low-rank adaptation removed about half the residual gap, and in-domain pretraining raised the worst-performing demographic subgroup. Under privacy, what a model was pretrained on outweighs how it was pretrained.