Dynamic self-ensemble on multi-exit residual vision transformer for small-data image classification

Abstract

Vision Transformer (ViT) has garnered increased attention for image classification in recent years. However, ViT is notorious for its data-hungry problem as it achieves high-performance results when only trained on large-size datasets. In this paper, we study how ViT specifically works on small-scale datasets with limited data where the availability of image data is often constrained in practice, especially in fields such as medical imaging. To mitigate the issue, we designed a multi-stage residual ViT, dubbed ReSEViT, with the self-ensemble method to utilize multi-level information from intermediate features. Due to the complex decision boundaries for classifications, relying on a single classifier can lead to prediction errors. Hence, we propose a new learning method for ViT, named self-ensemble learning, which ensembles multi-stage classifiers in a single model. Our self-ensemble comprises a novel approach, Dynamic Learner Selection (DLS), to adaptively select the importance of different stages. ReSEViT incorporates residual token embeddings which is a ResNet-like block to model effective local structures alongside with transformer blocks in each stage. The integration of residual tokens and self-ensemble learning into a multi-stage ViT highlights the reduction in redundancy, prediction error and variance while maintaining an efficient model size for small-scale image datasets.