R\'enyi Sharpness: A Novel Sharpness that Strongly Correlates with Generalization

Abstract

Sharpness (of the loss minima) is widely believed to be a good indicator of generalization of neural networks. Unfortunately, the correlation between existing sharpness measures and generalization is not as strong as expected, and sometimes even contradiction occurs. To address this problem, a key observation in this paper is: what really matters for generalization is the average spread (or unevenness) of the spectrum of loss Hessian $\mathbf{H}$. For this reason, conventional sharpness measures, such as trace sharpness $\operatorname{tr}(\mathbf{H})$, which cares about the average value of the spectrum, or max-eigenvalue sharpness $\lambda_{\max}(\mathbf{H})$, which concerns the maximum spread of the spectrum, are not sufficient to well predict generalization. To characterize the average spread of the Hessian spectrum, we leverage the notion of R\'enyi entropy in information theory, which captures the unevenness of a probability vector and can thus be extended to a general non-negative vector, such as the Hessian spectrum at loss minima. Specifically, we propose R\'enyi sharpness, defined as the negative of the R\'enyi entropy of loss Hessian $\mathbf{H}$. Extensive experiments demonstrate that R\'enyi sharpness exhibits strong and consistent correlation with generalization in various scenarios. Moreover, two generalization bounds with respect to R\'enyi sharpness are established by exploiting its desirable reparametrization invariance property. Finally, as an initial attempt to exploit R\'enyi sharpness for regularization, R\'enyi Sharpness Aware Minimization (RSAM) is proposed, where a variant of R\'enyi sharpness is used as the regularizer. RSAM is competitive with state-of-the-art SAM algorithms and far better than conventional SAM based on max-eigenvalue sharpness.