Stepsize modulation for cross entropy from local quadratic optimization

Abstract

Two methods are presented that perturb a given learning rate schedule in stochastic gradient descent with cross entropy loss. Both methods use an extra forward pass to estimate various derivatives of the minibatch loss. The first method selects a value $η$ by linearising the softmax values and minimizing the squared error $\tfrac 1 2 ||y - q - ηq'||_F^2$, where $q$ is the matrix representing the softmax values of the minibatch and $y$ is the matrix with one-hot encodings of the labels. The second method considers a quadratic Taylor approximation of the minibatch loss under the outer approximation of the Hessian. Thus, it minimizes $L(0) + ηL'(0) + \tfrac 1 2 η^2 L''(0)$. Here, the derivatives $L'$ and $L''$ are computed from the derivatives of the logits (using the outer approximation of the Hessian of the loss). We observe that both methods give very similar values of $η$. Finally, these estimated $η$ are used to slightly increase or decrease the learning rate of an existing schedule. The method is evaluated on Cifar10 and Cifar100 datasets for training Resnets, where they decrease the loss faster than the schedule that is perturbed and increase monotonicity. For Cifar10 it also leads to a small improvement of test accuracy, but in Cifar100 test accuracy remains the same.