arXiv subjects

Hans J. Skaug

Publications and source records attributed to Hans J. Skaug.

7 recordsLinked to original sources

arXiv2021

A Comparison of the Delta Method and the Bootstrap in Deep Learning Classification

We validate the recently introduced deep learning classification adapted Delta method by a comparison with the classical Bootstrap. We show that there is a strong linear relationship between the quantified predictive epistemic uncertainty levels obtained from the two methods when applied on two LeNet-based neural network classifiers using the MNIST and CIFAR-10 datasets. Furthermore, we demonstrate that the Delta method offers a five times computation time reduction compared to the Bootstrap.

cs.LG↗

arXiv2021

Efficient Computation of Hessian Matrices in TensorFlow

The Hessian matrix has a number of important applications in a variety of different fields, such as optimzation, image processing and statistics. In this paper we focus on the practical aspects of efficiently computing Hessian matrices in the context of deep learning using the Python scripting language and the TensorFlow library. We define a general feed-forward neural network model and show how to efficiently compute two quantities: the cost function's exact Hessian matrix, and the cost function's approximate Hessian matrix, known as the Outer Product of Gradients (OPG) matrix. Furthermore, as the number of parameters (weights and biases) in deep learning usually is very large, we show how to reduce the quadratic space complexity by an efficient implementation based on approximate eigendecompositions.

cs.LG↗

arXiv2021

Epistemic Uncertainty Quantification in Deep Learning Classification by the Delta Method

The Delta method is a classical procedure for quantifying epistemic uncertainty in statistical models, but its direct application to deep neural networks is prevented by the large number of parameters $P$. We propose a low cost variant of the Delta method applicable to $L_2$-regularized deep neural networks based on the top $K$ eigenpairs of the Fisher information matrix. We address efficient computation of full-rank approximate eigendecompositions in terms of either the exact inverse Hessian, the inverse outer-products of gradients approximation or the so-called Sandwich estimator. Moreover, we provide a bound on the approximation error for the uncertainty of the predictive class probabilities. We observe that when the smallest eigenvalue of the Fisher information matrix is near the $L_2$-regularization rate, the approximation error is close to zero even when $K\ll P$. A demonstration of the methodology is presented using a TensorFlow implementation, and we show that meaningful rankings of images based on predictive uncertainty can be obtained for two LeNet-based neural networks using the MNIST and CIFAR-10 datasets. Further, we observe that false positives have on average a higher predictive epistemic uncertainty than true positives. This suggests that there is supplementing information in the uncertainty measure not captured by the classification alone.

cs.LG↗

arXiv2020

Saddlepoint-adjusted inversion of characteristic functions

For certain types of statistical models, the characteristic function (Fourier transform) is available in closed form, whereas the probability density function has an intractable form, typically as an infinite sum of probability weighted densities. Important such examples include solutions of stochastic differential equations with jumps, the Tweedie model, and Poisson mixture models. We propose a novel and general numerical method for retrieving the probability density function from the characteristic function, conditioned on the existence of the moment generating function. Unlike methods based on direct application of quadrature to the inverse Fourier transform, the proposed method allows accurate evaluation of the log-probability density function arbitrarily far out in the tail. Moreover, unlike ordinary saddlepoint approximations, the proposed methodology is in principle exact modulus discretization and truncation error of quadrature applied to inversion in a high-density region. Owing to these properties, the proposed method is computationally stable and very accurate under log-likelihood optimisation. The method is illustrated for a normal variance-mean mixture, and in an application of maximum likelihood estimation to a jump diffusion model for financial data.

stat.CO↗

arXiv2020

Heritability curves: a local measure of heritability

This paper introduces a new measure of heritability which relaxes the classical assumption that the degree of heritability of a continuous trait can be summarized by a single number.This measure can be used in situations where the trait dependence structure between family members is nonlinear, in which case traditional mixed effects models and covariance (correlation) based methods are inadequate. Our idea is to combine the notion of a correlation curve with traditional correlation based measures of heritability, such as the formula of Falconer. For estimation purposes, we use a multivariate Gaussian mixture, which is able to capture non-linear dependence and respects certain distributional constraints. We derive an analytical expression for the associated correlation curve, and investigate its limiting behaviour when the trait value becomes either large or small. The result is a measure of heritability that varies with the trait value. When applied to birth weight data on Norwegian mother father child trios, the conclusion is that low and high birth weight are less heritable traits than medium birth weight. On the other hand, we find no similar heterogeneity in the heritability of Body Mass Index (BMI) when studying monozygotic and dizygotic twins.

stat.AP↗

arXiv2018

Whole genome resequencing reveals diagnostic markers for investigating global migration and hybridization between minke whale species

Background: In the marine environment, where there are few absolute physical barriers, contemporary contact between previously isolated species can occur across great distances, and in some cases, may be inter-oceanic. [..] in the minke whale species complex [...] migrations [..] have been documented and fertile hybrids and back-crossed individuals between both species have also been identified. However, it is not known whether this represents a contemporary event, potentially driven by ecosystem changes in the Antarctic, or a sporadic occurrence happening over an evolutionary time-scale. We successfully used whole genome resequencing to identify a panel of diagnostic SNPs which now enable us address this evolutionary question. Results: A large number of SNPs displaying fixed or nearly fixed allele frequency differences among the minke whale species were identified from the sequence data. Five panels of putatively diagnostic markers were established on a genotyping platform for validation of allele frequencies; two panels (26 and 24 SNPs) separating the two species of minke whale, and three panels (22, 23, and 24 SNPs) differentiating the three subspecies of common minke whale. The panels were validated against a set of reference samples, demonstrating the ability to accurately identify back-crossed whales up to three generations. Conclusions: This work has resulted in the development of a panel of novel diagnostic genetic markers to address inter-oceanic and global contact among the genetically isolated minke whale species and sub-species. These markers, including a globally relevant genetic reference data set for this species complex, are now openly available for researchers [..]. The approach used here, combining whole genome resequencing and high-throughput genotyping, represents a universal approach to develop similar tools for other species and population complexes.

q-bio.GN↗

arXiv2016

On the application of higher order symplectic integrators in Hamiltonian Monte Carlo

We explore the construction of new symplectic numerical integration schemes to be used in Hamiltonian Monte Carlo and study their efficiency. Two integration schemes from Blanes et al. (2014), and a new scheme based on optimal acceptance probability, are considered as candidates to the commonly used leapfrog method. All integration schemes are tested within the framework of the No-U-Turn sampler (NUTS), both for a logistic regression model and a student $t$-model. The results show that the leapfrog method is inferior to all the new methods both in terms of asymptotic expected acceptance probability for a model problem and the and efficient sample size per computing time for the realistic models.

stat.CO↗