arXiv subjects

Fumiyasu Komaki

Publications and source records attributed to Fumiyasu Komaki.

33 records · Page 2Linked to original sources

arXiv2018

Analysis of Noise Contrastive Estimation from the Perspective of Asymptotic Variance

There are many models, often called unnormalized models, whose normalizing constants are not calculated in closed form. Maximum likelihood estimation is not directly applicable to unnormalized models. Score matching, contrastive divergence method, pseudo-likelihood, Monte Carlo maximum likelihood, and noise contrastive estimation (NCE) are popular methods for estimating parameters of such models. In this paper, we focus on NCE. The estimator derived from NCE is consistent and asymptotically normal because it is an M-estimator. NCE characteristically uses an auxiliary distribution to calculate the normalizing constant in the same spirit of the importance sampling. In addition, there are several candidates as objective functions of NCE. We focus on how to reduce asymptotic variance. First, we propose a method for reducing asymptotic variance by estimating the parameters of the auxiliary distribution. Then, we determine the form of the objective functions, where the asymptotic variance takes the smallest values in the original estimator class and the proposed estimator classes. We further analyze the robustness of the estimator.

stat.ML↗

arXiv2017

On $\varepsilon$-Admissibility in High Dimension and Nonparametrics

In this paper, we discuss the use of $\varepsilon$-admissibility for estimation in high-dimensional and nonparametric statistical models. The minimax rate of convergence is widely used to compare the performance of estimators in high-dimensional and nonparametric models. However, it often works poorly as a criterion of comparison. In such cases, the addition of comparison by $\varepsilon$-admissibility provides a better outcome. We demonstrate the usefulness of $\varepsilon$-admissibility through high-dimensional Poisson model and Gaussian infinite sequence model, and present noble results.

math.ST↗

arXiv2017

Asymptotically Minimax Prediction in Infinite Sequence Models

We study asymptotically minimax predictive distributions in an infinite sequence model. First, we discuss the connection between the prediction in the infinite sequence model and the prediction in the function model. Second, we construct an asymptotically minimax predictive distribution when the parameter space is a known ellipsoid. We show that the Bayesian predictive distribution based on the Gaussian prior distribution is asymptotically minimax in the ellipsoid. Third, we construct an asymptotically minimax predictive distribution for any Sobolev ellipsoid. We show that the Bayesian predictive distribution based on the product of Stein's priors is asymptotically minimax for any Sobolev ellipsoid. Finally, we present an efficient sampling method from the proposed Bayesian predictive distribution.

math.ST↗

arXiv2017

Empirical Bayes Matrix Completion

We develop an empirical Bayes (EB) algorithm for the matrix completion problems. The EB algorithm is motivated from the singular value shrinkage estimator for matrix means by Efron and Morris (1972). Since the EB algorithm is essentially the EM algorithm applied to a simple model, it does not require heuristic parameter tuning other than tolerance. Numerical results demonstrated that the EB algorithm achieves a good trade-off between accuracy and efficiency compared to existing algorithms and that it works particularly well when the difference between the number of rows and columns is large. Application to real data also shows the practical utility of the EB algorithm.

stat.ML↗

arXiv2015

Asymptotic Properties of Bayesian Predictive Densities When the Distributions of Data and Target Variables are Different

Bayesian predictive densities when the observed data $x$ and the target variable $y$ to be predicted have different distributions are investigated by using the framework of information geometry. The performance of predictive densities is evaluated by the Kullback--Leibler divergence. The parametric models are formulated as Riemannian manifolds. In the conventional setting in which $x$ and $y$ have the same distribution, the Fisher--Rao metric and the Jeffreys prior play essential roles. In the present setting in which $x$ and $y$ have different distributions, a new metric, which we call the predictive metric, constructed by using the Fisher information matrices of $x$ and $y$, and the volume element based on the predictive metric play the corresponding roles. It is shown that Bayesian predictive densities based on priors constructed by using non-constant positive superharmonic functions with respect to the predictive metric asymptotically dominate those based on the volume element prior of the predictive metric.

math.ST↗

arXiv2015

Information criteria for multistep ahead predictions

We propose an information criterion for multistep ahead predictions. It is also used for extrapolations. For the derivation, we consider multistep ahead predictions under local misspecification. In the prediction, we show that Bayesian predictive distributions asymptotically have smaller Kullback--Leibler risks than plug-in predictive distributions. From the results, we construct an information criterion for multistep ahead predictions by using an asymptotically unbiased estimator of the Kullback--Leibler risk of Bayesian predictive distributions. We show the effectiveness of the proposed information criterion throughout the numerical experiments.

math.ST↗

arXiv2014

Relations Between the Conditional Normalized Maximum Likelihood Distributions and the Latent Information Priors

We reveal the relations between the conditional normalized maximum likelihood (CNML) distributions and Bayesian predictive densities based on the latent information priors (LIPs). In particular, CNML3, which is one type of CNML distributions, is investigated. The Bayes projection of a predictive density, which is an information projection of the predictive density on a set of Bayesian predictive densities, is considered. We prove that the sum of the Bayes projection divergence of CNML3 and the conditional mutual information is asymptotically constant. This result implies that the Bayes projection of CNML3 (BPCNML3) is asymptotically identical to the Bayesian predictive density based on LIP. In addition, under some stronger assumptions, we show that BPCNML3 exactly coincides with the Bayesian predictive density based on LIP.

math.ST↗

arXiv2014

Determinantal Point Process Priors for Bayesian Variable Selection in Linear Regression

We propose discrete determinantal point processes (DPPs) for priors on the model parameter in Bayesian variable selection. By our variable selection method, collinear predictors are less likely to be selected simultaneously because of the repulsion property of discrete DPPs. Three types of DPP priors are proposed. We show the efficiency of the proposed priors through numerical experiments and applications to collinear datasets.

stat.ME↗

arXiv2014

Simultaneous prediction for independent Poisson processes with different durations

Simultaneous predictive densities for independent Poisson observables are investigated. The observed data and the target variables to be predicted are independently distributed according to different Poisson distributions parametrized by the same parameter. The performance of predictive densities is evaluated by the Kullback-Leibler divergence. A class of prior distributions depending on the objective of prediction is introduced. A Bayesian predictive density based on a prior in this class dominates the Bayesian predictive density based on the Jeffreys prior.

math.ST↗

arXiv2011

Asymptotically minimax Bayesian predictive densities for multinomial models

One-step ahead prediction for the multinomial model is considered. The performance of a predictive density is evaluated by the average Kullback-Leibler divergence from the true density to the predictive density. Asymptotic approximations of risk functions of Bayesian predictive densities based on Dirichlet priors are obtained. It is shown that a Bayesian predictive density based on a specific Dirichlet prior is asymptotically minimax. The asymptotically minimax prior is different from known objective priors such as the Jeffreys prior or the uniform prior.

math.ST↗

arXiv2010

Bayesian Predictive Densities Based on Latent Information Priors

Construction methods for prior densities are investigated from a predictive viewpoint. Predictive densities for future observables are constructed by using observed data. The simultaneous distribution of future observables and observed data is assumed to belong to a parametric submodel of a multinomial model. Future observables and data are possibly dependent. The discrepancy of a predictive density to the true conditional density of future observables given observed data is evaluated by the Kullback-Leibler divergence. It is proved that limits of Bayesian predictive densities form an essentially complete class. Latent information priors are defined as priors maximizing the conditional mutual information between the parameter and the future observables given the observed data. Minimax predictive densities are constructed as limits of Bayesian predictive densities based on prior sequences converging to the latent information priors.

math.ST↗

arXiv2007

Bayesian shrinkage prediction for the regression problem

We consider Bayesian shrinkage predictions for the Normal regression problem under the frequentist Kullback-Leibler risk function. Firstly, we consider the multivariate Normal model with an unknown mean and a known covariance. While the unknown mean is fixed, the covariance of future samples can be different from training samples. We show that the Bayesian predictive distribution based on the uniform prior is dominated by that based on a class of priors if the prior distributions for the covariance and future covariance matrices are rotation invariant. Then, we consider a class of priors for the mean parameters depending on the future covariance matrix. With such a prior, we can construct a Bayesian predictive distribution dominating that based on the uniform prior. Lastly, applying this result to the prediction of response variables in the Normal linear regression model, we show that there exists a Bayesian predictive distribution dominating that based on the uniform prior. Minimaxity of these Bayesian predictions follows from these results.

math.ST↗

arXiv2006

Shrinkage priors for Bayesian prediction

We investigate shrinkage priors for constructing Bayesian predictive distributions. It is shown that there exist shrinkage predictive distributions asymptotically dominating Bayesian predictive distributions based on the Jeffreys prior or other vague priors if the model manifold satisfies some differential geometric conditions. Kullback--Leibler divergence from the true distribution to a predictive distribution is adopted as a loss function. Conformal transformations of model manifolds corresponding to vague priors are introduced. We show several examples where shrinkage predictive distributions dominate Bayesian predictive distributions based on vague priors.

math.ST↗

arXiv2005

Asymptotic Expansion of the Risk Difference of the Bayesian Spectral Density in the ARMA model

The autoregressive moving average (ARMA) model is one of the most important models in time series analysis.We consider the Bayesian estimation of an unknown spectral density in the ARMA model.In the i.i.d. cases, Komaki showed that Bayesian predictive densities based on a superharmonic prior asymptotically dominate those based on the Jeffreys prior.It is shown by using the asymptotic expansion of the risk difference.We obtain the corresponding result in the ARMA model.

math.ST↗

arXiv2004

Simultaneous prediction of independent Poisson observables

Simultaneous predictive distributions for independent Poisson observables are investigated. A class of improper prior distributions for Poisson means is introduced. The Bayesian predictive distributions based on priors from the introduced class are shown to be admissible under the Kullback-Leibler loss. A Bayesian predictive distribution based on a prior in this class dominates the Bayesian predictive distribution based on the Jeffreys prior.

math.ST↗