arXiv subjects

Kenji Kawaguchi

Publications and source records attributed to Kenji Kawaguchi.

At least 163 records · Page 9Linked to original sources

arXiv2020

Gradient Descent Finds Global Minima for Generalizable Deep Neural Networks of Practical Sizes

In this paper, we theoretically prove that gradient descent can find a global minimum of non-convex optimization of all layers for nonlinear deep neural networks of sizes commonly encountered in practice. The theory developed in this paper only requires the practical degrees of over-parameterization unlike previous theories. Our theory only requires the number of trainable parameters to increase linearly as the number of training samples increases. This allows the size of the deep neural networks to be consistent with practice and to be several orders of magnitude smaller than that required by the previous theories. Moreover, we prove that the linear increase of the size of the network is the optimal rate and that it cannot be improved, except by a logarithmic factor. Furthermore, deep neural networks with the trainability guarantee are shown to generalize well to unseen test samples with a natural dataset but not a random dataset.

stat.ML↗

arXiv2020

Ordered SGD: A New Stochastic Optimization Framework for Empirical Risk Minimization

We propose a new stochastic optimization framework for empirical risk minimization problems such as those that arise in machine learning. The traditional approaches, such as (mini-batch) stochastic gradient descent (SGD), utilize an unbiased gradient estimator of the empirical average loss. In contrast, we develop a computationally efficient method to construct a gradient estimator that is purposely biased toward those observations with higher current losses. On the theory side, we show that the proposed method minimizes a new ordered modification of the empirical average loss, and is guaranteed to converge at a sublinear rate to a global optimum for convex loss and to a critical point for weakly convex (non-convex) loss. Furthermore, we prove a new generalization bound for the proposed algorithm. On the empirical side, the numerical experiments show that our proposed method consistently improves the test errors compared with the standard mini-batch SGD in various models including SVM, logistic regression, and deep learning problems.

stat.ML↗

arXiv2020

Elimination of All Bad Local Minima in Deep Learning

In this paper, we theoretically prove that adding one special neuron per output unit eliminates all suboptimal local minima of any deep neural network, for multi-class classification, binary classification, and regression with an arbitrary loss function, under practical assumptions. At every local minimum of any deep neural network with these added neurons, the set of parameters of the original neural network (without added neurons) is guaranteed to be a global minimum of the original neural network. The effects of the added neurons are proven to automatically vanish at every local minimum. Moreover, we provide a novel theoretical characterization of a failure mode of eliminating suboptimal local minima via an additional theorem and several examples. This paper also introduces a novel proof technique based on the perturbable gradient basis (PGB) necessary condition of local minima, which provides new insight into the elimination of local minima and is applicable to analyze various models and transformations of objective functions beyond the elimination of local minima.

cs.LG↗

arXiv2020

Evidence for planetary hypothesis for PTFO 8-8695b with five-year optical/infrared monitoring observations

PTFO 8-8695b (CVSO 30b) is a young planet candidate whose host star is a $\sim$ 2.6 Myr-old T-Tauri star, and there have been continuous discussions about the nature of this system. To unveil the mystery of this system, we observed PTFO8-8695 for around five years at optical and infrared bands simultaneously using Kanata telescope at the Higashi-Hiroshima Observatory. Through our observations, we found that the reported fading event split into two: deeper but phase-shifted "dip-A" and shallower but equiphase "dip-B". These dips disappeared at different epochs, and then, dip-B reappeared. Based on the observed wavelength dependence of dip depths, a dust clump and a precessing planet are likely origins of dip-A and B, respectively. Here we propose "a precessing planet associated with a dust cloud" scenario for this system. This scenario is consistent with the reported change in the depth of fading events, and even with the reported results, which were thought to be negative evidence to the planetary hypothesis, such as the past non-detection of the Rossiter-McLaughlin effect. If this scenario is correct, this is the third case of a young (<3 Myr) planet around a pre-main sequence star. This finding implies that a planet can be formed within a few Myr.

astro-ph.EP↗

arXiv2019

Every Local Minimum Value is the Global Minimum Value of Induced Model in Non-convex Machine Learning

For nonconvex optimization in machine learning, this article proves that every local minimum achieves the globally optimal value of the perturbable gradient basis model at any differentiable point. As a result, nonconvex machine learning is theoretically as supported as convex machine learning with a handcrafted basis in terms of the loss at differentiable local minima, except in the case when a preference is given to the handcrafted basis over the perturbable gradient basis. The proofs of these results are derived under mild assumptions. Accordingly, the proven results are directly applicable to many machine learning models, including practical deep neural networks, without any modification of practical methods. Furthermore, as special cases of our general results, this article improves or complements several state-of-the-art theoretical results on deep neural networks, deep residual networks, and overparameterized deep neural networks with a unified proof technique and novel geometric insights. A special case of our results also contributes to the theoretical foundation of representation learning.

stat.ML↗

arXiv2019

Depth with Nonlinearity Creates No Bad Local Minima in ResNets

In this paper, we prove that depth with nonlinearity creates no bad local minima in a type of arbitrarily deep ResNets with arbitrary nonlinear activation functions, in the sense that the values of all local minima are no worse than the global minimum value of corresponding classical machine-learning models, and are guaranteed to further improve via residual representations. As a result, this paper provides an affirmative answer to an open question stated in a paper in the conference on Neural Information Processing Systems 2018. This paper advances the optimization theory of deep learning only for ResNets and not for other network architectures.

stat.ML↗

arXiv2019

Effect of Depth and Width on Local Minima in Deep Learning

In this paper, we analyze the effects of depth and width on the quality of local minima, without strong over-parameterization and simplification assumptions in the literature. Without any simplification assumption, for deep nonlinear neural networks with the squared loss, we theoretically show that the quality of local minima tends to improve towards the global minimum value as depth and width increase. Furthermore, with a locally-induced structure on deep nonlinear neural networks, the values of local minima of neural networks are theoretically proven to be no worse than the globally optimal values of corresponding classical machine learning models. We empirically support our theoretical observation with a synthetic dataset as well as MNIST, CIFAR-10 and SVHN datasets. When compared to previous studies with strong over-parameterization assumptions, the results in this paper do not require over-parameterization, and instead show the gradual effects of over-parameterization as consequences of general results.

cs.LG↗

arXiv2019

Generalization in Machine Learning via Analytical Learning Theory

This paper introduces a novel measure-theoretic theory for machine learning that does not require statistical assumptions. Based on this theory, a new regularization method in deep learning is derived and shown to outperform previous methods in CIFAR-10, CIFAR-100, and SVHN. Moreover, the proposed theory provides a theoretical basis for a family of practically successful regularization methods in deep learning. We discuss several consequences of our results on one-shot learning, representation learning, deep learning, and curriculum learning. Unlike statistical learning theory, the proposed learning theory analyzes each problem instance individually via measure theory, rather than a set of problem instances via statistics. As a result, it provides different types of results and insights when compared to statistical learning theory.

stat.ML↗

arXiv2019

Eliminating all bad Local Minima from Loss Landscapes without even adding an Extra Unit

Recent work has noted that all bad local minima can be removed from neural network loss landscapes, by adding a single unit with a particular parameterization. We show that the core technique from these papers can be used to remove all bad local minima from any loss landscape, so long as the global minimum has a loss of zero. This procedure does not require the addition of auxiliary units, or even that the loss be associated with a neural network. The method of action involves all bad local minima being converted into bad (non-local) minima at infinity in terms of auxiliary parameters.

stat.ML↗

arXiv2018

Theory of Deep Learning III: explaining the non-overfitting puzzle

A main puzzle of deep networks revolves around the absence of overfitting despite large overparametrization and despite the large capacity demonstrated by zero training error on randomly labeled data. In this note, we show that the dynamics associated to gradient descent minimization of nonlinear networks is topologically equivalent, near the asymptotically stable minima of the empirical error, to linear gradient system in a quadratic potential with a degenerate (for square loss) or almost degenerate (for logistic or crossentropy loss) Hessian. The proposition depends on the qualitative theory of dynamical systems and is supported by numerical results. Our main propositions extend to deep nonlinear networks two properties of gradient descent for linear networks, that have been recently established (1) to be key to their generalization properties: 1. Gradient descent enforces a form of implicit regularization controlled by the number of iterations, and asymptotically converges to the minimum norm solution for appropriate initial conditions of gradient descent. This implies that there is usually an optimum early stopping that avoids overfitting of the loss. This property, valid for the square loss and many other loss functions, is relevant especially for regression. 2. For classification, the asymptotic convergence to the minimum norm solution implies convergence to the maximum margin solution which guarantees good classification error for "low noise" datasets. This property holds for loss functions such as the logistic and cross-entropy loss independently of the initial conditions. The robustness to overparametrization has suggestive implications for the robustness of the architecture of deep convolutional networks with respect to the curse of dimensionality.

cs.LG↗

arXiv2017

Deep Semi-Random Features for Nonlinear Function Approximation

We propose semi-random features for nonlinear function approximation. The flexibility of semi-random feature lies between the fully adjustable units in deep learning and the random features used in kernel methods. For one hidden layer models with semi-random features, we prove with no unrealistic assumptions that the model classes contain an arbitrarily good function as the width increases (universality), and despite non-convexity, we can find such a good function (optimization theory) that generalizes to unseen new data (generalization bound). For deep models, with no unrealistic assumptions, we prove universal approximation ability, a lower bound on approximation error, a partial optimization guarantee, and a generalization bound. Depending on the problems, the generalization bound of deep semi-random features can be exponentially better than the known bounds of deep ReLU nets; our generalization error bound can be independent of the depth, the number of trainable weights as well as the input dimensionality. In experiments, we show that semi-random features can match the performance of neural networks by using slightly more units, and it outperforms random features by using significantly fewer units. Moreover, we introduce a new implicit ensemble method by using semi-random features.

cs.LG↗

arXiv2017

OISTER Optical and Near-Infrared Monitoring Observations of a Peculiar Radio-Loud Active Galactic Nucleus SDSS J110006.07+442144.3

We present monitoring campaign observations at optical and near-infrared (NIR) wavelengths for a radio-loud active galactic nucleus (AGN) at z=0.840, SDSS~J110006.07+442144.3 (hereafter, J1100+4421), which was identified during a flare phase in late February, 2014. The campaigns consist of three intensive observing runs from the discovery to March, 2015, mostly within the scheme of the OISTER collaboration. Optical-NIR light curves and simultaneous spectral energy distributions (SEDs) are obtained. Our measurements show the strongest brightening in March, 2015. We found that the optical-NIR SEDs of J1100+4421 show an almost steady shape despite the large and rapid intranight variability. This constant SED shape is confirmed to extend to $\sim5~μ$m in the observed frame using the archival WISE data. Given the lack of absorption lines and the steep power-law spectrum of $α_ν\sim-1.4$, where $f_ν\proptoν^{α_ν}$, synchrotron radiation by a relativistic jet with no or small contributions from the host galaxy and the accretion disk seems most plausible as an optical-NIR emission mechanism. The steep optical-NIR spectral shape and the large amplitude of variability are consistent with this object being a low $ν_{\rm{peak}}$ jet-dominated AGN. In addition, sub-arcsec resolution optical imaging data taken with Subaru Hyper Suprime-Cam does not show a clear extended component and the spatial scales are significantly smaller than the large extensions detected at radio wavelengths. The optical spectrum of a possible faint companion galaxy does not show any emission lines at the same redshift and hence a merging hypothesis for this AGN-related activity is not supported by our observations.

astro-ph.GA↗

arXiv2017

Depth Creates No Bad Local Minima

In deep learning, \textit{depth}, as well as \textit{nonlinearity}, create non-convex loss surfaces. Then, does depth alone create bad local minima? In this paper, we prove that without nonlinearity, depth alone does not create bad local minima, although it induces non-convex loss surface. Using this insight, we greatly simplify a recently proposed proof to show that all of the local minima of feedforward deep linear neural networks are global minima. Our theoretical results generalize previous results with fewer assumptions, and this analysis provides a method to show similar results beyond square loss in deep linear models.

cs.LG↗

arXiv2016

Deep Learning without Poor Local Minima

In this paper, we prove a conjecture published in 1989 and also partially address an open problem announced at the Conference on Learning Theory (COLT) 2015. With no unrealistic assumption, we first prove the following statements for the squared loss function of deep linear neural networks with any depth and any widths: 1) the function is non-convex and non-concave, 2) every local minimum is a global minimum, 3) every critical point that is not a global minimum is a saddle point, and 4) there exist "bad" saddle points (where the Hessian has no negative eigenvalue) for the deeper networks (with more than three layers), whereas there is no bad saddle point for the shallow networks (with three layers). Moreover, for deep nonlinear neural networks, we prove the same four statements via a reduction to a deep linear model under the independence assumption adopted from recent work. As a result, we present an instance, for which we can answer the following question: how difficult is it to directly train a deep model in theory? It is more difficult than the classical machine learning models (because of the non-convexity), but not too difficult (because of the nonexistence of poor local minima). Furthermore, the mathematically proven existence of bad saddle points for deeper models would suggest a possible open problem. We note that even though we have advanced the theoretical foundations of deep learning and non-convex optimization, there is still a gap between theory and practice.

stat.ML↗

arXiv2016

Streaming Normalization: Towards Simpler and More Biologically-plausible Normalizations for Online and Recurrent Learning

We systematically explored a spectrum of normalization algorithms related to Batch Normalization (BN) and propose a generalized formulation that simultaneously solves two major limitations of BN: (1) online learning and (2) recurrent learning. Our proposal is simpler and more biologically-plausible. Unlike previous approaches, our technique can be applied out of the box to all learning scenarios (e.g., online learning, batch learning, fully-connected, convolutional, feedforward, recurrent and mixed --- recurrent and convolutional) and compare favorably with existing approaches. We also propose Lp Normalization for normalizing by different orders of statistical moments. In particular, L1 normalization is well-performing, simple to implement, fast to compute, more biologically-plausible and thus ideal for GPU or hardware implementations.

cs.LG↗

arXiv2016

Global Continuous Optimization with Error Bound and Fast Convergence

This paper considers global optimization with a black-box unknown objective function that can be non-convex and non-differentiable. Such a difficult optimization problem arises in many real-world applications, such as parameter tuning in machine learning, engineering design problem, and planning with a complex physics simulator. This paper proposes a new global optimization algorithm, called Locally Oriented Global Optimization (LOGO), to aim for both fast convergence in practice and finite-time error bound in theory. The advantage and usage of the new algorithm are illustrated via theoretical analysis and an experiment conducted with 11 benchmark test functions. Further, we modify the LOGO algorithm to specifically solve a planning problem via policy search with continuous state/action space and long time horizon while maintaining its finite-time error bound. We apply the proposed planning method to accident management of a nuclear power plant. The result of the application study demonstrates the practical utility of our method.

math.OC↗

arXiv2016

X-ray and Optical Correlation of Type I Seyfert NGC 3516 Studied with Suzaku and Japanese Ground-Based Telescopes

From 2013 April to 2014 April, we performed an X-ray and optical simultaneous monitoring of the type 1.5 Seyfert galaxy NGC 3516. It employed Suzaku, and 5 Japanese ground-based telescopes, the Pirka, Kiso Schmidt, Nayuta, MITSuME, and the Kanata telescopes. The Suzaku observations were conducted seven times with various intervals ranging from days, weeks, to months, with an exposure of $\sim50$ ksec each. The optical $B$-band observations not only covered those of Suzaku almost simultaneously, but also followed the source as frequently as possible. As a result, NGC 3516 was found in its faint phase with the 2-10 keV flux of $0.21-2.70 \times 10^{-11}$ erg s$^{-1}$ cm$^{-2}$. The 2-45 keV X-ray spectra were composed of a dominant variable hard power-law continuum with a photon index of $\sim1.7$, and a non-relativistic reflection component with a prominent Fe-K$α$ emission line. Producing the $B$-band light curve by differential image photometry, we found that the $B$-band flux changed by $\sim2.7 \times 10^{-11}$ erg s$^{-1}$ cm$^{-2}$, which is comparable to the X-ray variation, and detected a significant flux correlation between the hard power-law component in X-rays and the $B$-band radiation, for the first time in NGC 3516. By examining their correlation, we found that the X-ray flux preceded that of $B$ band by $2.0^{+0.7}_{-0.6}$ days ($1σ$ error). Although this result supports the X-ray reprocessing model, the derived lag is too large to be explained by the standard view which assumes a "lamppost"-type X-ray illuminator located near a standard accretion disk. Our results are better explained by assuming a hot accretion flow and a truncated disk.

astro-ph.GA↗

arXiv2016

Bayesian Optimization with Exponential Convergence

This paper presents a Bayesian optimization method with exponential convergence without the need of auxiliary optimization and without the delta-cover sampling. Most Bayesian optimization methods require auxiliary optimization: an additional non-convex global optimization problem, which can be time-consuming and hard to implement in practice. Also, the existing Bayesian optimization method with exponential convergence requires access to the delta-cover sampling, which was considered to be impractical. Our approach eliminates both requirements and achieves an exponential convergence rate.

stat.ML↗