arXiv subjects

Ryuichi Kanoh

Publications and source records attributed to Ryuichi Kanoh.

7 recordsLinked to original sources

arXiv2026

Double Descent in Gradient Boosting Decision Trees via Split-Candidate Scaling

Double descent is commonly studied by scaling an explicit capacity parameter, such as neural-network width. For gradient boosting decision trees (GBDTs), however, an analogous single-axis capacity parameter has not been established. We propose the number of split candidates as an operational capacity parameter for GBDTs. Holding other training controls fixed, increasing the split-candidate budget refines the feature-quantization grid and expands the dictionary of root-to-leaf paths from which boosting selects its updates. To analyze this expansion, we construct an empirical tree-kernel diagnostic that summarizes how candidate-induced paths group the training examples. A regime in which the empirical kernel rank grows toward the sample size and very small positive eigenvalues emerge exposes noise-sensitive directions; in this regime, test error peaks before decreasing again at larger split-candidate budgets. This perspective predicts that deeper trees should reach the regime with fewer split candidates, larger training sets should require finer grids, and label noise should make the peak more pronounced. Experiments support these predictions and show test-error peaks at intermediate split-candidate budgets across XGBoost, LightGBM, and CatBoost, whereas a random-forest control improves monotonically under the same split-candidate sweep. Taken together, our analysis and experiments support split-candidate scaling as a single-axis capacity intervention for studying GBDTs and suggest that the observed double descent arises from an interaction between candidate-induced geometry and boosting dynamics.

cs.LG↗

arXiv2025

Linear Mode Connectivity in Differentiable Tree Ensembles

Linear Mode Connectivity (LMC) refers to the phenomenon that performance remains consistent for linearly interpolated models in the parameter space. For independently optimized model pairs from different random initializations, achieving LMC is considered crucial for understanding the stable success of the non-convex optimization in modern machine learning models and for facilitating practical parameter-based operations such as model merging. While LMC has been achieved for neural networks by considering the permutation invariance of neurons in each hidden layer, its attainment for other models remains an open question. In this paper, we first achieve LMC for soft tree ensembles, which are tree-based differentiable models extensively used in practice. We show the necessity of incorporating two invariances: subtree flip invariance and splitting order invariance, which do not exist in neural networks but are inherent to tree architectures, in addition to permutation invariance of trees. Moreover, we demonstrate that it is even possible to exclude such additional invariances while keeping LMC by designing decision list-based tree architectures, where such invariances do not exist by definition. Our findings indicate the significance of accounting for architecture-specific invariances in achieving LMC.

cs.LG↗

arXiv2023

Analyzing Tree Architectures in Ensembles via Neural Tangent Kernel

A soft tree is an actively studied variant of a decision tree that updates splitting rules using the gradient method. Although soft trees can take various architectures, their impact is not theoretically well known. In this paper, we formulate and analyze the Neural Tangent Kernel (NTK) induced by soft tree ensembles for arbitrary tree architectures. This kernel leads to the remarkable finding that only the number of leaves at each depth is relevant for the tree architecture in ensemble learning with an infinite number of trees. In other words, if the number of leaves at each depth is fixed, the training behavior in function space and the generalization performance are exactly the same across different tree architectures, even if they are not isomorphic. We also show that the NTK of asymmetric trees like decision lists does not degenerate when they get infinitely deep. This is in contrast to the perfect binary trees, whose NTK is known to degenerate and leads to worse generalization performance for deeper trees.

cs.LG↗

arXiv2022

A Neural Tangent Kernel Perspective of Infinite Tree Ensembles

In practical situations, the tree ensemble is one of the most popular models along with neural networks. A soft tree is a variant of a decision tree. Instead of using a greedy method for searching splitting rules, the soft tree is trained using a gradient method in which the entire splitting operation is formulated in a differentiable form. Although ensembles of such soft trees have been used increasingly in recent years, little theoretical work has been done to understand their behavior. By considering an ensemble of infinite soft trees, this paper introduces and studies the Tree Neural Tangent Kernel (TNTK), which provides new insights into the behavior of the infinite ensemble of soft trees. Using the TNTK, we theoretically identify several non-trivial properties, such as global convergence of the training, the equivalence of the oblivious tree structure, and the degeneracy of the TNTK induced by the deepening of the trees.

cs.LG↗

arXiv2021

Unintended Effects on Adaptive Learning Rate for Training Neural Network with Output Scale Change

A multiplicative constant scaling factor is often applied to the model output to adjust the dynamics of neural network parameters. This has been used as one of the key interventions in an empirical study of lazy and active behavior. However, we show that the combination of such scaling and a commonly used adaptive learning rate optimizer strongly affects the training behavior of the neural network. This is problematic as it can cause \emph{unintended behavior} of neural networks, resulting in the misinterpretation of experimental results. Specifically, for some scaling settings, the effect of the adaptive learning rate disappears or is strongly influenced by the scaling factor. To avoid the unintended effect, we present a modification of an optimization algorithm and demonstrate remarkable differences between adaptive learning rate optimization and simple gradient descent, especially with a small ($<1.0$) scaling factor.

cs.LG↗

arXiv2021

Interpretable Mixture Density Estimation by use of Differentiable Tree-module

In order to develop reliable services using machine learning, it is important to understand the uncertainty of the model outputs. Often the probability distribution that the prediction target follows has a complex shape, and a mixture distribution is assumed as a distribution that uncertainty follows. Since the output of mixture density estimation is complicated, its interpretability becomes important when considering its use in real services. In this paper, we propose a method for mixture density estimation that utilizes an interpretable tree structure. Further, a fast inference procedure based on time-invariant information cache achieves both high speed and interpretability.

cs.LG↗

arXiv2016

Hinode and IRIS observations of the magnetohydrodynamic waves propagating from the photosphere to the chromosphere in a sunspot

Magnetohydrodynamic (MHD) waves have been considered as energy sources for heating the solar chromosphere and the corona. Although MHD waves have been observed in the solar atmosphere, there are a lack of quantitative estimates on the energy transfer and dissipation in the atmosphere. We performed simultaneous Hinode and IRIS observations of a sunspot umbra to derive the upward energy fluxes at two different atmospheric layers (photosphere and lower transition region) and estimate the energy dissipation. The observations revealed some properties of the observed periodic oscillations in physical quantities, such as their phase relations, temporal behaviors, and power spectra, making a conclusion that standing slow-mode waves are dominant at the photosphere with their high-frequency leakage, which is observed as upward waves at the chromosphere and the lower transition region. Our estimates of upward energy fluxes are $2.0\times10^7$ erg cm$^{-2}$ s$^{-1}$ at the photospheric level and $8.3\times10^4$ erg cm$^{-2}$ s$^{-1}$ at the lower transition region level. The difference between the energy fluxes is larger than the energy required to maintain the chromosphere in the sunspot umbrae, suggesting that the observed waves can make a crucial contribution to the heating of the chromosphere in the sunspot umbrae. In contrast, the upward energy flux derived at the lower transition region level is smaller than the energy flux required for heating the corona, implying that we may need another heating mechanism. We should, however, note a possibility that the energy dissipated at the chromosphere might be overestimated because of the opacity effect.

astro-ph.SR↗