Searcharxiv⌕ Search
Home/Archive/arXiv

arXiv · math/0508276

Boosting with early stopping: Convergence and consistency

Tong Zhang·Bin Yu
View original source ↗Read source document ↗Export metadata ↓

Abstract

Boosting is one of the most significant advances in machine learning for classification and regression. In its original and computationally flexible version, boosting seeks to minimize empirically a loss function in a greedy fashion. The resulting estimator takes an additive function form and is built iteratively by applying a base estimator (or learner) to updated samples depending on the previous iterations. An unusual regularization technique, early stopping, is employed based on CV or a test set. This paper studies numerical convergence, consistency and statistical rates of convergence of boosting with early stopping, when it is carried out over the linear span of a family of basis functions. For general loss functions, we prove the convergence of boosting's greedy optimization to the infinimum of the loss function over the linear span. Using the numerical convergence result, we find early-stopping strategies under which boosting is shown to be consistent based on i.i.d. samples, and we obtain bounds on the rates of convergence for boosting estimators. Simulation studies are also presented to illustrate the relevance of our theoretical results for providing insights to practical aspects of boosting. As a side product, these results also reveal the importance of restricting the greedy search step-sizes, as known in practice through the work of Friedman and others. Moreover, our results lead to a rigorous proof that for a linearly separable problem, AdaBoost with ε\to0 step-size becomes an L^1-margin maximizer when left to run to convergence.

Explore related subjects

math.STstat.TH

Keep this discovery

Explore connections, maps & timelines

BibTeXRIS

Tong Zhang, Bin Yu. 2005-08-16. Boosting with early stopping: Convergence and consistency. https://doi.org/10.1214/009053605000000255

Cite the original work for its findings. Save a collection to share your selection of sources.

KEEP EXPLORING

Related papers

arXiv2026

Adaptive Inference in Multivariate Nonparametric Regression Models Under Monotonicity

We consider the problem of adaptive inference on a regression function at a point under a multivariate nonparametric regression setting. The regression function belongs to a Hölder class and is assumed to be monotone with respect to some or all of the arguments. We derive the minimax rate of convergence for confidence intervals (CIs) that adapt to the underlying smoothness, and provide an adaptive inference procedure that obtains this minimax rate. When the regression function is monotone in $k_{+}$ of its $k$ arguments, the minimax expected length over a Hölder class with exponent $γ'$ of CIs that maintain coverage over a rougher class with exponent $γ$ is of order $n^{-1/(2+k_{+}/γ'+(k-k_{+})/γ)}$. The rate is unchanged if the rougher class restricts the smoothness only at the point, along the nonmonotone arguments. Under monotonicity along a closed convex cone that contains no line, only the dimension of its linear span enters the rate. The procedure differs from that of Cai and Low (2004), intended to yield shorter CIs under practically relevant specifications. It is valid in finite samples under Gaussian errors with a known variance. The proposed method applies to general linear functionals of the regression function.

math.ST↗
arXiv2026

Adaptive Bayesian Estimation of Conditional Densities Based on Orthogonal Polynomials

We study Bayesian estimation of conditional densities represented by normalized squares of orthogonal polynomial series. We specify the conditional density without modeling the marginal distribution of covariates. A prior on smoothness, the number of terms, and the coefficients gives adaptive posterior contraction at the minimax rate without a logarithmic factor when the normalized square-root representation of the true density is bounded away from zero and sufficiently smooth. For one outcome and $d_x$ covariates, the rate over classes with degree smoothness $s$ is $n^{-s/(2s+d_x+1)}$. The posterior predictive density attains the corresponding minimax squared Hellinger risk. For bounded nonnegative representations in Legendre and Chebyshev classes, we establish the minimax rate at every $s>0$. A prior that also permits a small mixture with the reference density adapts over these classes up to a logarithmic factor and retains the exact rate on the positive, sufficiently smooth subclass. Additional results concern identification, conditional distribution functions and quantiles, coefficient recovery and uniform convergence, independent Gaussian coefficient priors, and selection of polynomial terms.

math.ST↗
arXiv2026

On the median of the beta distribution in relation to the median of the gamma distribution

The median of the beta distribution has no closed form, but it is closely tied to the median of the gamma distribution. Kerman's approximation has the ratio form $g(α)/(g(α)+g(β))$ with $g(x) = x - 1/3$, a lower bound on the gamma median, and it is a lower bound on the beta median for $1/3 < α< β$. The ratio form built from the true gamma median is a tighter lower bound, asymptotically exact as $β\to \infty$. The ratio form built from the Wilson-Hilferty approximation to the gamma median, implicit in a classical approximation to the $F$ distribution but little known as an approximation to the beta median, is an upper bound on the edge $α= 1$ and for large shape parameters. The ratio form that is exact on the lines $α= 1$ and $β= 1$ yields a new approximation to the gamma median, which is an upper bound on it for shape parameter below 1 and a lower bound above 1; no ratio form gives the beta median exactly. For large shape parameters, the logit of the median has an explicit second-order expansion. For small shape parameters, bounds in elementary functions are asymptotically tight, and odds-corrected forms of the low-$α$ asymptote, with an elementary factor that is exact on the edges $α= 1$ and $β= 1$, are, numerically, one-sided bounds within 0.44% of the median that are exact on the lines $α= 1$, $β= 1$, and $α= β$. Steering their exponents with four fitted constants, and interpolating between two ratio forms where both shape parameters exceed 1, gives an approximation in elementary functions that is exact on those lines and numerically within 0.065% of the median over the whole domain. Most of the proofs are machine-verified in Lean 4.

math.ST↗