arXiv subjects

Ratmir Miftachov

Publications and source records attributed to Ratmir Miftachov.

6 recordsLinked to original sources

arXiv2026

Interpretable Tabular Foundation Models via In-Context Kernel Regression

Tabular foundation models like TabPFN and TabICL achieve state-of-the-art performance through in-context learning, yet their architectures remain fundamentally opaque. We introduce KernelICL, a framework to enhance tabular foundation models with quantifiable sample-based interpretability. Building on the insight that in-context learning is akin to kernel regression, we make this mechanism explicit by replacing the final prediction layer with kernel functions (Gaussian, dot-product, kNN) so that every prediction is a transparent weighted average of training labels. We introduce a two-dimensional taxonomy that formally unifies standard kernel methods, modern neighbor-based approaches, and attention mechanisms under a single framework, and quantify inspectability via the perplexity of the weight distribution over training samples. On 55 TALENT benchmark datasets, KernelICL achieves performance on par with existing tabular foundation models, demonstrating that explicit kernel constraints on the final layer enable inspectable predictions without sacrificing performance.

cs.LG↗

arXiv2026

High-Dimensional Search, Low-Dimensional Solution: Decoupling Optimization from Representation

State-of-the-art models rely on massive widths despite exhibiting low Intrinsic Dimension (ID). We posit that this redundancy serves the non-convex optimization search rather than the final representation. We validate this hypothesis by decoupling the solution geometry via data-independent random projections, demonstrating that ResNet, ViT, and BERT representations can be compressed by up to 16x with negligible performance degradation of around 1%. Notably, these oblivious projections achieve parity with PCA and learned baselines, confirming the solution manifold is intrinsically robust. These findings establish the foundation for Subspace-Native Distillation: a paradigm where student models target this intrinsic manifold directly, bypassing the high-dimensional optimization bottleneck to realize the vision of "Train Big, Deploy Small"

cs.LG↗

arXiv2025

Risk Premia in the Bitcoin Market

We analyze the first and second moment risk premia in the Bitcoin market based on options and realized returns and contrast them to the premia embedded in the main US stock index market. First, Bitcoin is much more volatile and has a higher variance risk premium than the S&P 500. By decomposing the return premium into different regions of the return state space, we find that while most of the S&P 500 equity premium comes from mildly negative returns, the corresponding negative Bitcoin returns (between three and one standard deviations) account for only one-third of the total Bitcoin premium (BP). Further, applying a novel clustering algorithm to a collection of estimated Bitcoin option-implied risk-neutral densities, we find that risk premia vary over time as a function of two distinct market volatility regimes. The low-volatility regime implies a relatively high share of BP attributable to positive returns and a high Bitcoin Variance Risk Premium (BVRP). In high-volatility states, the BP attributable to positive and negative returns is more balanced, and the BVRP is lower. These results suggest Bitcoin investors are more concerned about variance and upside risk in a low-volatility regime.

econ.GN↗

arXiv2025

Early Stopping for Regression Trees

We develop early stopping rules for growing regression tree estimators. The fully data-driven stopping rule is based on monitoring the global residual norm. The best-first search and the breadth-first search algorithms together with linear interpolation give rise to generalized projection or regularization flows. A general theory of early stopping is established. Oracle inequalities for the early-stopped regression tree are derived without any smoothness assumption on the regression function, assuming the original CART splitting rule, yet with a much broader scope. The remainder terms are of smaller order than the best achievable rates for Lipschitz functions in dimension $d\ge 2$. In real and synthetic data the early stopping regression tree estimators attain the statistical performance of cost-complexity pruning while significantly reducing computational costs.

math.ST↗

arXiv2025

EarlyStopping: Implicit Regularization for Iterative Learning Procedures in Python

Iterative learning procedures are ubiquitous in machine learning and modern statistics. Regularision is typically required to prevent inflating the expected loss of a procedure in later iterations via the propagation of noise inherent in the data. Significant emphasis has been placed on achieving this regularisation implicitly by stopping procedures early. The EarlyStopping-package provides a toolbox of (in-sample) sequential early stopping rules for several well-known iterative estimation procedures, such as truncated SVD, Landweber (gradient descent), conjugate gradient descent, L2-boosting and regression trees. One of the central features of the package is that the algorithms allow the specification of the true data-generating process and keep track of relevant theoretical quantities. In this paper, we detail the principles governing the implementation of the EarlyStopping-package and provide a survey of recent foundational advances in the theoretical literature. We demonstrate how to use the EarlyStopping-package to explore core features of implicit regularisation and replicate results from the literature.

stat.ML↗

arXiv2024

Shapley Curves: A Smoothing Perspective

This paper fills the limited statistical understanding of Shapley values as a variable importance measure from a nonparametric (or smoothing) perspective. We introduce population-level \textit{Shapley curves} to measure the true variable importance, determined by the conditional expectation function and the distribution of covariates. Having defined the estimand, we derive minimax convergence rates and asymptotic normality under general conditions for the two leading estimation strategies. For finite sample inference, we propose a novel version of the wild bootstrap procedure tailored for capturing lower-order terms in the estimation of Shapley curves. Numerical studies confirm our theoretical findings, and an empirical application analyzes the determining factors of vehicle prices.

stat.ML↗