arXiv subjects

Vadim Sokolov

Publications and source records attributed to Vadim Sokolov.

At least 37 records · Page 2Linked to original sources

arXiv2025

Conformal Prediction = Bayes?

Conformal prediction (CP) is widely presented as distribution-free predictive inference with finite-sample marginal coverage under exchangeability. We argue that CP is best understood as a rank-calibrated descendant of the Fisher-Dempster-Hill fiducial/direct-probability tradition rather than as Bayesian conditioning in disguise. We establish four separations from coherent countably additive predictive semantics. First, canonical conformal constructions violate conditional extensionality: prediction sets can depend on the marginal design P(X) even when P(Y|X) is fixed. Second, any finitely additive sequential extension preserving rank calibration is nonconglomerable, implying countable Dutch-book vulnerabilities. Third, rank-calibrated updates cannot be realized as regular conditionals of any countably additive exchangeable law on Y^infty. Fourth, formalizing both paradigms as families of one-step predictive kernels, conformal and Bayesian kernels coincide only on a Baire-meagre subset of the space of predictive laws. We further show that rank- and proxy-based reductions are generically Blackwell-deficient relative to full-data experiments, yielding positive Le Cam deficiency for suitable losses. Extending the analysis to prediction-powered inference (PPI) yields an analogous message: bias-corrected, proxy-rectified estimators can be valid as confidence devices while failing to define transportable belief states across stages, shifts, or adaptive selection. Together, the results sharpen a general limitation of wrappers: finite-sample calibration guarantees do not by themselves supply composable semantics for sequential updating or downstream decision-making.

math.ST↗

arXiv2025

Generative Bayesian Hyperparameter Tuning

\noindent Hyper-parameter selection is a central practical problem in modern machine learning, governing regularization strength, model capacity, and robustness choices. Cross-validation is often computationally prohibitive at scale, while fully Bayesian hyper-parameter learning can be difficult due to the cost of posterior sampling. We develop a generative perspective on hyper-parameter tuning that combines two ideas: (i) optimization-based approximations to Bayesian posteriors via randomized, weighted objectives (weighted Bayesian bootstrap), and (ii) amortization of repeated optimization across many hyper-parameter settings by learning a transport map from hyper-parameters (including random weights) to the corresponding optimizer. This yields a ``generator look-up table'' for estimators, enabling rapid evaluation over grids or continuous ranges of hyper-parameters and supporting both predictive tuning objectives and approximate Bayesian uncertainty quantification. We connect this viewpoint to weighted $M$-estimation, envelope/auxiliary-variable representations that reduce non-quadratic losses to weighted least squares, and recent generative samplers for weighted $M$-estimators.

stat.ML↗

arXiv2025

Bayesian Global-Local Regularization

We propose a unified framework for global-local regularization that bridges the gap between classical techniques -- such as ridge regression and the nonnegative garotte -- and modern Bayesian hierarchical modeling. By estimating local regularization strengths via marginal likelihood under order constraints, our approach generalizes Stein's positive-part estimator and provides a principled mechanism for adaptive shrinkage in high-dimensional settings. We establish that this isotonic empirical Bayes estimator achieves near-minimax risk (up to logarithmic factors) over sparse ordered model classes, constituting a significant advance in high-dimensional statistical inference. Applications to orthogonal polynomial regression demonstrate the methodology's flexibility, while our theoretical results clarify the connections between empirical Bayes, shape-constrained estimation, and degrees-of-freedom adjustments.

stat.ME↗

arXiv2025

$MC^2$ Mixed Integer and Linear Programming

In this paper, we design $MC^2$ algorithms for Mixed Integer and Linear Programming. By expressing a constrained optimisation as one of simulation from a Boltzmann distribution, we reformulate integer and linear programming as Monte Carlo optimisation problems. The key insight is that solving these optimisation problems requires the ability to simulate from truncated distributions, namely multivariate exponentials and Gaussians. Efficient simulation can be achieved using the algorithms of Kent and Davis. We demonstrate our methodology on portfolio optimisation and the classical farmer problem from stochastic programming. Finally, we conclude with directions for future research.

stat.CO↗

arXiv2025

Generative Quantile Bayesian Prediction

Prediction is a central task of machine learning. Our goal is to solve large scale prediction problems using Generative Quantile Bayesian Prediction (GQBP).By directly learning predictive quantiles rather than densities we achieve a number of theoretical and practical advantages. We contrast our approach with state-of-the-art methods including conformal prediction, fiducial prediction and marginal likelihood. Our distinguishing feature of our method is the use of generative methods for predictive quantile maps. We illustrate our methodology for normal-normal learning and causal inference. Finally, we conclude with directions for future research.

stat.ME↗

arXiv2025

Bayesian Double Descent

Double descent is a phenomenon of over-parameterized statistical models such as deep neural networks which have a re-descending property in their risk function. As the complexity of the model increases, risk exhibits a U-shaped region due to the traditional bias-variance trade-off, then as the number of parameters equals the number of observations and the model becomes one of interpolation where the risk can be unbounded and finally, in the over-parameterized region, it re-descends -- the double descent effect. Our goal is to show that this has a natural Bayesian interpretation. We also show that this is not in conflict with the traditional Occam's razor -- simpler models are preferred to complex ones, all else being equal. Our theoretical foundations use Bayesian model selection, the Dickey-Savage density ratio, and connect generalized ridge regression and global-local shrinkage methods with double descent. We illustrate our approach for high dimensional neural networks and provide detailed treatments of infinite Gaussian means models and non-parametric regression. Finally, we conclude with directions for future research.

stat.ML↗

arXiv2025

Kolmogorov GAM Networks are all you need!

Kolmogorov GAM (K-GAM) networks are shown to be an efficient architecture for training and inference. They are an additive model with an embedding that is independent of the function of interest. They provide an alternative to the transformer architecture. They are the machine learning version of Kolmogorov's Superposition Theorem (KST) which provides an efficient representations of a multivariate function. Such representations have use in machine learning for encoding dictionaries (a.k.a. "look-up" tables). KST theory also provides a representation based on translates of the Köppen function. The goal of our paper is to interpret this representation in a machine learning context for applications in Artificial Intelligence (AI). Our architecture is equivalent to a topological embedding which is independent of the function together with an additive layer that uses a Generalized Additive Model (GAM). This provides a class of learning procedures with far fewer parameters than current deep learning algorithms. Implementation can be parallelizable which makes our algorithms computationally attractive. To illustrate our methodology, we use the Iris data from statistical learning. We also show that our additive model with non-linear embedding provides an alternative to transformer architectures which from a statistical viewpoint are kernel smoothers. Additive KAN models therefore provide a natural alternative to transformers. Finally, we conclude with directions for future research.

cs.LG↗

arXiv2024

Generative Causal Inference

Generative Bayesian Computation (GBC) methods are developed for Casual Inference. Generative methods are simulation-based methods that use a large training dataset to represent posterior distributions as a map (a.k.a. optimal transport) to a base distribution. They avoid the use of MCMC by replacing the conditional posterior inference problem with a supervised learning problem. We further propose the use Quantile ReLU networks which are density free and hence apply in a variety of Econometric settings where data generating processes are specified by deterministic latent variables updates or as moment constraints. Generative approaches directly simulate large samples of observables and unobservable (parameters, latent variables) and then apply high-dimensional quantile regression to learn a nonlinear transport map from base distribution to parameter inference. We illustrate our methodology in the field of causal inference. Our approach can also handle nonlinearity and heterogeneity. Finally, we conclude with the directions for future research.

stat.ME↗

arXiv2024

Kramnik vs Nakamura: A Chess Scandal

We provide a statistical analysis of the recent controversy between Vladimir Kramnik (ex chess world champion) and Hikaru Nakamura. Hikaru Nakamura is a chess prodigy and a five-time United States chess champion. Kramnik called into question Nakamura's 45.5 out of 46 win streak in an online blitz contest at chess.com. We assess the weight of evidence using a priori assessment of Viswanathan Anand and the streak evidence. Based on this evidence, we show that Nakamura has a 99.6 percent chance of not cheating. We study the statistical fallacies prevalent in both their analyses. On the one hand Kramnik bases his argument on the probability of such a streak is very small. This falls precisely into the Prosecutor's Fallacy. On the other hand, Nakamura tries to refute the argument using a cherry-picking argument. This violates the likelihood principle. We conclude with a discussion of the relevant statistical literature on the topic of fraud detection and the analysis of streaks in sports data.

stat.AP↗

arXiv2024

Negative Probability

Negative probabilities arise primarily in physics, statistical quantum mechanics and quantum computing. Negative probabilities arise as mixing distributions of unobserved latent variables in Bayesian modeling. Our goal is to provide a link between these two viewpoints. Bartlett provides a definition of negative probabilities based on extraordinary random variables and properties of their characteristic function. A version of Bayes rule is given with negative mixing weights. The classic half coin distribution and Polya-Gamma mixing is discussed. Heisenberg's principle of uncertainty and the duality of scale mixtures of Normals is also discussed. A number of examples of dual densities with negative mixing measures are provided including the Linnik and Wigner distributions. Finally, we conclude with directions for future research.

quant-ph↗

arXiv2024

Generative Bayesian Computation for Maximum Expected Utility

Generative Bayesian Computation (GBC) methods are developed to provide an efficient computational solution for maximum expected utility (MEU). We propose a density-free generative method based on quantiles that naturally calculates expected utility as a marginal of quantiles. Our approach uses a deep quantile neural estimator to directly estimate distributional utilities. Generative methods assume only the ability to simulate from the model and parameters and as such are likelihood-free. A large training dataset is generated from parameters and output together with a base distribution. Our method a number of computational advantages primarily being density-free with an efficient estimator of expected utility. A link with the dual theory of expected utility and risk taking is also discussed. To illustrate our methodology, we solve an optimal portfolio allocation problem with Bayesian learning and a power utility (a.k.a. fractional Kelly criterion). Finally, we conclude with directions for future research.

stat.CO↗

arXiv2024

Counting $N$ Queens

Gauss proposed the problem of how to enumerate the number of solutions for placing $N$ queens on an $N\times N$ chess board, so no two queens attack each other. The N-queen problem is a classic problem in combinatorics. We describe a variety of Monte Carlo (MC) methods for counting the number of solutions. In particular, we propose a quantile re-ordering based on the Lorenz curve of a sum that is related to counting the number of solutions. We show his approach leads to an efficient polynomial-time solution. Other MC methods include vertical likelihood Monte Carlo, importance sampling, slice sampling, simulated annealing, energy-level sampling, and nested-sampling. Sampling binary matrices that identify the locations of the queens on the board can be done with a Swendsen-Wang style algorithm. Our Monte Carlo approach counts the number of solutions in polynomial time.

stat.CO↗

arXiv2024

Generative AI for Bayesian Computation

Bayesian Generative AI (BayesGen-AI) methods are developed and applied to Bayesian computation. BayesGen-AI reconstructs the posterior distribution by directly modeling the parameter of interest as a mapping (a.k.a. deep learner) from a large simulated dataset. This provides a generator that we can evaluate at the observed data and provide draws from the posterior distribution. This method applies to all forms of Bayesian inference including parametric models, likelihood-free models, prediction and maximum expected utility problems. Bayesian computation is then equivalent to high dimensional non-parametric regression. Bayes Gen-AI main advantage is that it is density-free and therefore provides an alternative to Markov Chain Monte Carlo. It has a number of advantages over vanilla generative adversarial networks (GAN) and approximate Bayesian computation (ABC) methods due to the fact that the generator is simpler to learn than a GAN architecture and is more flexible than kernel smoothing implicit in ABC methods. Design of the Network Architecture requires careful selection of features (a.k.a. dimension reduction) and nonlinear architecture for inference. As a generic architecture, we propose a deep quantile neural network and a uniform base distribution at which to evaluate the generator. To illustrate our methodology, we provide two real data examples, the first in traffic flow prediction and the second in building a surrogate for satellite drag data-set. Finally, we conclude with directions for future research.

stat.CO↗

arXiv2023

Deep Learning: A Tutorial

Our goal is to provide a review of deep learning methods which provide insight into structured high-dimensional data. Rather than using shallow additive architectures common to most statistical models, deep learning uses layers of semi-affine input transformations to provide a predictive rule. Applying these layers of transformations leads to a set of attributes (or, features) to which probabilistic statistical methods can be applied. Thus, the best of both worlds can be achieved: scalable prediction rules fortified with uncertainty quantification, where sparse regularization finds the features.

stat.ML↗

arXiv2023

The Value of Chess Squares

We propose a neural network-based approach to calculate the value of a chess square-piece combination. Our model takes a triplet (Color, Piece, Square) as an input and calculates a value that measures the advantage/disadvantage of having this piece on this square. Our methods build on recent advances in chess AI, and can accurately assess the worth of positions in a game of chess. The conventional approach assigns fixed values to pieces $(\symking=\infty, \symqueen=9, \symrook=5, \symbishop=3, \symknight=3, \sympawn=1)$. We enhance this analysis by introducing marginal valuations. We use deep Q-learning to estimate the parameters of our model. We demonstrate our method by examining the positioning of Knights and Bishops, and also provide valuable insights into the valuation of pawns. Finally, we conclude by suggesting potential avenues for future research.

cs.AI↗

arXiv2023

Feature Selection for Personalized Policy Analysis

In this paper, we propose Forest-PLS, a feature selection method for analyzing policy effect heterogeneity in a more flexible and comprehensive manner than is typically available with conventional methods. In particular, our method is able to capture policy effect heterogeneity both within and across subgroups of the population defined by observable characteristics. To achieve this, we employ partial least squares to identify target components of the population and causal forests to estimate personalized policy effects across these components. We show that the method is consistent and leads to asymptotically normally distributed policy effects. To demonstrate the efficacy of our approach, we apply it to the data from the Pennsylvania Reemployment Bonus Experiments, which were conducted in 1988-1989. The analysis reveals that financial incentives can motivate some young non-white individuals to enter the labor market. However, these incentives may also provide a temporary financial cushion for others, dissuading them from actively seeking employment. Our findings highlight the need for targeted, personalized measures for young non-white male participants.

econ.EM↗

arXiv2023

Deep Partial Least Squares for Instrumental Variable Regression

In this paper, we propose deep partial least squares for the estimation of high-dimensional nonlinear instrumental variable regression. As a precursor to a flexible deep neural network architecture, our methodology uses partial least squares for dimension reduction and feature selection from the set of instruments and covariates. A central theoretical result, due to Brillinger (2012) shows that the feature selection provided by partial least squares is consistent and the weights are estimated up to a proportionality constant. We illustrate our methodology with synthetic datasets with a sparse and correlated network structure and draw applications to the effect of childbearing on the mother's labor supply based on classic data of Angrist and Evans (1996). The results on synthetic data as well as applications show that the deep partial least squares method significantly outperforms other related methods. Finally, we conclude with directions for future research.

stat.ME↗

arXiv2023

Quantum Bayesian Computation

Quantum Bayesian Computation (QBC) is an emerging field that levers the computational gains available from quantum computers to provide an exponential speed-up in Bayesian computation. Our paper adds to the literature in two ways. First, we show how von Neumann quantum measurement can be used to simulate machine learning algorithms such as Markov chain Monte Carlo (MCMC) and Deep Learning (DL) that are fundamental to Bayesian learning. Second, we describe data encoding methods needed to implement quantum machine learning including the counterparts to traditional feature extraction and kernel embeddings methods. Our goal then is to show how to apply quantum algorithms directly to statistical machine learning problems. On the theoretical side, we provide quantum versions of high dimensional regression, Gaussian processes (Q-GP) and stochastic gradient descent (Q-SGD). On the empirical side, we apply a Quantum FFT model to Chicago housing data. Finally, we conclude with directions for future research.

stat.ML↗