arXiv subjects

Nikola Surjanovic

Publications and source records attributed to Nikola Surjanovic.

14 recordsLinked to original sources

arXiv2026

Hessian-informed, Coordinate Friendly Hamiltonian Monte Carlo in Linear Time

Riemannian Hamiltonian Monte Carlo (RHMC) is a promising MCMC methodology thanks to its ability to accommodate position-dependent preconditioning and multi-step proposals. While RHMC performs well in low dimensions, it becomes infeasible in high dimensions due to its $O(d^3)$ cost per fixed-point iteration, where $d$ is the dimension of the target density. Even when the position-dependent preconditioner is based on the diagonal of the Hessian, the cost is still $O(d^2)$ per fixed-point iteration. In this paper, we propose a computational method to reduce the computational complexity of RHMC fixed-point iterations with diagonal preconditioners from $O(d^2)$ to $O(d)$ for targets with ``coordinate friendly'' structures. This distribution class includes generalized linear models as well as other dense and sparse graphical models. The method is expressed as manipulating the compute graph and can therefore be automated to work on black box targets. Finally, we show empirically that our implementation of RHMC results in better sample quality per unit of compute time for various target distributions compared to state-of-the-art HMC NUTS algorithms with both position-independent and position-dependent preconditioners.

stat.CO↗

arXiv2025

AutoGD: Automatic Learning Rate Selection for Gradient Descent

The performance of gradient-based optimization methods, such as standard gradient descent (GD), greatly depends on the choice of learning rate. However, it can require a non-trivial amount of user tuning effort to select an appropriate learning rate schedule. When such methods appear as inner loops of other algorithms, expecting the user to tune the learning rates may be impractical. To address this, we introduce AutoGD: a gradient descent method that automatically determines whether to increase or decrease the learning rate at a given iteration. We establish the convergence of AutoGD, and show that we can recover the optimal rate of GD (up to a constant) for a broad class of functions without knowledge of smoothness constants. Experiments on a variety of traditional problems and variational inference optimization tasks demonstrate strong performance of the method, along with its extensions to AutoBFGS and AutoLBFGS.

cs.LG↗

arXiv2025

AutoSGD: Automatic Learning Rate Selection for Stochastic Gradient Descent

The learning rate is an important tuning parameter for stochastic gradient descent (SGD) and can greatly influence its performance. However, appropriate selection of a learning rate schedule across all iterations typically requires a non-trivial amount of user tuning effort. To address this, we introduce AutoSGD: an SGD method that automatically determines whether to increase or decrease the learning rate at a given iteration and then takes appropriate action. We introduce theory supporting the convergence of AutoSGD, along with its deterministic counterpart for standard gradient descent. Empirical results suggest strong performance of the method on a variety of traditional optimization problems and machine learning tasks.

cs.LG↗

arXiv2025

AutoStep: Locally adaptive involutive MCMC

Many common Markov chain Monte Carlo (MCMC) kernels can be formulated using a deterministic involutive proposal with a step size parameter. Selecting an appropriate step size is often a challenging task in practice; and for complex multiscale targets, there may not be one choice of step size that works well globally. In this work, we address this problem with a novel class of involutive MCMC methods -- AutoStep MCMC -- that selects an appropriate step size at each iteration adapted to the local geometry of the target distribution. We prove that under mild conditions AutoStep MCMC is $π$-invariant, irreducible, and aperiodic, and obtain bounds on expected energy jump distance and cost per iteration. Empirical results examine the robustness and efficacy of our proposed step size selection procedure, and show that AutoStep MCMC is competitive with state-of-the-art methods in terms of effective sample size per unit cost on a range of challenging target distributions.

stat.CO↗

arXiv2025

Is Gibbs sampling faster than Hamiltonian Monte Carlo on GLMs?

The Hamiltonian Monte Carlo (HMC) algorithm is often lauded for its ability to effectively sample from high-dimensional distributions. In this paper we challenge the presumed domination of HMC for the Bayesian analysis of GLMs. By utilizing the structure of the compute graph rather than the graphical model, we show a reduction of the time per sweep of a full-scan Gibbs sampler from $O(d^2)$ to $O(d)$, where $d$ is the number of GLM parameters. A simple change to the implementation of the Gibbs sampler allows us to perform Bayesian inference on high-dimensional GLMs that are practically infeasible with traditional Gibbs sampler implementations. We empirically demonstrate a substantial increase in effective sample size per time when comparing our Gibbs algorithms to state-of-the-art HMC algorithms. While Gibbs is superior in terms of dimension scaling, neither Gibbs nor HMC dominate the other: we provide numerical and theoretical evidence that HMC retains an edge in certain circumstances thanks to its advantageous condition number scaling. Interestingly, for GLMs of fixed data size, we observe that increasing dimensionality can stabilize or even decrease condition number, shedding light on the empirical advantage of our efficient Gibbs sampler.

stat.CO↗

arXiv2025

On the Orbit of the Binary Brown Dwarf Companion GL229 Ba and Bb

The companion GL229B was recently resolved by Xuan et al. (2024) as a tight binary of two brown dwarfs (Ba and Bb) through VLTI-GRAVITY interferometry and VLT-CRIRES+ RV measurements. Here, we present Bayesian models of the interferometric and RV data in additional detail, along with an updated outer orbit of the brown dwarf pair about the primary. To create a model of the inner orbit with robust uncertainties, we apply kernel phases to the GRAVITY data to address baseline redundancy in the raw closure phases. Using parallel tempering, we constrain the binary's orbit using only VLTI-GRAVITY data, despite each epoch having low visibility-plane coverage and/or SNR. We demonstrate very agreement the VLTI-GRAVITY and CRIRES+ datasets and find that the inner binary has a period of 12.1346$\pm$0.0011 days, eccentricity of 0.2317$\pm$0.0025, and total mass of 71.0$\pm$0.4 Mjup, with Ba and Bb having masses of 37.7$\pm$1.1Mjup and 33.4$\pm$1.0Mjup respectively. With new Keck/NIRC2 astrometry, we update the outer orbit GL229B around the primary. We find a semi-major axis of 42.9+3.0-2.4AU, eccentricity of 0.736$\pm$0.014, and a total mass for B of 71.7$\pm$0.6Mjup, consistent with that derived from the inner orbit. We find a mutual inclination of 31$\pm$2.5deg, below the threshold for Kozai-Lidov oscillations. The agreement on the mass of Ba+Bb between the inner and outer orbits is an important test of our ability to model RV, astrometry, and Hipparcos-Gaia proper motion anomaly. Our methodological advances in handling interferometric data with low SNR and sparse UV-coverage will benefit future observations of rapidly-orbiting companions with VLTI-GRAVITY.

astro-ph.SR↗

arXiv2024

Alpha-Trimming: Locally Adaptive Tree Pruning for Random Forests

We demonstrate that adaptively controlling the size of individual regression trees in a random forest can improve predictive performance, contrary to the conventional wisdom that trees should be fully grown. A fast pruning algorithm, alpha-trimming, is proposed as an effective approach to pruning trees within a random forest, where more aggressive pruning is performed in regions with a low signal-to-noise ratio. The amount of overall pruning is controlled by adjusting the weight on an information criterion penalty as a tuning parameter, with the standard random forest being a special case of our alpha-trimmed random forest. A remarkable feature of alpha-trimming is that its tuning parameter can be adjusted without refitting the trees in the random forest once the trees have been fully grown once. In a benchmark suite of 46 example data sets, mean squared prediction error is often substantially lowered by using our pruning algorithm and is never substantially increased compared to a random forest with fully-grown trees at default parameter settings.

stat.ML↗

arXiv2024

Uniform Ergodicity of Parallel Tempering With Efficient Local Exploration

Non-reversible parallel tempering (NRPT) is an effective algorithm for sampling from target distributions with complex geometry, such as those arising from posterior distributions of weakly identifiable and high-dimensional Bayesian models. In this work we establish the uniform (geometric) ergodicity of NRPT under a model of efficient local exploration. The uniform ergodicity log rates are inversely proportional to an easily-estimable divergence, the global communication barrier (GCB), which was recently introduced in the literature. We obtain analogous ergodicity results for classical reversible parallel tempering, providing new evidence that NRPT dominates its reversible counterpart. Our results are based on an analysis of the hitting time of a continuous-time persistent random walk, which is also of independent interest. The rates that we obtain reflect real experiments well for distributions where global exploration is not possible without parallel tempering.

stat.CO↗

arXiv2024

autoMALA: Locally adaptive Metropolis-adjusted Langevin algorithm

Selecting the step size for the Metropolis-adjusted Langevin algorithm (MALA) is necessary in order to obtain satisfactory performance. However, finding an adequate step size for an arbitrary target distribution can be a difficult task and even the best step size can perform poorly in specific regions of the space when the target distribution is sufficiently complex. To resolve this issue we introduce autoMALA, a new Markov chain Monte Carlo algorithm based on MALA that automatically sets its step size at each iteration based on the local geometry of the target distribution. We prove that autoMALA has the correct invariant distribution, despite continual automatic adjustments of the step size. Our experiments demonstrate that autoMALA is competitive with related state-of-the-art MCMC methods, in terms of the number of log density evaluations per effective sample, and it outperforms state-of-the-art samplers on targets with varying geometries. Furthermore, we find that autoMALA tends to find step sizes comparable to optimally-tuned MALA when a fixed step size suffices for the whole domain.

stat.CO↗

arXiv2024

MCMC-driven learning

This paper is intended to appear as a chapter for the Handbook of Markov Chain Monte Carlo. The goal of this chapter is to unify various problems at the intersection of Markov chain Monte Carlo (MCMC) and machine learning$\unicode{x2014}$which includes black-box variational inference, adaptive MCMC, normalizing flow construction and transport-assisted MCMC, surrogate-likelihood MCMC, coreset construction for MCMC with big data, Markov chain gradient descent, Markovian score climbing, and more$\unicode{x2014}$within one common framework. By doing so, the theory and methods developed for each may be translated and generalized.

stat.ML↗

arXiv2023

Improving the Hosmer-Lemeshow Goodness-of-Fit Test in Large Models with Replicated Trials

The Hosmer-Lemeshow (HL) test is a commonly used global goodness-of-fit (GOF) test that assesses the quality of the overall fit of a logistic regression model. In this paper, we give results from simulations showing that the type 1 error rate (and hence power) of the HL test decreases as model complexity grows, provided that the sample size remains fixed and binary replicates are present in the data. We demonstrate that the generalized version of the HL test by Surjanovic et al. (2020) can offer some protection against this power loss. We conclude with a brief discussion explaining the behaviour of the HL test, along with some guidance on how to choose between the two tests.

stat.ME↗

arXiv2023

Pigeons.jl: Distributed Sampling From Intractable Distributions

We introduce a software package, Pigeons.jl, that provides a way to leverage distributed computation to obtain samples from complicated probability distributions, such as multimodal posteriors arising in Bayesian inference and high-dimensional distributions in statistical mechanics. Pigeons.jl provides simple APIs to perform such computations single-threaded, multi-threaded, and/or distributed over thousands of MPI-communicating machines. In addition, Pigeons.jl guarantees a property that we call strong parallelism invariance: the output for a given seed is identical irrespective of the number of threads and processes, which is crucial for scientific reproducibility and software validation. We describe the key features of Pigeons.jl and the approach taken to implement a distributed and randomized algorithm that satisfies strong parallelism invariance.

stat.CO↗

arXiv2023

Parallel Tempering With a Variational Reference

Sampling from complex target distributions is a challenging task fundamental to Bayesian inference. Parallel tempering (PT) addresses this problem by constructing a Markov chain on the expanded state space of a sequence of distributions interpolating between the posterior distribution and a fixed reference distribution, which is typically chosen to be the prior. However, in the typical case where the prior and posterior are nearly mutually singular, PT methods are computationally prohibitive. In this work we address this challenge by constructing a generalized annealing path connecting the posterior to an adaptively tuned variational reference. The reference distribution is tuned to minimize the forward (inclusive) KL divergence to the posterior distribution using a simple, gradient-free moment-matching procedure. We show that our adaptive procedure converges to the forward KL minimizer, and that the forward KL divergence serves as a good proxy to a previously developed measure of PT performance. We also show that in the large-data limit in typical Bayesian models, the proposed method improves in performance, while traditional PT deteriorates arbitrarily. Finally, we introduce PT with two references -- one fixed, one variational -- with a novel split annealing path that ensures stable variational reference adaptation. The paper concludes with experiments that demonstrate the large empirical gains achieved by our method in a wide range of realistic Bayesian inference scenarios.

stat.CO↗

arXiv2021

A Generalized Hosmer-Lemeshow Goodness-of-Fit Test for a Family of Generalized Linear Models

Generalized linear models (GLMs) are used within a vast number of application domains. However, formal goodness of fit (GOF) tests for the overall fit of the model$-$so-called "global" tests$-$seem to be in wide use only for certain classes of GLMs. In this paper we develop and apply a new global goodness-of-fit test, similar to the well-known and commonly used Hosmer-Lemeshow (HL) test, that can be used with a wide variety of GLMs. The test statistic is a variant of the HL test statistic, but we rigorously derive an asymptotically correct sampling distribution of the test statistic using methods of Stute and Zhu (2002). Our new test is relatively straightforward to implement and interpret. We demonstrate the test on a real data set, and compare the performance of our new test with other global GOF tests for GLMs, finding that our test provides competitive or comparable power in various simulation settings. Our test also avoids the use of kernel-based estimators, used in various GOF tests for regression, thereby avoiding the issues of bandwidth selection and the curse of dimensionality. Since the asymptotic sampling distribution is known, a bootstrap procedure for the calculation of a p-value is also not necessary, and we therefore find that performing our test is computationally efficient.

stat.ME↗