arXiv subjects

Linda S. L. Tan

Publications and source records attributed to Linda S. L. Tan.

At least 19 recordsLinked to original sources

arXiv2026

Generalized reparametrized variational Bayes with skew-symmetric normalization

Bayesian hierarchical models with high-dimensional latent structure require scalable posterior approximations that preserve key dependencies while remaining computationally tractable. Mean-field variational inference (MFVI) is efficient, but can be unreliable when local variables are strongly correlated or tightly coupled to global variables. We propose KNorm-RVB, a generalized reparametrized variational Bayes framework for latent Gaussian and latent non-Gaussian models with sparse local precision matrices. KNorm-RVB maps the conditional posterior of local variables toward a standard Gaussian via normalization followed by skewness reduction, enabled by a novel K-component skew-symmetric density representation. This reparametrization centers the transformed conditional local posterior at an optimized reflection point and decorrelates local and global variables, making MFVI much more effective. Under symmetry conditions, we show that MFVI recovers the local posterior mean and correlation matrix exactly, motivating KNorm-RVB's normalization and symmetrization of the conditional local posterior before applying MFVI. We combine a Gaussian variational family for reparametrized local variables with a flexible closed skew normal family for the remaining variables. Across generalized linear mixed models, mixed multinomial logit models, spatial autoregressive models, and stochastic volatility models, KNorm-RVB improves posterior approximation accuracy over existing methods.

stat.ME↗

arXiv2026

Optimization-centric cutting feedback for semiparametric models

Complex statistical models are often built by combining multiple submodels, called modules. Here we consider modular inference where the modules contain both parametric and nonparametric components. In such cases, standard Bayesian inference can be highly sensitive to misspecification in any module, and influential prior specifications for the nonparametric components can compromise inference for the parametric components, and vice versa. We propose a novel "optimization-centric" approach to cutting feedback for semiparametric modular inference, which can address misspecification and prior-data conflicts. The proposed cut posteriors are defined via a variational optimization problem like other generalized posteriors, but regularization is based on Rényi divergence, instead of Kullback-Leibler divergence (KLD). We show empirically that defining the cut posterior using Rényi divergence delivers more robust inference than KLD, and Rényi divergence reduces the tendency to underestimate uncertainty when the variational approximations impose strong parametric or independence assumptions. Novel posterior concentration results that accommodate the Rényi divergence and allow for semiparametric components are derived, extending existing results for cut posteriors that only apply to KLD and parametric models. These new methods are demonstrated in a benchmark example and two real examples: Gaussian process adjustments for confounding in causal inference and misspecified copula models with nonparametric marginals.

stat.ME↗

arXiv2025

Weighted Fisher divergence for high-dimensional Gaussian variational inference

Bayesian inference has many advantages for complex models, but standard Monte Carlo methods for summarizing the posterior can be computationally demanding, and it is attractive to consider optimization-based variational methods. Our work considers Gaussian approximations with sparse precision matrices which are tractable to optimize in high-dimensions. The optimal Gaussian approximation is usually defined as being closest to the posterior in Kullback-Leibler divergence, but it is useful to consider other divergences when the Gaussian assumption is crude, to capture important posterior features for given applications. Our work studies the weighted Fisher divergence, which focuses on gradient differences between the target posterior and its approximation, with the Fisher and score-based divergences as special cases. We make three main contributions. First, we compare approximations for weighted Fisher divergences under mean-field assumptions for Gaussian and non-Gaussian targets with Kullback-Leibler approximations. Second, we go beyond mean-field and consider approximations with sparse precision matrices reflecting posterior conditional independence structure for hierarchical models. Using stochastic gradient descent to enforce sparsity, we develop two approaches to minimize the Fisher and score-based divergences, based on the reparametrization trick and a batch approximation of the objective. Finally, we study the performances of our methods using logistic regression, generalized linear mixed models and stochastic volatility models.

stat.CO↗

arXiv2025

Variational inference for hierarchical models with conditional scale and skewness corrections

Gaussian variational approximations are widely used for summarizing posterior distributions in Bayesian models, especially in high-dimensional settings. However, a drawback of such approximations is the inability to capture skewness or more complex features of the posterior. Recent work suggests applying skewness corrections to existing Gaussian or other symmetric approximations to address this limitation. We propose to incorporate the skewness correction into the definition of an approximating variational family. We consider approximating the posterior for hierarchical models, in which there are ``global'' and ``local'' parameters. A baseline variational approximation is defined as the product of a Gaussian marginal posterior for global parameters and a Gaussian conditional posterior for local parameters given the global ones. Skewness corrections are then considered. The adjustment of the conditional posterior term for local variables is adaptive to the global parameter value. Optimization of baseline variational parameters is performed jointly with the skewness correction. Our approach allows the location, scale and skewness to be captured separately, without using additional parameters for skewness adjustments. The proposed method substantially improves accuracy for only a modest increase in computational cost compared to state-of-the-art Gaussian approximations. Good performance is demonstrated in generalized linear mixed models and multinomial logit discrete choice models.

stat.ME↗

arXiv2025

Analytic natural gradient updates for Cholesky factor in Gaussian variational approximation

Natural gradients can improve convergence in stochastic variational inference significantly but inverting the Fisher information matrix is daunting in high dimensions. Moreover, in Gaussian variational approximation, natural gradient updates of the precision matrix do not ensure positive definiteness. To tackle this issue, we derive analytic natural gradient updates of the Cholesky factor of the covariance or precision matrix, and consider sparsity constraints representing different posterior correlation structures. Stochastic normalized natural gradient ascent with momentum is proposed for implementation in generalized linear mixed models and deep neural networks.

stat.CO↗

arXiv2024

Variational inference based on a subclass of closed skew normals

Gaussian distributions are widely used in Bayesian variational inference to approximate intractable posterior densities, but the ability to accommodate skewness can improve approximation accuracy significantly, when data or prior information is scarce. We study the properties of a subclass of closed skew normals constructed using affine transformation of independent standardized univariate skew normals as the variational density, and illustrate how it provides increased flexibility and accuracy in approximating the joint posterior in various applications, by overcoming limitations in existing skew normal variational approximations. The evidence lower bound is optimized using stochastic gradient ascent, where analytic natural gradient updates are derived. We also demonstrate how problems in maximum likelihood estimation of skew normal parameters occur similarly in stochastic variational inference, and can be resolved using the centered parametrization. Supplemental materials are available online.

stat.ME↗

arXiv2022

Second order stochastic gradient update for Cholesky factor in Gaussian variational approximation from Stein's Lemma

In stochastic variational inference, use of the reparametrization trick for the multivariate Gaussian gives rise to efficient updates for the mean and Cholesky factor of the covariance matrix, which depend on the first order derivative of the log joint model density. In this article, we show that an alternative unbiased gradient estimate for the Cholesky factor which depends on the second order derivative of the log joint model density can be derived using Stein's Lemma. This leads to a second order stochastic gradient update for the Cholesky factor which is able to improve convergence, as it has variance lower than the first order update (almost negligible) when close to the mode. We also derive second order update for the Cholesky factor of the precision matrix, which is useful when the precision matrix has a sparse structure reflecting conditional independence in the true posterior distribution. Our results can be used to obtain second order natural gradient updates for the Cholesky factor as well, which are more robust compared to updates based on Euclidean gradients.

stat.ME↗

arXiv2022

Efficient data augmentation techniques for some classes of state space models

Data augmentation improves the convergence of iterative algorithms, such as the EM algorithm and Gibbs sampler by introducing carefully designed latent variables. In this article, we first propose a data augmentation scheme for the first-order autoregression plus noise model, where optimal values of working parameters introduced for recentering and rescaling of the latent states, can be derived analytically by minimizing the fraction of missing information in the EM algorithm. The proposed data augmentation scheme is then utilized to design efficient Markov chain Monte Carlo (MCMC) algorithms for Bayesian inference of some non-Gaussian and nonlinear state space models, via a mixture of normals approximation coupled with a block-specific reparametrization strategy. Applications on simulated and benchmark real datasets indicate that the proposed MCMC sampler can yield improvements in simulation efficiency compared with centering, noncentering and even the ancillarity-sufficiency interweaving strategy.

stat.ME↗

arXiv2020

Use of model reparametrization to improve variational Bayes

We propose using model reparametrization to improve variational Bayes inference for hierarchical models whose variables can be classified as global (shared across observations) or local (observation specific). Posterior dependence between local and global variables is minimized by applying an invertible affine transformation on the local variables. The functional form of this transformation is deduced by approximating the posterior distribution of each local variable conditional on the global variables by a Gaussian density via a second order Taylor expansion. Variational Bayes inference for the reparametrized model is then obtained using stochastic approximation. Our approach can be readily extended to large datasets via a divide and recombine strategy. Using generalized linear mixed models, we demonstrate that reparametrized variational Bayes (RVB) provides improvements in both accuracy and convergence rate compared to state of the art Gaussian variational approximation methods.

stat.ME↗

arXiv2019

Bayesian variational inference for exponential random graph models

Deriving Bayesian inference for exponential random graph models (ERGMs) is a challenging "doubly intractable" problem as the normalizing constants of the likelihood and posterior density are both intractable. Markov chain Monte Carlo (MCMC) methods which yield Bayesian inference for ERGMs, such as the exchange algorithm, are asymptotically exact but computationally intensive, as a network has to be drawn from the likelihood at every step using, for instance, a "tie no tie" sampler. In this article, we develop a variety of variational methods for Gaussian approximation of the posterior density and model selection. These include nonconjugate variational message passing based on an adjusted pseudolikelihood and stochastic variational inference. To overcome the computational hurdle of drawing a network from the likelihood at each iteration, we propose stochastic gradient ascent with biased but consistent gradient estimates computed using adaptive self-normalized importance sampling. These methods provide attractive fast alternatives to MCMC for posterior approximation. We illustrate the variational methods using real networks and compare their accuracy with results obtained via MCMC and Laplace approximation.

stat.CO↗

arXiv2019

Conditionally structured variational Gaussian approximation with importance weights

We develop flexible methods of deriving variational inference for models with complex latent variable structure. By splitting the variables in these models into "global" parameters and "local" latent variables, we define a class of variational approximations that exploit this partitioning and go beyond Gaussian variational approximation. This approximation is motivated by the fact that in many hierarchical models, there are global variance parameters which determine the scale of local latent variables in their posterior conditional on the global parameters. We also consider parsimonious parametrizations by using conditional independence structure, and improved estimation of the log marginal likelihood and variational density using importance weights. These methods are shown to improve significantly on Gaussian variational approximation methods for a similar computational cost. Application of the methodology is illustrated using generalized linear mixed models and state space models.

stat.CO↗

arXiv2018

Explicit inverse of tridiagonal matrix with applications in autoregressive modeling

We present the explicit inverse of a class of symmetric tridiagonal matrices which is almost Toeplitz, except that the first and last diagonal elements are different from the rest. This class of tridiagonal matrices are of special interest in complex statistical models which uses the first order autoregression to induce dependence in the covariance structure, for instance, in econometrics or spatial modeling. They also arise in interpolation problems using the cubic spline. We show that the inverse can be expressed as a linear combination of Chebyshev polynomials of the second kind and present results on the properties of the inverse, such as bounds on the row sums, the trace of the inverse and its square, and their limits as the order of the matrix increases.

math.NA↗

arXiv2017

Dynamic degree-corrected blockmodels for social networks: a nonparametric approach

A nonparametric approach to the modeling of social networks using degree-corrected stochastic blockmodels is proposed. The model for static network consists of a stochastic blockmodel using a probit regression formulation and popularity parameters are incorporated to account for degree heterogeneity. Dirichlet processes are used to detect community structure as well as induce clustering in the popularity parameters. This approach is flexible yet parsimonious as it allows the appropriate number of communities and popularity clusters to be determined automatically by the data. We further discuss some ways of extending the static model to dynamic networks. We consider a Bayesian approach and derive Gibbs samplers for posterior inference. The models are illustrated using several real-world benchmark social networks.

stat.AP↗

arXiv2017

Gaussian variational approximation with sparse precision matrices

We consider the problem of learning a Gaussian variational approximation to the posterior distribution for a high-dimensional parameter, where we impose sparsity in the precision matrix to reflect appropriate conditional independence structure in the model. Incorporating sparsity in the precision matrix allows the Gaussian variational distribution to be both flexible and parsimonious, and the sparsity is achieved through parameterization in terms of the Cholesky factor. Efficient stochastic gradient methods which make appropriate use of gradient information for the target distribution are developed for the optimization. We consider alternative estimators of the stochastic gradients which have lower variation and are more stable. Our approach is illustrated using generalized linear mixed models and state space models for time series.

stat.CO↗

arXiv2017

Bayesian inference for multiple Gaussian graphical models with application to metabolic association networks

We investigate the effect of cadmium (a toxic environmental pollutant) on the correlation structure of a number of urinary metabolites using Gaussian graphical models (GGMs). The inferred metabolic associations can provide important information on the physiological state of a metabolic system and insights on complex metabolic relationships. Using the fitted GGMs, we construct differential networks, which highlight significant changes in metabolite interactions under different experimental conditions. The analysis of such metabolic association networks can reveal differences in the underlying biological reactions caused by cadmium exposure. We consider Bayesian inference and propose using the multiplicative (or Chung-Lu random graph) model as a prior on the graphical space. In the multiplicative model, each edge is chosen independently with probability equal to the product of the connectivities of the end nodes. This class of prior is parsimonious yet highly flexible; it can be used to encourage sparsity or graphs with a pre-specified degree distribution when such prior knowledge is available. We extend the multiplicative model to multiple GGMs linking the probability of edge inclusion through logistic regression and demonstrate how this leads to joint inference for multiple GGMs. A sequential Monte Carlo (SMC) algorithm is developed for estimating the posterior distribution of the graphs.

stat.ME↗

arXiv2015

Topic-adjusted visibility metric for scientific articles

Measuring the impact of scientific articles is important for evaluating the research output of individual scientists, academic institutions and journals. While citations are raw data for constructing impact measures, there exist biases and potential issues if factors affecting citation patterns are not properly accounted for. In this work, we address the problem of field variation and introduce an article level metric useful for evaluating individual articles' visibility. This measure derives from joint probabilistic modeling of the content in the articles and the citations amongst them using latent Dirichlet allocation (LDA) and the mixed membership stochastic blockmodel (MMSB). Our proposed model provides a visibility metric for individual articles adjusted for field variation in citation rates, a structural understanding of citation behavior in different fields, and article recommendations which take into account article visibility and citation patterns. We develop an efficient algorithm for model fitting using variational methods. To scale up to large networks, we develop an online variant using stochastic gradient methods and case-control likelihood approximation. We apply our methods to the benchmark KDD Cup 2003 dataset with approximately 30,000 high energy physics papers.

stat.ML↗

arXiv2015

Stochastic variational inference for large-scale discrete choice models using adaptive batch sizes

Discrete choice models describe the choices made by decision makers among alternatives and play an important role in transportation planning, marketing research and other applications. The mixed multinomial logit (MMNL) model is a popular discrete choice model that captures heterogeneity in the preferences of decision makers through random coefficients. While Markov chain Monte Carlo methods provide the Bayesian analogue to classical procedures for estimating MMNL models, computations can be prohibitively expensive for large datasets. Approximate inference can be obtained using variational methods at a lower computational cost with competitive accuracy. In this paper, we develop variational methods for estimating MMNL models that allow random coefficients to be correlated in the posterior and can be extended easily to large-scale datasets. We explore three alternatives: (1) Laplace variational inference, (2) nonconjugate variational message passing and (3) stochastic linear regression. Their performances are compared using real and simulated data. To accelerate convergence for large datasets, we develop stochastic variational inference for MMNL models using each of the above alternatives. Stochastic variational inference allows data to be processed in minibatches by optimizing global variational parameters using stochastic gradient approximation. A novel strategy for increasing minibatch sizes adaptively within stochastic variational inference is proposed.

stat.AP↗

arXiv2015

Variational inference for sparse spectrum Gaussian process regression

We develop a fast variational approximation scheme for Gaussian process (GP) regression, where the spectrum of the covariance function is subjected to a sparse approximation. Our approach enables uncertainty in covariance function hyperparameters to be treated without using Monte Carlo methods and is robust to overfitting. Our article makes three contributions. First, we present a variational Bayes algorithm for fitting sparse spectrum GP regression models that uses nonconjugate variational message passing to derive fast and efficient updates. Second, we propose a novel adaptive neighbourhood technique for obtaining predictive inference that is effective in dealing with nonstationarity. Regression is performed locally at each point to be predicted and the neighbourhood is determined using a measure defined based on lengthscales estimated from an initial fit. Weighting dimensions according to lengthscales, this downweights variables of little relevance, leading to automatic variable selection and improved prediction. Third, we introduce a technique for accelerating convergence in nonconjugate variational message passing by adapting step sizes in the direction of the natural gradient of the lower bound. Our adaptive strategy can be easily implemented and empirical results indicate significant speedups.

stat.CO↗