A flexible Bayesian framework for detecting cross-sample spatial expression variability in heterogeneous tissues

Abstract

Spatial transcriptomics measures gene expression alongside the spatial coordinates of each capture spot or cell across tissue samples. The detection of spatially variable (SV) genes, whose expression exhibits systematic spatial variation, enables the delineation of functional tissue domains and the identification of region-specific transcriptional changes that underlie disease heterogeneity. However, empirical evidence from spatial transcriptomics data indicates that existing methods are hindered by two practical issues: reliance on predefined spatial patterns that often miss tissue complexity, and a lack of standardized approaches for multi-sample integration, which undermines reproducibility and biological interpretability. To address these issues, we propose a novel integrated Bayesian hierarchical model that combines flexible nonparametric spatial modeling with information sharing across samples. The model uses an adaptive spatial process that can capture a wide range of spatial patterns while remaining interpretable. We also introduce a new prior that borrows strength across samples, enabling robust detection of SV genes from multiple tissue sections. An efficient variational approximation is developed for scalable posterior computation. Analyzing spatial transcriptomics data from human brain and skin cancer tissues, our framework identifies spatially structured SV genes, enabling the delineation of tissue domains and the discovery of functionally coherent gene clusters through pathway enrichment analysis.