Observation-Specific Uncertainty Modeling for Latent-Variable Regression in High-Dimensional Analytical Data

Abstract

Across many fields, including medical diagnostics, process monitoring, and quality control, decisions often rely on point predictions for single samples. Observation-specific uncertainty is needed to assess the risk of decision error when predictions are compared with thresholds. Latent-variable regression is widely used for such predictions, but observation-specific uncertainty remains difficult to estimate. Traditional methods assume unbiased predictions, attributing prediction uncertainty solely to variance. However, dimensionality reduction can introduce observation-specific bias, violating this assumption. We introduce a framework using Gaussian processes trained on latent scores and prediction errors to estimate observation-specific bias and error variance, separating systematic error from epistemic and aleatoric variance. In Raman measurements of a fructose-sucrose model system, estimated prediction-error distributions agreed with empirical distributions; bias correction reduced mean squared error, and decision limits incorporating bias and variance improved analyte detection over variance-only limits. An aqueous-solubility application using molecular descriptors demonstrated the framework's generalisation beyond spectroscopy.