Diagnosing the conditional-mean barrier in scientific machine-learning surrogates

Abstract

Many prediction tasks in computational science and engineering become one-to-many after coarse graining and partial observation. In such settings, deterministic surrogates trained by squared loss may learn a well-defined mathematical object, the conditional mean, while still missing the task-relevant variability in the underlying conditional law. In this work, we formulate this limitation as the conditional-mean barrier and develop a diagnostic framework for identifying it in fitted scientific machine-learning surrogates. The framework combines residual-feature orthogonality and effect-size diagnostics to distinguish deterministic underfitting from irreducible conditional variability. We also make explicit a simple consequence of paired squared loss: stochastic outputs do not by themselves overcome the barrier, because the objective penalizes model variance and drives the predictor back to the conditional mean. The diagnosis therefore yields a modeling prescription: when residual variability matters, the loss must score richer features of the conditional law rather than a point prediction. Reproducible numerical studies on a controlled two-branch law and a two-scale Lorenz-96 closure problem show how the diagnostic identifies the barrier, how deterministic closures can suppress collective fluctuation statistics in rollout, and how a minimal likelihood-based stochastic-scale model can recover substantially more variability.