On Inverse Problems, Parameter Estimation, and Domain Generalization

Abstract

Signal restoration and inverse problems are key elements in most real-world data science applications. In the past decades, with the emergence of machine learning methods, inversion of measurements has become a popular step in almost all physical applications, normally executed prior to downstream tasks that often involve parameter estimation. In this work, we propose a general framework for theoretical analysis of parameter estimation in inverse problem settings. We distinguish between continuous and discrete parameter estimation, corresponding with regression and classification problems, respectively. We investigate this setting for invertible and non-invertible degradation processes, with parameter estimation that is executed directly from the observed measurements, comparing with parameter estimation after data-processing performing an inversion of the observations. Our theoretical findings align with the well-known information-theoretic data processing inequality, and to a certain degree question the common misconception that data-processing for inversion, based on modern generative models that may often produce outstanding perceptual quality, will necessarily improve the following parameter estimation objective. Importantly, by re-formulating the domain-shift problem in direct relation with discrete parameter estimation, we expose a significant vulnerability in current popular practical attempts to enforce domain generalization, which we dubbed the Double Meaning Theorem. These theoretical findings are experimentally illustrated for domain shift examples in image deblurring and speckle suppression in medical imaging. It is our hope that this paper will provide practitioners with deeper insights that may be leveraged in the future for the development of more efficient and informed strategic system planning, critical in safety-sensitive applications.