Euclid preparation. LXXXIX. Accurate and precise data-driven angular power spectrum covariances
We develop techniques for generating accurate and precise internal covariances for measurements of clustering and weak-lensing angular power spectra. These methods have been designed to produce non-singular and unbiased covariances for Euclid's large anticipated data vector and will be critical for validation against observational systematic effects. We constructed jackknife segments that are equal in area to a high precision by adapting the binary space partition algorithm to work on arbitrarily shaped regions on the unit sphere. Jackknife estimates of the covariances are internally derived and require no assumptions about cosmology or galaxy population and bias. Our covariance estimation, called DICES (Debiased Internal Covariance Estimation with Shrinkage), first estimated a noisy covariance through conventional delete-1 jackknife resampling. This was followed by linear shrinkage of the empirical correlation matrix towards the Gaussian prediction, rather than linear shrinkage of the covariance matrix. Shrinkage ensures the covariance is non-singular and therefore invertible, which is critical for the estimation of likelihoods and validation. We then applied a delete-2 jackknife bias correction to the diagonal components of the jackknife covariance that removed the general tendency for jackknife error estimates to be biased high. We validated internally derived covariances, which used the jackknife resampling technique, on synthetic Euclid-like lognormal catalogues. We demonstrate that DICES produces accurate, non-singular covariance estimates, with the relative error improving by 33% for the covariance and 48% for the correlation structure in comparison to jackknife estimates. These estimates can be used for highly accurate regression and inference.