arXiv subjects

Noud van Giersbergen

Publications and source records attributed to Noud van Giersbergen.

6 recordsLinked to original sources

arXiv2026

Whitening Improves Robustness to Spurious Correlations in Linear Probes

Deep neural networks tend to rely on simple features that may be spurious and thus fail to generalize. We study this problem in the setting of linear probes, where a (generalized) linear model is fitted on the representations of a (pretrained) model. We use the connection of these models to the max-margin classifier, and show they favor directions associated with large eigenvalues of the covariance matrix. Whitening removes this preference by equalizing the eigenvalues of the covariance matrix. This observation motivates whitening as a preprocessing step that can reduce reliance on spurious correlations without requiring prior knowledge of their presence or labeled data. We examine the effect of whitening on a synthetic data-generating process and standard spurious correlation benchmarks, and find that it improves robustness. We also find that whitening can improve robustness when added to existing approaches.

cs.LG↗

arXiv2025

Optimizing importance weighting in the presence of sub-population shifts

A distribution shift between the training and test data can severely harm performance of machine learning models. Importance weighting addresses this issue by assigning different weights to data points during training. We argue that existing heuristics for determining the weights are suboptimal, as they neglect the increase of the variance of the estimated model due to the finite sample size of the training data. We interpret the optimal weights in terms of a bias-variance trade-off, and propose a bi-level optimization procedure in which the weights and model parameters are optimized simultaneously. We apply this optimization to existing importance weighting techniques for last-layer retraining of deep neural networks in the presence of sub-population shifts and show empirically that optimizing weights significantly improves generalization performance.

stat.ML↗

arXiv2024

Moderating the Mediation Bootstrap for Causal Inference

Mediation analysis is a form of causal inference that investigates indirect effects and causal mechanisms. Confidence intervals for indirect effects play a central role in conducting inference. The problem is non-standard leading to coverage rates that deviate considerably from their nominal level. The default inference method in the mediation model is the paired bootstrap, which resamples directly from the observed data. However, a residual bootstrap that explicitly exploits the assumed causal structure (X->M->Y) could also be applied. There is also a debate whether the bias-corrected (BC) bootstrap method is superior to the percentile method, with the former showing liberal behavior (actual coverage too low) in certain circumstances. Moreover, bootstrap methods tend to be very conservative (coverage higher than required) when mediation effects are small. Finally, iterated bootstrap methods like the double bootstrap have not been considered due to their high computational demands. We investigate the issues mentioned in the simple mediation model by a large-scale simulation. Results are explained using graphical methods and the newly derived finite-sample distribution. The main findings are: (i) conservative behavior of the bootstrap is caused by extreme dependence of the bootstrap distribution's shape on the estimated coefficients (ii) this dependence leads to counterproductive correction of the the double bootstrap. The added randomness of the BC method inflates the coverage in the absence of mediation, but still leads to (invalid) liberal inference when the mediation effect is small.

econ.EM↗

arXiv2024

Removing Spurious Concepts from Neural Network Representations via Joint Subspace Estimation

Out-of-distribution generalization in neural networks is often hampered by spurious correlations. A common strategy is to mitigate this by removing spurious concepts from the neural network representation of the data. Existing concept-removal methods tend to be overzealous by inadvertently eliminating features associated with the main task of the model, thereby harming model performance. We propose an iterative algorithm that separates spurious from main-task concepts by jointly identifying two low-dimensional orthogonal subspaces in the neural network representation. We evaluate the algorithm on benchmark datasets for computer vision (Waterbirds, CelebA) and natural language processing (MultiNLI), and show that it outperforms existing concept removal methods

cs.LG↗

arXiv2024

Improved Tests for Mediation

Testing for a mediation effect is important in many disciplines, but is made difficult - even asymptotically - by the influence of nuisance parameters. Classical tests such as likelihood ratio (LR) and Wald (Sobel) tests have very poor power properties in parts of the parameter space, and many attempts have been made to produce improved tests, with limited success. In this paper we show that augmenting the critical region of the LR test can produce a test with much improved behavior everywhere. In fact, we first show that there exists a test of this type that is (asymptotically) exact for certain test levels $α$, including the common choices $α=.01,.05,.10.$ The critical region of this exact test has some undesirable properties. We go on to show that there is a very simple class of augmented LR critical regions which provides tests that are nearly exact, and avoid the issues inherent in the exact test. We suggest an optimal and coherent member of this class, provide the table needed to implement the test and to report p-values if desired. Simulation confirms validity with non-Gaussian disturbances, under heteroskedasticity, and in a nonlinear (logit) model. A short application of the method to an entrepreneurial attitudes study is included for illustration.

econ.EM↗

arXiv2022

A Nearly Similar Powerful Test for Mediation

This paper derives a new powerful test for mediation that is easy to use. Testing for mediation is empirically very important in psychology, sociology, medicine, economics and business, generating over 100,000 citations to a single key paper. The no-mediation hypothesis $H_{0}:θ_{1}θ_{2}=0$ also poses a theoretically interesting statistical problem since it defines a manifold that is non-regular in the origin where rejection probabilities of standard tests are extremely low. We prove that a similar test for mediation only exists if the size is the reciprocal of an integer. It is unique, but has objectionable properties. We propose a new test that is nearly similar with power close to the envelope without these abject properties and is easy to use in practice. Construction uses the general varying $g$-method that we propose. We illustrate the results in an educational setting with gender role beliefs and in a trade union sentiment application.

econ.EM↗