Evaluation-Recording Contamination in Learned Nano-Quadrotor Dynamics: A Fresh-Seed Audit

Abstract

Learned flight-dynamics models are often trained on short windows extracted from longer recordings. Randomly splitting these windows can place dependent samples from the same physical flight in both training and evaluation sets. We audit this issue using a fixed snapshot of the NanoBench Crazyflie 2.1 dataset. The experiment holds evaluation flights, rollout starts, training-window count, validation data, and optimization budget fixed. In the contaminated protocol, [LeakagePercent] percent of a recording-disjoint training set is replaced with windows from evaluation recordings, while both arms use the same clean validation set. We compare a world-coordinate delta multilayer perceptron with a relative-coordinate control across [NumSeeds] fresh training seeds and [NumEvalFlights] complete evaluation recordings. The primary endpoint is failure-aware position RMSE over 1-second rollouts, analyzed with paired crossed recording-by-seed inference. Contamination lowers apparent world-model error by 12.1 percent, from 0.299 m to 0.262 m, but the predeclared 95 percent confidence interval for contaminated minus disjoint error is -0.0735 to 0.0011 m. Because the interval crosses zero, the confirmatory result is negative. Secondary horizons and the relative-coordinate model show similar trends, but no coordinate-representation interaction is supported. Reliable leakage assessment therefore requires recording-level splits, failure-aware rollouts, replication across fresh seeds, and inference over both recordings and optimization seeds.