Near-Floor Geometry Is Generic: Leverage Dispersion in Trained Overcomplete Codes

Abstract

An overcomplete code packs $F$ features into $d<F$ dimensions, so linear readout of one feature picks up cross-talk from the others, bounded below by the rank-trace floor $W(F,d)=(F-d)/(d(F-1))$. Proximity to this floor is read as evidence of an efficient arrangement. It is not: an i.i.d. code of the same shape attains it at every shape and load we measure (1.0000-1.0005 over 7 matched controls), and the reason is exact. For the gain-calibrated pseudoinverse the attainment ratio is $\frac{d}{F(F-d)}(\sum_i h_i^{-1}-F)$ with $\sum_i h_i=d$, so it equals one precisely when the leverage $h_i$ is equalised across features: distance from the floor IS leverage dispersion. On 40 released decoder matrices - 24 sparse-autoencoder decoders plus 16 MLP down-projections of a hybrid model - all sit above their matched control (1.0055-1.2018), and all reach guaranteed affine failure 4 to 40 sparsity levels earlier: where a random code of the same shape is still safe, a released dictionary already has a feature no affine rule recovers. Autoencoders trained by the same recipe on a randomly initialised copy of the same model return to the floor (1.0038-1.0065) in all 10 paired layers, so the excess reflects what the model learned, not the act of fitting an autoencoder. Under controlled conditions the loss decides whether an interface exists at all: $L^2$ reaches $R_{geom}=18.5635$ to 32.3131 with its frontier collapsed to $1$, against $1.0002$ for $L^4$, and that frontier follows a rate we registered before the widest run (10.79 predicted, 10.510 measured). Two limits are stated rather than implied: frontier statements are worst cases, not typical-case claims; and because the probe class contains the network's own decoder, our decoder comparison bounds supervised fitting rather than affine expressiveness.