arXiv subjects

Gunner Levi Howe

Publications and source records attributed to Gunner Levi Howe.

5 recordsLinked to original sources

arXiv2026

What Makes a Representational Prior Work? Feature Families, Label-Free Invariances, and Critical Windows in Grokking

Companion work showed the grokking delay is causally the time to form task-structured representations, injectable via a contrastive prior. Here we characterize what makes such a prior work, across four axes, in 188 new runs. Content: a coherent, learnable prior built from the wrong feature family (magnitude bands) blocks generalization like a random partition (1/15 vs 0/20 grok; $p=0.43$ between them), confirming the companion's prediction that priors act at the level of the circuit's features. Supervision: a fully label-free invariance prior -- positives are commuted pairs $(a,b)\sim(b,a)$ only -- generalizes in 15/15 runs at a median $2.7\times$ speedup, more reliably than the label-supervised prior itself ($p=0.038$), and combined with a weight-norm clamp yields the strongest method we test (median $17\times$, 5/5) -- strongest meaning reliably fast: plain cross-entropy with a clamp matches this speed only at the exact critical norm, while the prior keeps it fast across the entire clamp range. Timing: the prior is only needed early -- applied solely during the first 2000 epochs (4% of budget) it generalizes 10/10 at $2.7\times$, beating continuous application (8/10, $1.25\times$) and a duration-matched later window ($2.1\times$). Setting: the dissociation replicates on modular multiplication and across depths and normalization variants, and a clamp sweep quantifies the companion's central claim: structure injection flattens the weight-norm delay-law exponent about 17-fold (plain cross-entropy slows $31\times$ per +10 norm units, a lower bound as higher cells are censored, versus $1.22\times$ with the prior). Honest boundary: tasks that generalize before memorizing have no delay to control. Feature-family alignment decides whether a prior permits generalization; invariance content suffices for acceleration without labels; a brief early window captures nearly all of the benefit.

cs.LG↗

arXiv2026

Intrinsic-Noise Consolidation: A Doob-Barrier-Conditioned Diffusion Turns Analog Device Noise into a Continual-Learning Resource

On analog neuromorphic hardware, intrinsic device noise is normally an accuracy tax. We ask whether it can instead consolidate memories. We cast per-synapse consolidation as a Doob h-transform: condition each weight's stochastic dynamics on never crossing a memory-critical barrier around its consolidated value. The conditioned diffusion gains an extra drift sigma^2 d/dw log h, a restoring force amplified by the noise variance itself that diverges at the barrier. We are explicit about novelty: the anchored drift -s(w-mu) our rule also contains is not ours (the limit of OUA, MESU, and EWC), and we surrender it. We claim only the conjunction of (a) the Doob barrier-conditioning as a synaptic rule, to our knowledge unclaimed (every h-transform use we found is generative modeling, none synaptic), and (b) a falsifiable prediction: increasing intrinsic noise non-monotonically improves sequential-task retention, an inverted-U that anchored-drift methods cannot produce. We pre-registered this as a go/no-go gate; it passes. On single-head Split-MNIST (8 seeds) the rule lifts retention 10.9 points at an interior optimum (paired Wilcoxon p=0.004), while matched OU/EWC/MESU anchors are monotone. Ablating the conditioning removes the effect; the optimum tracks the barrier; the inverted-U survives a second task stream and the realization where noise enters the forward pass. We then measure the intrinsic noise on real BrainScaleS-2 silicon (additive, trial-to-trial independent, tunable via on-chip averaging) and run the rule on the chip with its noise in the training loop: barrier-conditioning retains a prior task 15.6 points better than the matched control at matched average accuracy, a stability-plasticity shift, not a net-accuracy win (single seed; retention measured, energy modelled). Intrinsic analog noise thus becomes a consolidation dividend a digital accelerator must spend energy to generate.

cs.LG↗

arXiv2026

Heckman-Corrected Epistemic Uncertainty: Selection on Unobservables Defeats Importance Weighting

Training data for machine learning is routinely collected by a selection process the model never sees: loans are observed only when granted, outcomes only when a test was ordered. The standard fixes -- importance weighting, covariate-shift correction, MAR imputation -- assume selection is ignorable given observables. Econometrics solved the harder case in 1979: Heckman's two-equation model jointly fits a probit selection equation and an outcome equation linked through correlated errors, and the inverse-Mills-ratio term corrects for selection on unobservables, where importance weighting is structurally helpless. We instantiate this for deep epistemic uncertainty: a deep outcome network, a linear selection head, and a joint bivariate-normal likelihood over all units, ensembled for predictive variance. In a controlled generator where sampling probability depends on an unobservable correlated (rho up to 0.9) with the outcome noise, deep ensembles, MC dropout, and GP baselines are overconfident exactly where data was avoided: coverage of nominal-90% intervals falls to 64.4% at rho=0.9, and importance weighting with oracle propensities does not fix it (43.1%) -- reweighting corrects the covariate distribution, not the conditional bias E[y|x,selected] != E[y|x]. The Heckman correction restores coverage (88.9%) when the selection equation has an instrument -- a variable affecting selection but not the outcome -- and degrades measurably without one (40.3%); we chart this honesty curve rather than hide it. On real tabular data with induced MNAR selection, the corrected intervals are the best-calibrated (lowest region-ECE) non-oracle method in selected-against regions; baselines matching its raw coverage do so only by over-widening everywhere. Our estimators reproduce classic Stata output to seven digits. We state which identification regime a practitioner is in, and release the code.

cs.LG↗

arXiv2026

Level-Crossing Density as a Mesh-Free High-Frequency Auxiliary Loss for Implicit Neural Representations

The Minkowski functionals of a field's excursion sets -- area, boundary measure, and Euler characteristic -- describe its level-set morphology; the Euler characteristic is the cheapest handle on topology. We derive smooth Monte-Carlo estimators for all three of a continuous neural field, evaluated at scattered points via the co-area formula and Gauss-Bonnet, using only autodiff: no grid, no complex, no persistence. The estimator is accurate to 1-3% against exact topology in 2D and 3D, and costs about 3 ms per iteration where a persistent-homology (PH) loss on a cubical grid costs 650-1000 ms -- a 250x gap. We establish four design rules without which these losses silently fail: a dense level ladder (invariants are flat in the parameters away from transitions), a $C^2$ backbone (ReLU nets hide curvature in kinks), the full Minkowski vector (Euler characteristic alone is an alternating sum, gamed by debris-hole cancellation; pricing perimeter closes the channel), and sampling-scale coverage. In 2D the vector-valued cap is the only method in a controlled comparison that both repairs topology (3/3 seeds) and preserves fidelity -- uniform smoothing repairs at 11-17x the fidelity cost, and the Euler term alone repairs nothing. In 3D neural-SDF fitting, however, a failure mode we believe general to any sampled soft topology objective appears: gradient descent adversarially hides topological noise below the sampling density, where the estimator is blind -- spurious-feature counts are invariant to 4x more samples, and closing the window needs cubically many points, erasing the cost advantage. A grid-based PH baseline, whose complex is the evaluation resolution, solves the same benchmark ($4/9$ exact; median $b_1$ error 1 vs. ours above $10^4$). The 250x cost of persistence is, at present, the price of having no null space. We release estimators, receipts, and benchmarks.

cs.LG↗

arXiv2026

Structure-Specific Representational Priors Causally Control the Grokking Delay

Grokking -- generalization long after training-set interpolation -- has been accelerated by structure-agnostic interventions (gradient filtering, weight-norm clamping, geometric penalties). Whether the delay specifically measures the time to form task-structured representations has remained observational. We test it causally by injecting representational priors of varying content into a one-layer transformer learning modular addition, via a supervised-contrastive loss whose positives encode (i) the task's true structure ($(a+b) \bmod p$), (ii) a coherent-but-wrong sibling ($(a-b) \bmod p$), or (iii) a random partition -- all with identical loss form, strength, class sizes, and geometry. Whether generalization occurs follows a clean gradation: true 22/30 runs, sibling (same periodic features, wrong combination) 14/15, random (only memorizable) 0/20 (Fisher $p=1.3\times10^{-7}$). A weight-norm-matched control replaying the norm trajectory onto plain cross-entropy generalizes 0/15, ruling out the norm as mediator. Probes show structure formation precedes and predicts generalization in all runs. Only the true structure also accelerates grokking (up to $2.75\times$), but this is dose-dependent and bimodal. We then confirm the mechanism by prediction: because the acceleration is gated by a weight-norm side-effect, clamping the norm during training yields a reliable, standalone accelerator with a median $8.6\times$ speedup (up to $22\times$ on the fastest seeds, under 1000 epochs), growing monotonically as the norm is held lower; the residual stalls also vanish, though significant only pooled over the two mitigations run at both strengths ($0/40$ vs $6/20$, $p=7.7\times10^{-4}$), not per method. The grokking delay is, causally, the time to form the right representational structure -- decided at the level of features, not labels.

cs.LG↗