arXiv subjects

Hector Borobia

Publications and source records attributed to Hector Borobia.

5 recordsLinked to original sources

arXiv2026

Near-Floor Geometry Is Generic: Leverage Dispersion in Trained Overcomplete Codes

An overcomplete code packs $F$ features into $d<F$ dimensions, so linear readout of one feature picks up cross-talk from the others, bounded below by the rank-trace floor $W(F,d)=(F-d)/(d(F-1))$. Proximity to this floor is read as evidence of an efficient arrangement. It is not: an i.i.d. code of the same shape attains it at every shape and load we measure (1.0000-1.0005 over 7 matched controls), and the reason is exact. For the gain-calibrated pseudoinverse the attainment ratio is $\frac{d}{F(F-d)}(\sum_i h_i^{-1}-F)$ with $\sum_i h_i=d$, so it equals one precisely when the leverage $h_i$ is equalised across features: distance from the floor IS leverage dispersion. On 40 released decoder matrices - 24 sparse-autoencoder decoders plus 16 MLP down-projections of a hybrid model - all sit above their matched control (1.0055-1.2018), and all reach guaranteed affine failure 4 to 40 sparsity levels earlier: where a random code of the same shape is still safe, a released dictionary already has a feature no affine rule recovers. Autoencoders trained by the same recipe on a randomly initialised copy of the same model return to the floor (1.0038-1.0065) in all 10 paired layers, so the excess reflects what the model learned, not the act of fitting an autoencoder. Under controlled conditions the loss decides whether an interface exists at all: $L^2$ reaches $R_{geom}=18.5635$ to 32.3131 with its frontier collapsed to $1$, against $1.0002$ for $L^4$, and that frontier follows a rate we registered before the widest run (10.79 predicted, 10.510 measured). Two limits are stated rather than implied: frontier statements are worst cases, not typical-case claims; and because the probe class contains the network's own decoder, our decoder comparison bounds supervised fitting rather than affine expressiveness.

cs.LG↗

arXiv2026

Component-Aware Self-Speculative Decoding in Hybrid Language Models

Speculative decoding accelerates autoregressive inference by drafting candidate tokens with a fast model and verifying them in parallel with the target. Self-speculative methods avoid the need for an external drafter but have been studied exclusively in homogeneous Transformer architectures. We introduce component-aware self-speculative decoding, the first method to exploit the internal architectural heterogeneity of hybrid language models, isolating the SSM/linear-attention subgraph as a zero-cost internal draft. We evaluate this on two architecturally distinct hybrid families: Falcon-H1 (parallel: Mamba-2 + attention per layer) and Qwen3.5 (sequential: interleaved linear and attention layers), with a pure Transformer control (Qwen2.5). Parallel hybrids achieve acceptance rates of alpha = 0.68 at draft length k=2 under greedy decoding, while sequential hybrids yield only alpha = 0.038 -- an 18x gap attributable to how each architecture integrates its components. The property is scale-invariant: Falcon-H1 at 3B reproduces the rates observed at 0.5B. We further show that perplexity degradation from a companion ablation study predicts speculative viability without running speculative decoding: a 3.15x ratio (Falcon) maps to alpha = 0.37 at k=4, while 81.96x (Qwen) maps to alpha = 0.019. For sequential hybrids, generic LayerSkip achieves 12x higher acceptance rates than the component-aware strategy. The composition pattern of hybrid models -- not merely the presence of alternative components -- determines whether component-level self-speculation is viable.

cs.CL↗

arXiv2026

Where Should LoRA Go? Component-Type Placement in Hybrid Language Models

Hybrid language models that interleave attention with recurrent components are increasingly competitive with pure Transformers, yet standard LoRA practice applies adapters uniformly without considering the distinct functional roles of each component type. We systematically study component-type LoRA placement across two hybrid architectures -- Qwen3.5-0.8B (sequential, GatedDeltaNet + softmax attention) and Falcon-H1-0.5B (parallel, Mamba-2 SSM + attention) -- fine-tuned on three domains and evaluated on five benchmarks. We find that the attention pathway -- despite being the minority component -- consistently outperforms full-model adaptation with 5-10x fewer trainable parameters. Crucially, adapting the recurrent backbone is destructive in sequential hybrids (-14.8 pp on GSM8K) but constructive in parallel ones (+8.6 pp). We further document a transfer asymmetry: parallel hybrids exhibit positive cross-task transfer while sequential hybrids suffer catastrophic forgetting. These results establish that hybrid topology fundamentally determines adaptation response, and that component-aware LoRA placement is a necessary design dimension for hybrid architectures.

cs.CL↗

arXiv2026

How Pruning Reshapes Features: Sparse Autoencoder Analysis of Weight-Pruned Language Models

Weight pruning is a standard technique for compressing large language models, yet its effect on learned internal representations remains poorly understood. We present the first systematic study of how unstructured pruning reshapes the feature geometry of language models, using Sparse Autoencoders (SAEs) as interpretability probes. Across three model families (Gemma 3 1B, Gemma 2 2B, Llama 3.2 1B), two pruning methods (magnitude and Wanda), and six sparsity levels (0--60%), we investigate five research questions spanning seed stability, feature survival, SAE transferability, feature fragility, and causal relevance. Our most striking finding is that rare SAE features--those with low firing rates--survive pruning far better than frequent ones, with within-condition Spearman correlations of rho = -1.0 in 11 of 17 experimental conditions. This counter-intuitive result suggests that pruning acts as implicit feature selection, preferentially destroying high-frequency generic features while preserving specialized rare ones. We further show that Wanda pruning preserves feature structure up to 3.7x better than magnitude pruning, that pre-trained SAEs remain viable on Wanda-pruned models up to 50% sparsity, and that geometric feature survival does not predict causal importance--a dissociation with implications for interpretability under compression.

cs.LG↗

arXiv2026

Component Ablation for Efficient Hybrid Language Model Architectures: Performance, Resilience, and Compression Implications

Hybrid language models combine softmax attention with linear-time sequence mechanisms such as state-space or linear-attention layers, but the functional contribution of each component type remains insufficiently characterized. We study component-level ablation in two sub-1B hybrid language models, Qwen3.5-0.8B and Falcon-H1-0.5B, using likelihood-based evaluation, downstream benchmarks, layer-wise interventions, random controls, and representation-level diagnostics. Across the tested models, removing either attention or the alternative sequence-processing pathway substantially degrades performance, indicating that both component types contribute to model behavior. Likelihood metrics are especially sensitive to the linear-attention or state-space pathway, while downstream benchmark degradation depends on task and architecture. Layer-wise ablations show that component importance is position-dependent, with the strongest effects concentrated in early or mid-network components rather than uniformly across depth. Random-removal controls further show that hybrid architectures and same-family Transformer baselines degrade differently under structural perturbation. These results suggest that component ablation is a useful diagnostic for understanding hybrid language model architectures. The findings provide evidence relevant to efficient model design, compression, robustness analysis, and deployment decisions in architectures that combine attention with alternative sequence-processing mechanisms.

cs.CL↗