Searcharxiv⌕ Search

arXiv subjects

Rassul Magauin

Publications and source records attributed to Rassul Magauin.

2 recordsLinked to original sources

Structure--Residual Inductive Biases for Language Model Pretraining

Next-token prediction is relied upon to learn both predictive behavior and internal representations. We ask whether representation structure itself should be an explicit inductive bias. Motivated by the two-part coding view of Kolmogorov complexity, we study a structure--residual principle in which a representation separates structure from realization detail. We instantiate this principle in two complementary ways. Boundary Bottleneck Concept (BBC) routes information from token segments through boundary carriers while preserving local autoregressive computation. Continuous Concept (CC) regularizes representation geometry and induces a spectral decomposition into low-frequency structure and high-frequency residual. We derive coding bounds for both views and show that multiplicity-corrected boundary attention becomes exact as the within-segment path energy vanishes. Empirically, BBC and CC exhibit strikingly different depth preferences: BBC works best in early layers, whereas CC is most effective in later layers. At 300M training tokens, combining BBC and CC improves NLL by 1.44\% on GPT-2 Medium. These gains persist with training runs up to 20B tokens. Representation diagnostics show that BBC increases the relative predictive importance of boundary states, while CC reduces spectral residual energy. Together, these results suggest that structure--residual biases can improve language-model pretraining beyond next-token prediction. Code: https://github.com/haihuang-ml/structure-residual

cs.CL↗

Trade-Adaptive Aggregation of Probabilistic Financial Forecasts

Financial NLP systems produce probabilistic forecasts from news, reports, and filings. Prediction markets can aggregate these forecasts sequentially, but their fees must reward information without overcharging low-risk updates. Existing quadratic-fee mechanisms use a state-blind bound, while a local-curvature envelope remains conservative because it prices every trade at the largest permitted span. We introduce SpanPM, a prediction-market mechanism that sets the local-curvature multiplier from each trade's realized payoff spread. Its fee dominates exact Bregman exposure trade by trade, preserves no arbitrage, information incorporation, expressiveness, and bounded worst-case loss, and yields a tighter overcharge factor approaching one as trade span vanishes. Repeated global best responses converge to a common belief and become full Newton steps locally, giving quadratic rather than damped-linear convergence. We implement a deterministic bounded one-dimensional multi-basin search, audited against a dense grid. Across paired synthetic experiments, SpanPM improves 20-round consensus error by several orders of magnitude over a fixed-envelope local baseline under the same hard cap. With evolving beliefs, it preserves 96--97\% of the trader surplus achieved with exact Bregman fees while cutting excess fees by 94\% relative to the global quadratic mechanism. These results establish a trade-adaptive prediction-market mechanism for sequential aggregation of probabilistic financial forecasts.

cs.CE↗