What FLARE's Published Numbers Actually Measure

Dung Cao; Palaash Gang; Arjun Veluri

Home/Archive/arXiv

arXiv · 2610.33604

What FLARE's Published Numbers Actually Measure

Dung Cao·Palaash Gang·Arjun Veluri

View original source ↗Read source document ↗Export metadata ↓

Abstract

We audit the scoring pipeline of FLARE, the financial task suite behind PIXIU and FinBen, and find that in a decisive fraction of cells the score is a property of the answer-extraction rule, not the model. The clearest evidence is already published: of the 34 released Headlines scores, nine equal to four decimals what the benchmark's own code returns when no output is byte-equal to Yes, and only four clear the per-stratum majority floor. FLARE credits the first entry of the dataset card's choices list found anywhere in a generation; the lm-evaluation-harness it forks takes the first match by position; on identical generations the two differ by up to 0.4216 accuracy, and no leaderboard states which rule produced its numbers. An unspecified rule leaves the score partially identified, and because the coarsening is multiplicity rather than missingness the interval's width is counted rather than bounded. Across a 204-cell factorial and a 46-cell diagnostic, 10 cells are certified invariant to the rule, 4 only because almost nothing parses, so certification is paired with coverage. On Financial PhraseBank four of seven models fall below the 0.5948 majority floor under one shipped rule and clear it under the other; on three of four substring-scored tasks no open-weight model we test is distinguishable from a degenerate predictor. Reading a FLARE score needs three quantities it does not report -- ambiguity rate, coverage, floor -- and floorbench computes all three from generations already produced.

Explore related subjects

q-fin.GN

Keep this discovery

Explore connections, maps & timelines

BibTeX RIS

Dung Cao, Palaash Gang, Arjun Veluri. 2026-10-24. What FLARE's Published Numbers Actually Measure. https://arxiv.org/abs/2610.33604

Cite the original work for its findings. Save a collection to share your selection of sources.

KEEP EXPLORING

Related papers

arXiv2026

Uncertainty Quantification for Financial Foundation Models: A Survey

Financial foundation models increasingly support workflows spanning financial question answering, disclosure analysis, forecasting, and decision-making. In these settings, uncertainty may concern whether evidence is valid and available at the relevant time, whether an interpretation is justified, which outcomes remain plausible, or which action is preferred under costs and risk constraints. Because these objects arise at different points in a workflow, uncertainty cannot be reduced to confidence in a final output. We introduce a source-aware framework that distinguishes six sources of uncertainty across evidence and context, model and inference, and outcome and action, while treating agentic workflows as the execution layer through which uncertainty can propagate, change form, or be reduced. We develop a mechanism-based taxonomy grounded in how methods construct or verify uncertainty and reliability representations, and use it to assess their applicability across financial task domains. We further map finance-specific empirical studies across the taxonomy and task domains to characterize the current research landscape. Our analysis shows that most reviewed studies address uncertainty in individual parts of financial workflows, with limited work connecting evidence and reasoning reliability to downstream prediction and constrained action. By connecting where uncertainty arises, how it is represented, and how it bears on downstream decisions, this survey provides a unified foundation for studying and evaluating uncertainty in financial foundation models.

q-fin.GN↗

arXiv2026

Tempting the Agent: The Economics of Reputation without Persistent Identity in AI Agent Markets

Reputation is a fundamental mechanism through which markets sustain trust when service quality cannot be perfectly assessed ex ante, constituting a form of intertemporal economic capital by attracting future demand. Its effectiveness as a disciplinary mechanism depends not only on past interactions but also on the persistence of the identity to which reputation is attached. When identities can be abandoned and recreated cheaply, reputational capital may itself become an object of opportunistic exploitation. This paper develops a dynamic economic framework to study when reputation is sufficient to discipline autonomous agents. We model reputation as capital attracting future economic activity. At each point, an agent chooses between operating honestly, investing in quality to preserve future gains, or executing a one-shot deviation to extract its reputation's value and restart from a penalized identity. Our analysis relates the temptation to opportunistic behavior to identity-reset costs, reputation persistence, demand sensitivity, and enforcement design, deriving comparative statics on optimal quality provision. Autonomous AI-agent operating on the blockchain are a relevant application: infrastructures such as ERC-8004, ERC-8183, and x402 combine reputation, identity, and payments in permissionless markets. Nonetheless, our framework applies to any environment where reputation generates future business and identities are replaceable.

q-fin.GN↗

arXiv2026

Amortized Neural Filtering for Calibrated Posteriors in Statistical Arbitrage

Statistical arbitrage trades on an estimate of the state of a residual spread, and conventional practice discards the uncertainty of that estimate. We keep the uncertainty and use it in the trading decision. A Neural Filter Estimator (NFE) is trained on simulated paths of a mean-reverting process whose innovation law is selected by testing candidates against real residuals; once trained, it returns the posterior over the latent state of every spread in a cross-section in one forward pass. Three results follow. On simulated paths with known truth, every practical estimator, ours included, falls in one narrow band of accuracy well below an estimator given full information, so the posterior gains no accuracy over a point estimate; its contribution is a correct account of its own uncertainty. On Korean equities, net of costs, the conventional point estimate forecasts about a thousand basis points per holding period more than it realizes, and the amortized methods cut that gap by a factor of roughly two to eight. Sizing positions from the posterior on the same learned factors as the end-to-end Attention Factor Model matches the Sharpe ratio of those factors under the base rule within the period's noise, at no more risk.

q-fin.GN↗