arXiv subjects

Ryan Stevens

Publications and source records attributed to Ryan Stevens.

3 recordsLinked to original sources

arXiv2026

APEX-Accounting

We introduce APEX-Accounting, a benchmark built by Mercor in partnership with Ramp, to assess whether frontier models can do the real work of accountants. Tasks include reconciling accounts, accruing expenses, posting transactions, and producing reports. The private eval set comprises 160 tasks, split across 10 worlds. Each world contains an accounting system, as well as spreadsheets, PDFs, and other files. Every task was authored and solved by experts in accounting and bookkeeping, who also wrote grading rubrics. Across nine frontier models, Claude-Fable-5 (Max) leads with 56.4% Mean Criteria@3, ahead of Muse-Spark-1.1 (xHigh) at 52.6%. No model scores more than 2.6% Pass^8 (GPT-5.6-Sol (Max+Pro)) and the highest Pass@8 is 21.5% (Muse-Spark-1.1 (xHigh)). We experiment with increasing the token budget from $1 to $50 and observe an instance of Simpson's paradox: scores increase as the token budget increases but within a given budget-constrained harness, scores are lower on tasks where the model spends more tokens. As APEX-Accounting is a closed benchmark, leaderboard evals can be run for any frontier model on request.

cs.CL↗

arXiv2026

Payrolls to Prompts: Firm-Level Evidence on the Substitution of Labor for AI

Generative AI has the potential to transform how firms produce output. Yet, credible evidence on how AI is actually substituting for human labor remains limited. In this paper, we study firm-level substitution between contracted online labor and generative AI using payments data from a large U.S. expense management platform. We track quarterly spending from Q3 2021 to Q3 2025 on online labor marketplaces (such as Upwork and Fiverr) and leading AI model providers. To identify causal effects, we exploit the October 2022 release of ChatGPT as a common adoption shock and estimate a difference-in-differences model. We provide a novel measure of exposure based on the share of spending at online labor marketplaces prior to the shock. Firms with greater exposure to online labor adopt AI earlier and more intensively following the shock, while simultaneously reducing spending on contracted labor. By Q3 2025, firms in the highest exposure quartile increase their share of spending on AI model providers by 0.8 percentage points relative to the lowest exposure quartile, alongside significant declines in labor marketplace spending. Combining these responses yields a direct estimate of substitution: among the most exposed firms, a \$1 decline in online labor spending is associated with approximately \$0.03 of additional AI spending, implying order-of-magnitude cost savings from replacing outsourced tasks with AI services. These effects are heterogeneous across firms and emerge gradually over time. Taken together, our results provide the first direct, micro-level evidence that generative AI is being used as a partial substitute for human labor in production.

econ.GN↗

arXiv2024

Facade: High-Precision Insider Threat Detection Using Deep Contextual Anomaly Detection

Insiders with privileged access have the power to cause great harm to their organization. Even a single insider threat incident can be catastrophic, resulting in both financial losses and reputation damage. These threats are some of the most difficult to detect, as attack activity is interspersed in large volumes of legitimate activity. Although it is a serious threat, the literature is sparse aside from a few studies with various limitations, preventing their practical deployment in large-scale organizations. We present Facade: fast and accurate contextual anomaly detection, a high-precision, deep-learning system that has served as the last line of defense against insider threats at Google since 2018. Facade is an innovative self-supervised system that detects suspicious events by considering the context surrounding each event, including relevant facts about the user and resource involved. It is built around a new multi-modal model that is trained on corporate document access, SQL table access, and HTTP/RPC request logs. To overcome the scarcity of labeled incident data, Facade employs a novel contrastive learning strategy trained exclusively on benign activity. Facade detects insider attackers with an extremely low false-positive rate, lower than 0.01%. For single rogue events, such as the illegitimate access to a sensitive document, the false-positive rate is as low as 0.0003%. To the best of our knowledge, Facade is the only proposed insider risk anomaly detection system with a false positive rate low enough for use in large corporate environments.

cs.CR↗