Searcharxiv⌕ Search
Home/Archive/arXiv

arXiv · 2610.00885

FORALL-LEAN-AGENT for Auditable Reasoning in Formal Mathematics and Software Verification

Naing Oo Lwin
View original source ↗Read source document ↗Export metadata ↓

Abstract

Coding agents increasingly automate Lean proof development, but successful compilation alone does not establish that a candidate proves the intended statement under acceptable assumptions. We present FORALL-LEAN-AGENT, a frontend-agnostic framework for auditable reasoning in formal mathematics and software verification. The framework combines isolated workspaces, Lean tools, and fresh review with statement comparison, axiom audits, and independent proof checking where supported. Verification evidence and reviewer decisions are bound to the same candidate artifact, making acceptance traceable. We evaluate the framework on VeriSoftBench, PutnamBench, and both problems in the Lean Eval softwareverification track. On the 100-task VeriSoftBench subset, integration with FORALLLEAN-AGENT raises benchmark-rule success from 93 to 100 for GPT-5.6 Sol at low effort while reducing cost from $69 to $62. The PutnamBench evaluation accepts all 672 problems at an average of $4.72 each. These results show that agent harness design can improve correctness and efficiency while providing evidence beyond aggregate solve counts.

Explore related subjects

cs.SEcs.AIcs.LO

Keep this discovery

Explore connections, maps & timelines

BibTeXRIS

Naing Oo Lwin. 2026-10-01. FORALL-LEAN-AGENT for Auditable Reasoning in Formal Mathematics and Software Verification. https://arxiv.org/abs/2610.00885

Cite the original work for its findings. Save a collection to share your selection of sources.

KEEP EXPLORING

Related papers

arXiv2026

Are AI Coders Snitches? An Empirical Study of Pretraining Data Detection on Code Large Language Models

Recent advances in code large language models (CodeLLMs) have made them indispensable tools in modern software engineering. However, these models occasionally produce outputs that contain proprietary or sensitive code snippets, raising concerns about potential non-compliant use of training data, and posing risks to privacy and intellectual property. To ensure responsible and compliant deployment of CodeLLMs, training data detection (TDD) has become a critical task. While recent TDD methods have shown promise in natural language settings, their effectiveness on code data remains largely underexplored. This gap is particularly important given code's structured syntax and distinct similarity criteria compared to natural language. To address this, we conduct a comprehensive empirical study of seven state-of-the-art TDD methods on source code data, evaluating their performance across eight CodeLLMs. To support this evaluation, we introduce CodeSnitch, a function-level benchmark dataset comprising 9,000 code samples in three programming languages, each explicitly labeled as either included or excluded from CodeLLM training. Beyond evaluation on the original CodeSnitch, we design targeted mutation strategies to test the robustness of TDD methods under three distinct settings. These mutation strategies are grounded in the well-established Type-1 to Type-4 code clone detection taxonomy. Our study provides a systematic assessment of current TDD techniques for code and offers insights to guide the development of more effective and robust detection methods in the future.

cs.SE↗
arXiv2026

On the Illusion of Success: An Empirical Study of Job Reruns and Silent Failures in Industrial CI

Reliability of build outcomes is a cornerstone of effective Continuous Integration (CI). Yet in practice, developers often struggle with non-deterministic issues in the code or CI infrastructure, which undermine trust in build results. When faced with such unexpected outcomes, developers often repeatedly rerun jobs hoping for true success, but this practice is known to increase CI costs and reduce productivity. While recent studies have focused on intermittent job failures, no prior work has investigated silent failures, where build jobs are marked as successful but fail to complete all or part of their tasks. Such silent failures often go unnoticed, creating an illusion of success with detrimental consequences such as bugs escaping into production. This paper presents the first empirical study of silent failures through the practice of rerunning successful jobs. An analysis of 142,387 jobs across 81 industrial projects shows that 11% of successful jobs are rerun, with 35% of these reruns occurring after more than 24 hours. Using mixed-effects models on 32 independent variables (AUC of 85%), we identified key factors associated with reruns of successful jobs, notably testing and static analysis tasks, scripting languages like Shell, and developers prior rerun tendencies. A further analysis of 92 public issues revealed 11 categories of silent failures aligning with these factors, the most frequent being artifact operation errors, caching errors, and ignored exit codes. Overall, our findings provide valuable insights into the circumstances and causes of silent failures to raise awareness among teams, and present solutions to improve CI reliability.

cs.SE↗
arXiv2026

Incentives and Outcomes in Bug Bounties

Bug bounty programs have contributed significantly to security in technology firms in the last decade, but little is known about the role of reward incentives in producing useful outcomes. We analyze incentives and outcomes in Google's Vulnerability Rewards Program (VRP), one of the world's largest bug bounty programs. We analyze the responsiveness of the quality and quantity of bugs received to changes in payments, focusing on a change in Google's reward amounts posted in July, 2024, in which reward amounts increased by up to 200% for the highest impact tier. Our empirical results show an increase in the volume of high-value bugs received after the reward increase, as well as a high positive observed elasticity of labor supply for such bugs. We further break down the sources of this increase between veteran researchers and new researchers, showing that the reward increase both redirected the attention of veteran researchers and attracted new top security researchers into the program.

cs.SE↗