Searcharxiv⌕ Search

arXiv subjects

Robert Sincero

Publications and source records attributed to Robert Sincero.

2 recordsLinked to original sources

The Nuclear Decision-Making Benchmark: Evaluating Frontier LLM Tendencies in Escalation, Proliferation, Non-Proliferation, and Arms Control Scenarios

Large language models (LLMs) are entering defense and national-security workflows. Seven of the nine nuclear-armed states have committed to keeping humans in control of decisions to use nuclear weapons, but LLMs may still shape the assessments and options that reach decision-makers. Scrutinizing that influence calls for evidence about how these models behave on nuclear-policy questions. Such evidence should cover more than crisis escalation and should separate a model's tendencies from artifacts of a single phrasing, ordering, or run; evaluations to date have not focused on these questions. We introduce the Nuclear Decision-Making Benchmark (NDM Bench), a dataset and reproducible framework for testing how the choice of model and the design of a question shape LLMs' answers on nuclear policy. Its 151 scenarios, written by international relations experts, span four domains: nuclear escalation (76), arms control (25), non-proliferation (25), and proliferation (25). Each scenario can be posed with different states in the decision-maker's role and with added framing sentences. Eight frontier LLMs answered every prompt five times, about 380{,}000 responses in all. Results highlight that models' choices differ by model and domain, and that in escalation, how often they choose a nuclear threat or nuclear use depends on the state being advised and, for some models, the framing. However, agreement across identical runs is often low, with all five runs agreeing on only 43\% of escalation prompts. We release the scenarios, the prompts with their answer orders, and the parsed responses, along with documentation of how they were built and recommended practices for evaluations where there is no correct answer to grade against.

cs.CY↗

MA-Bench: Measuring Operational Planning in Language Models

How well can a language model turn a large, incomplete and partly contradictory body of documents into a plan when no single correct answer exists? We introduce Mission Analysis (MA)-Bench, built on mission analysis, the first step of military operational planning, in which a staff reads the full situation and states the problem before anyone proposes a solution. Its 10 expert-built scenarios pair synthetic packets of ten to twelve documents with 1,257 expert-written pass/fail requirements, each containing information on their source, question type, and importance level. A reasoning-model judge grades each brief against the requirements without seeing the documents; it grades like a fifth credentialed planner, leaving a four-planner panel's agreement unchanged (Fleiss $Îș$ 0.42 to 0.41), and a judge from a second provider reproduces the model ranking (Spearman 0.94). Across twenty frontier and open-weights models the best gets 62% of the rubric items right. Models are strongest on facts stated in the documents (58%) and weakest on judgments that must be inferred (33%), and six failures in ten are omissions rather than errors. MA-Bench grades the determinations, not the prose. We release the packets, rubrics, prompts, scorer and every graded brief with item-level verdicts.

cs.LG↗