The Nuclear Decision-Making Benchmark: Evaluating Frontier LLM Tendencies in Escalation, Proliferation, Non-Proliferation, and Arms Control Scenarios
Large language models (LLMs) are entering defense and national-security workflows. Seven of the nine nuclear-armed states have committed to keeping humans in control of decisions to use nuclear weapons, but LLMs may still shape the assessments and options that reach decision-makers. Scrutinizing that influence calls for evidence about how these models behave on nuclear-policy questions. Such evidence should cover more than crisis escalation and should separate a model's tendencies from artifacts of a single phrasing, ordering, or run; evaluations to date have not focused on these questions. We introduce the Nuclear Decision-Making Benchmark (NDM Bench), a dataset and reproducible framework for testing how the choice of model and the design of a question shape LLMs' answers on nuclear policy. Its 151 scenarios, written by international relations experts, span four domains: nuclear escalation (76), arms control (25), non-proliferation (25), and proliferation (25). Each scenario can be posed with different states in the decision-maker's role and with added framing sentences. Eight frontier LLMs answered every prompt five times, about 380{,}000 responses in all. Results highlight that models' choices differ by model and domain, and that in escalation, how often they choose a nuclear threat or nuclear use depends on the state being advised and, for some models, the framing. However, agreement across identical runs is often low, with all five runs agreeing on only 43\% of escalation prompts. We release the scenarios, the prompts with their answer orders, and the parsed responses, along with documentation of how they were built and recommended practices for evaluations where there is no correct answer to grade against.