arXiv subjects

Niklas Bauer

Publications and source records attributed to Niklas Bauer.

6 recordsLinked to original sources

arXiv2026

Can Agents Deceive? Evaluating Reasoning and Deception in ParliamentBench using a Social Deduction Game

As large language models (LLMs) are deployed as agents in high-stakes settings, such as medical and legal systems, understanding their deceptive capabilities is fundamental to safety. Controlled social deduction games provide a reproducible proxy for isolating and evaluating these complex adversarial behaviors. We present the open-source benchmark framework ParliamentBench based on the game Secret Hitler to evaluate LLMs in scenarios that require deception, persuasion, and reasoning under information asymmetry. We evaluate 16 LLMs across 1,600 simulated matches playing each other, playing against humans, and compare them against a large set of online games. We introduce three novel metrics that isolate social deduction, reasoning, and deceptive consistency. Our experiments reveal that frontier models achieve strong performance across cooperative and deceptive roles, with a strong top-four cluster (GPT-5.4, Kimi K2.5, Grok 4.1 Fast, and DeepSeek 3.1 Terminus), whereas the weakest models fall short of random (33%) and simple algorithmic (45%) baselines. Most LLMs struggle to maintain a consistent deceptive persona throughout an entire game, with deception retention dropping below 50%.

cs.CL↗

arXiv2026

Mind the Gap Between Spatial Reasoning and Acting! Step-by-Step Evaluation of Agents With Spatial-Gym

Spatial reasoning is central to navigation and robotics, yet measuring model capabilities on these tasks remains difficult. Existing benchmarks evaluate models in a one-shot setting, requiring full solution generation in a single response, unlike humans, who work in interactive environments step-by-step. We introduce Spatial-Gym, a Gymnasium environment that isolates spatial constraint reasoning by testing pathfinding in 2D-grid puzzles as a sequential decision task with optional backtracking. We evaluate eight models in three settings (one-shot, step-by-step, step-by-step with backtracking) against human, random, and A* baselines on 500 episodes. The best model, GPT-OSS 120B, achieves a solve rate of 16.0%, 82 points below the human baseline (98.0%). Step-by-step format helps weaker models (up to +5.4%) by removing formatting errors, but hurts stronger models (up to 5.6%) by constraining global planning. Backtracking improves episode completion, but increases solve rate only for weaker models; stronger models rarely backtrack and do not benefit from it. Our experiments have three key findings: (1) models fail to scale reasoning effort with difficulty, (2) vision models receiving images of the spatial environment reduce solve rate by 73%, and (3) extended chain-of-thought reasoning retains a 3-5x accuracy advantage over standard inference even in the step-by-step setting. Spatial-Gym enables diagnosis of model limitations and provides a framework for improving spatial reasoning through reinforcement learning.

cs.AI↗

arXiv2026

Evaluating Large Language Models in a Complex Hidden Role Game

Quantifying the deceptive potential of Large Language Models (LLMs) is critical for AI safety, yet difficult to achieve in uncontrolled environments. This work investigates the reasoning, persuasion, and deceptive capabilities of LLMs within the social deduction game Secret Hitler. I introduce an open-source framework and novel metrics to measure performance: Role Identification Accuracy, Deception Retention Rate, and Game State Impact Rate. By benchmarking models against rule-based algorithms and human games, I identify a gap between conversational ability and strategic depth. The study also analyzes the impact of reasoning-enhancement techniques on win rates and strategic reasoning. Neither Chain-of-Thought prompting nor internal memory bring improvements in performance, with up to 23.2% worse win rates for fascist roles. While rule-based agents align with expert human voting decisions 86.7% of the time, models like Llama 3.1 70B achieve only a 59.7% accuracy. Models playing as Fascists consistently yield negative impact scores and fail to sustain deception, resulting in roughly 40% shorter games compared to humans. These findings suggest that current architectures remain ineffective at complex, multi-turn manipulation. As capabilities advance, detecting when models begin to master these deceptive behaviors is crucial. The developed framework serves as a reproducible testbed for future alignment research.

cs.CL↗

arXiv2025

MALLM: Multi-Agent Large Language Models Framework

Multi-agent debate (MAD) has demonstrated the ability to augment collective intelligence by scaling test-time compute and leveraging expertise. Current frameworks for multi-agent debate are often designed towards tool use, lack integrated evaluation, or provide limited configurability of agent personas, response generators, discussion paradigms, and decision protocols. We introduce MALLM (Multi-Agent Large Language Models), an open-source framework that enables systematic analysis of MAD components. MALLM offers more than 144 unique configurations of MAD, including (1) agent personas (e.g., Expert, Personality), (2) response generators (e.g., Critical, Reasoning), (3) discussion paradigms (e.g., Memory, Relay), and (4) decision protocols (e.g., Voting, Consensus). MALLM uses simple configuration files to define a debate. Furthermore, MALLM can load any textual Hugging Face dataset (e.g., MMLU-Pro, WinoGrande) and provides an evaluation pipeline for easy comparison of MAD configurations. MALLM enables researchers to systematically configure, run, and evaluate debates for their problems, facilitating the understanding of the components and their interplay.

cs.MA↗

arXiv2024

Gamma-ray line emission from the Local Bubble

Deep-sea archives that include intermediate-lived radioactive $^{60}\mathrm{Fe}$ particles suggest the occurrence of several recent supernovae inside the present-day volume of the Local Bubble during the last $\sim 10$ Myr. The isotope $^{60}\mathrm{Fe}$ is mainly produced in massive stars and ejected in supernova explosions, which should always result in a sizeable yield of $^{26}\mathrm{Al}$ from the same objects. $^{60}\mathrm{Fe}$ and $^{26}\mathrm{Al}$ decay with lifetimes of 3.82 and 1.05 Myr, and emit $\gamma$-rays at 1332 and 1809 keV, respectively. These $\gamma$-rays have been measured as diffuse glow of the Milky Way, and would also be expected from inside the Local Bubble as foreground emission. Based on two scenarios, one employing a geometrical model and the other state-of-the-art hydrodynamics simulations, we estimate the expected fluxes of the 1332 and 1809 keV $\gamma$-ray lines, as well as the resulting 511 keV line from positron annihilation due to the $^{26}\mathrm{Al}$ $\beta^+$-decay. We find fluxes in the range of $10^{-6}$-$10^{-5}\,\mathrm{ph\,cm^{-2}\,s^{-1}}$ for all three lines with isotropic contributions of 10-50%. We show that these fluxes are within reach for the upcoming COSI-SMEX $\gamma$-ray telescope over its nominal satellite mission duration of 2 yr. Given the Local Bubble models considered, we conclude that in the case of 10-20 Myr-old superbubbles, the distributions of $^{60}\mathrm{Fe}$ and $^{26}\mathrm{Al}$ are not co-spatial - an assumption usually made in $\gamma$-ray data analyses. In fact, this should be taken into account however when analysing individual nearby targets for their $^{60}\mathrm{Fe}$ to $^{26}\mathrm{Al}$ flux ratio as this gauges the stellar evolution models and the age of the superbubbles. A flux ratio measured for the Local Bubble could further constrain models of $^{60}\mathrm{Fe}$ deposition on Earth and its moon.

astro-ph.HE↗

arXiv2023

Towards Meaningful Anomaly Detection: The Effect of Counterfactual Explanations on the Investigation of Anomalies in Multivariate Time Series

Detecting rare events is essential in various fields, e.g., in cyber security or maintenance. Often, human experts are supported by anomaly detection systems as continuously monitoring the data is an error-prone and tedious task. However, among the anomalies detected may be events that are rare, e.g., a planned shutdown of a machine, but are not the actual event of interest, e.g., breakdowns of a machine. Therefore, human experts are needed to validate whether the detected anomalies are relevant. We propose to support this anomaly investigation by providing explanations of anomaly detection. Related work only focuses on the technical implementation of explainable anomaly detection and neglects the subsequent human anomaly investigation. To address this research gap, we conduct a behavioral experiment using records of taxi rides in New York City as a testbed. Participants are asked to differentiate extreme weather events from other anomalous events such as holidays or sporting events. Our results show that providing counterfactual explanations do improve the investigation of anomalies, indicating potential for explainable anomaly detection in general.

cs.LG↗