Searcharxiv⌕ Search

arXiv subjects

Aaron Belbasis

Publications and source records attributed to Aaron Belbasis.

3 recordsLinked to original sources

Imperfectly Imperfect: Evaluating Graded Cognitive Diversity in LLM Persona Simulations

Large language model (LLM) agents are increasingly used as synthetic participants in social, behavioural, and decision-making research, yet it remains unclear whether they can express graded cognitive diversity as assigned when engaging in simulated behaviour. We introduce a cognition-suite evaluation that tests whether simulated populations reproduce human variation in reasoning, self-reported traits, their associations, and downstream behaviour. Agents conditioned on synthetic or human-derived profiles of cognitive reflection, Need for Cognition, and Big Five personality answer the same suite as a cohort of 473 adults: two newly assembled tests, a Cognitive Reflection Test (CRT) and a Behavioural Reflection Test (BRT) whose decision scenarios measure reflection in behaviour, alongside an established CRT and the two trait questionnaires. Across seven full-battery simulation runs of four models, agents recover Need for Cognition and Big Five trait distributions on self-report questionnaires, but fall short on simulating diversity of cognitive task performance and downstream behaviours. In the four Claude Sonnet 4.6 runs, every agent scores full marks on both CRTs, whatever reflection level its persona was assigned. In the other three runs agents make intuitive-lure errors (the tempting wrong answers CRT items invite), but 49 to 85% score full marks, against under 10% in the human sample. Claude Sonnet 5 comes closest to people's item-by-item pattern of lures; the other two runs give fewer lures and vary less between items. In the BRT scenarios, every run over-produces 8 to 11 of 15 coded behaviours by at least 20 percentage points, and people's links between reflection and behaviour carry over only in part. Useful synthetic populations must preserve not only trait descriptors but the structured variation in reasoning, error type, and behaviour those descriptors are meant to organise.

cs.CL↗

The Behavioural Reflection Test: A time-efficient measure of reflective reasoning in morally and epistemically charged decisions

How readily people override intuitive conclusions through reflection shapes how they navigate dense information environments with reliable and misleading sources; yet the effectiveness of a prominent measure, the Cognitive Reflection Test (CRT), is eroded by widespread exposure to classic items and leaves open how such tendencies manifest more generally in decision style and linguistic expression. The Behavioural Reflection Test (BRT) addresses these issues with a brief open-ended measure of reasoning in morally and epistemically charged scenarios, alongside a four-item bespoke CRT (bCRT) as a low-exposure anchor. Among 473 online adults, higher bCRT predicted more evidence-sensitive, ethically driven decisions and reliance on high-quality sources, marked by more emotionally engaged, risk-attentive, economical language; associations the familiarity-adjusted CRT did not recover. The bCRT showed convergent validity, added item information above mean ability. Though open-ended, the BRT remained a time-efficient (median 11.8 minutes) behavioural assay of reflection with scope to extend across domains.

cs.HC↗

Illusions of reflection: open-ended task reveals systematic failures in Large Language Models' reflective reasoning

Humans do not just find mistakes after the fact -- we often catch them mid-stream because 'reflection' is tied to the goal and its constraints. Today's large language models produce reasoning tokens and 'reflective' text, but is it functionally equivalent with human reflective reasoning? Prior work on closed-ended tasks -- with clear, external 'correctness' signals -- can make 'reflection' look effective while masking limits in self-correction. We therefore test eight frontier models on a simple, real-world task that is open-ended yet rule-constrained, with auditable success criteria: to produce valid scientific test items, then revise after considering their own critique. First-pass performance is poor (often zero valid items out of 4 required; mean $\approx$ 1), and reflection yields only modest gains (also $\approx$ 1). Crucially, the second attempt frequently repeats the same violation of constraint, indicating 'corrective gains' arise largely from chance production of a valid item rather than error detection and principled, constraint-sensitive repair. Performance before and after reflection deteriorates as open-endedness increases, and models marketed for 'reasoning' show no advantage. Our results suggest that current LLM 'reflection' lacks functional evidence of the active, goal-driven monitoring that helps humans respect constraints even on a first pass. Until such mechanisms are instantiated in the model itself, reliable performance requires external structure that enforces constraints. Our code is available at: https://github.com/cruiseresearchgroup/LLM_ReflectionTest

cs.AI↗