From the Fluency Fallacy to the Micro-to-Macro Validity Gap: Opportunities and Pitfalls of LLMs in Social Simulation
The integration of Large Language Models (LLMs) into social simulation has generated enthusiasm but also raises methodological and epistemological challenges. This critical review examines LLMs as cognitive or decision-making components of simulated agents from a computational social science perspective. We argue that their behavioral and epistemic limitations can become systemic vulnerabilities when scaled to multi-agent societies. We map the evolving landscape of LLM-driven platforms, from narrative sandboxes to population-scale simulations, and develop a framework for analyzing the Micro-to-Macro Validity Gap: the propagation, amplification, or contraction of micro-level limitations-including hallucinations, stochastic inconsistency, representational biases, and alignment effects-into agent-level distortions (e.g., persona convergence) and macro-level risks (e.g., artificial consensus and behavioral homogenization). We also highlight evaluative vulnerabilities such as the Fluency Fallacy and automation bias. We identify contexts where LLM-based agents offer operational value, including serious games, participatory environments, and exploratory modeling, distinguishing these uses from confirmatory research with unconstrained models and precise social forecasting. Finally, we examine theory-driven hybrid architectures embedding LLMs in explicit, mechanistic, and reproducible Agent-Based Modeling (ABM) frameworks. While promising, these architectures do not guarantee epistemic control: validity requires multi-level evaluation of environmental dynamics, individual behavior, cross-level interactions, and aggregate outcomes. They also remain vulnerable to physics washing.