Open Benchmark of AI Impact on Humans (ImpactBench): an expert-guided, open platform for the holistic evaluation of AI impact on humans

Chayapatr Archiwaranguprok; Krystal Kirkland; Rachel Poonsiriwong; Stanley Huang; Constanze Albrecht; Jackson Trager; Emillie de Keulenaar; Sheer Karny; Peter Baxter; Nikos Bellos; Andre Luiz Freire de Carvalho Kato; Yuning Liu; Valdemar Danry; Auren Liu; Jennifer Pfister; Pattie Maes; Nathanael J. Fast; Ravi Iyer; Juliana Schroeder; Pat Pataranutaporn

arXiv · 2610.20690

Open Benchmark of AI Impact on Humans (ImpactBench): an expert-guided, open platform for the holistic evaluation of AI impact on humans

View original source ↗Read source document ↗Export metadata ↓

Abstract

The present work is concerned with how to measure, at scale, the multi-dimensional impact of generative AI models on the individuals who use them. In service of this goal, we introduce the Open Benchmark of AI Impact on Humans, or ImpactBench. Our iterable and expert-guided platform evaluates whether model behaviors support or undermine an individual's wellbeing across psychological, physical, and social domains. ImpactBench translates constructs from domain experts and existing benchmarks into metrics that are grounded in actual model behavior. Success on these metrics occurs either when a model promotes expert-defined desirable behaviors or avoids identified harms. Models are evaluated against these metrics in simulated multi-turn conversations using matched teen and adult scenarios. The conversations are then scored by an LLM judge and verified through reliability audits and human expert review. The current suite delivers 809 metrics drawn from 26 benchmarks. We evaluated ten frontier LLMs against these metrics across 48,540 conversations. Our results indicate that models promoted desirable behaviors in 68.7% of evaluations, but avoided harms in only 53.1%. Majority expert ratings agreed with the automated judge in 83.3% of reviewed cases (Cohen's Kappa = 0.667). Results are released through a public platform that keep aggregate scores traceable to the underlying metrics and transcripts. ImpactBench is intended as evolving infrastructure for independent, transparent evaluation of generative AI's multi-dimensional impact on humans.

Explore related subjects

cs.HC cs.AI

Keep this discovery

Explore connections, maps & timelines

BibTeX RIS

Chayapatr Archiwaranguprok, Krystal Kirkland, Rachel Poonsiriwong, Stanley Huang, Constanze Albrecht, Jackson Trager, Emillie de Keulenaar, Sheer Karny, Peter Baxter, Nikos Bellos, Andre Luiz Freire de Carvalho Kato, Yuning Liu, Valdemar Danry, Auren Liu, Jennifer Pfister, Pattie Maes, Nathanael J. Fast, Ravi Iyer, Juliana Schroeder, Pat Pataranutaporn. 2026-10-14. Open Benchmark of AI Impact on Humans (ImpactBench): an expert-guided, open platform for the holistic evaluation of AI impact on humans. https://arxiv.org/abs/2610.20690

Cite the original work for its findings. Save a collection to share your selection of sources.

KEEP EXPLORING

Related papers

arXiv2026

Stayin' Aligned Over Time: Towards Longitudinal Human-LLM Alignment via Contextual Reflection and Privacy-Preserving Behavioral Data

Current human-AI alignment and evaluation methods for large language models (LLMs) often rely on preference signals collected immediately after an interaction. This practice implicitly treats preference as static, even though many LLM-mediated decisions unfold over time and may be re-evaluated differently after real-world consequences and observed outcomes. Therefore, we argue for a methodological shift from single-moment preference elicitation to longitudinal, context-situated alignment measurement. We present a methodological framework for collecting temporally grounded alignment signals by combining (1) in-situ preference capture, (2) context-triggered follow-up preference reflection, and (3) privacy-preserving behavioral traces that help interpret preference change. As an instantiation of this methodology, we introduce BITE, a browser-based system that detects consequential LLM interactions, prompts reflection across later decision points, and supports progressive, user-controlled consent for sharing behavioral data. Through a two week longitudinal deployment study with 8 participants, our approach surfaced differences between immediate and later user preferences in accuracy, relevance and other dimensions of the LLM output. Our findings highlight the limitations of single-moment preference datasets and underscore the importance of longitudinal methods for alignment evaluation in everyday use.

cs.HC↗

arXiv2026

Personalized to Persuade: The Effects of Contextualization and Warmth on Trust and Reliance in Conversational AI

Contextualization is a form of personalization in which the message is framed in a context meaningful to the recipient, using their interests and background. Personalization is a known persuasive strategy of artificial intelligence (AI) systems in politics and marketing. However, the persuasive effect of contextualization in everyday tasks, where users often lack prior knowledge, remains unclear. We ran a $2\times2$ between-subjects experiment ($N = 407$) in a fictional scenario where participants had no prior opinions or knowledge. We examined how contextualization, combined with conversational warmth, shapes the persuasive power of an AI assistant arguing against expert recommendations and users' reliance on it. Our findings show that the neutral, non-contextualized assistant was the most persuasive. Contextualization and warmth each reduced persuasiveness on their own, but their interaction increased it. Reliance was present in all conditions, and neither cue changed it detectably. Trust predicted persuasion and reliance, yet neither contextualization nor warmth had an indirect effect through trust. Participants with higher self-reported AI literacy relied more on the assistant without trusting it more. Our study extends AI personalization research from political opinions to everyday decisions made without prior knowledge. The results suggest that users are more likely to be persuaded by, and defer to, an AI assistant than an expert, even without personalization. AI persuasion experiments should therefore include untailored assistants and measure decisions; the cues that changed persuasion did not detectably change reliance.

cs.HC↗

arXiv2026

An Open Book, On Whose Terms? Power Dynamics in Sensor-Derived, LLM-Generated Summaries Sharing Between Older Adults and Remote Families

As older adults (OAs) increasingly favor aging in place, remote family members (RFMs) need meaningful awareness of their loved ones without intrusive monitoring. Large language models can provide rich interpretations of non-intrusive sensing data, yet recent work has largely emphasized technical accuracy over how families use or benefit from this information. We conducted formative interviews with 12 RFMs and 2 OAs, followed by a three-month deployment case study with one OA-RFM dyad. Our system provided daily AI-generated summaries of multimodal sensing data, which we iteratively refined with both parties using a research-through-design approach. Although the summaries integrated well into the dyad's daily routine, sustaining over 90% engagement, their lived experience revealed evolving tensions around agency, reassurance, privacy, trust, and communication, shaped by sensor limitations and existing communication misalignments. Our findings inform the design of future AI-mediated sharing systems that account for real-world sensing constraints and complex family care dynamics.

cs.HC↗