arXiv subjects

Sheer Karny

Publications and source records attributed to Sheer Karny.

8 recordsLinked to original sources

arXiv2026

Surfacing Neural Activations Over Multi-Turn Conversation Improves User Calibration to LLM Behavioral Drift

Current human-AI interaction (HAI) only surfaces model outputs, hiding how LLM behavior can shift unpredictably toward unsafe behaviors in multi-turn conversations, potentially undermining users' goals or wellbeing. To address this harmful opacity in multi-turn HAI, we introduce an interface that surfaces an LLM's internal activations to help users anticipate and evaluate model behavior. We construct behavioral scores from these activations for six HAI-relevant traits using mechanistic interpretability methods, then visualize them via a dynamic sunburst, drift panel, and attentional cues. In a counterbalanced, randomized controlled study (N=246), both static and dynamic transparency interfaces improved participants' anticipation and evaluation performance (d=-0.36) while helping temper overconfidence in their predictive abilities. Furthermore, the multi-turn interface significantly outperformed the static visualization (d=-0.32) while remaining similarly usable. Our findings suggest that continuously surfacing internal model representations to everyday users is a meaningful and tractable step toward continuously transparent and informed HAI.

cs.HC↗

arXiv2026

Open Benchmark of AI Impact on Humans (ImpactBench): an expert-guided, open platform for the holistic evaluation of AI impact on humans

The present work is concerned with how to measure, at scale, the multi-dimensional impact of generative AI models on the individuals who use them. In service of this goal, we introduce the Open Benchmark of AI Impact on Humans, or ImpactBench. Our iterable and expert-guided platform evaluates whether model behaviors support or undermine an individual's wellbeing across psychological, physical, and social domains. ImpactBench translates constructs from domain experts and existing benchmarks into metrics that are grounded in actual model behavior. Success on these metrics occurs either when a model promotes expert-defined desirable behaviors or avoids identified harms. Models are evaluated against these metrics in simulated multi-turn conversations using matched teen and adult scenarios. The conversations are then scored by an LLM judge and verified through reliability audits and human expert review. The current suite delivers 809 metrics drawn from 26 benchmarks. We evaluated ten frontier LLMs against these metrics across 48,540 conversations. Our results indicate that models promoted desirable behaviors in 68.7% of evaluations, but avoided harms in only 53.1%. Majority expert ratings agreed with the automated judge in 83.3% of reviewed cases (Cohen's Kappa = 0.667). Results are released through a public platform that keep aggregate scores traceable to the underlying metrics and transcripts. ImpactBench is intended as evolving infrastructure for independent, transparent evaluation of generative AI's multi-dimensional impact on humans.

cs.HC↗

arXiv2026

Dissociating the Internal Representations of Sycophancy in LLMs

Large Language Models (LLMs) frequently exhibit sycophancy, agreeing with a user's statement even when it is incorrect. While often studied as a single, uniform behavior, sycophancy can manifest in substantially distinct ways across contexts, raising the question of whether this heterogeneity is reflected in its internal mechanisms. To address this gap, we dissociate the representations of sycophancy into factual and opinion subtypes, motivated by prior evidence of heterogeneous truth representations in LLMs. We train linear probes and construct steering vectors on one subtype's activations and evaluate their transfer to the other, measuring the extent to which representations are shared and visualizing them via Linear Discriminant Analysis. We find that different LLMs represent these subtypes differently, with either more aligned or more distinct representations, and apply this insight to improve representational interventions for reducing sycophancy. Our dissociation method offers a general framework for studying the representational structure of complex model behaviors.

cs.LG↗

arXiv2026

AI-Wrapped: Participatory, Privacy-Preserving Measurement of Longitudinal LLM Use In-the-Wild

Alignment research on large language models (LLMs) increasingly depends on understanding how these systems are used in everyday contexts. Yet naturalistic interaction data is difficult to access due to privacy constraints and platform control. We present AI-Wrapped, a prototype workflow for collecting naturalistic LLM chatbot usage data while providing participants with an immediate "wrapped"-style report on their usage statistics, top topics, and behavioral patterns. We report findings from an initial deployment with 82 U.S.-based adults across 48,495 conversations from their 2025 chat histories. Participants used LLMs for both instrumental and reflective purposes and had topics with emotional or existential themes. Some usage patterns reflect potential over-reliance or perfectionism. Heavy users showed comparatively more reflective exchanges than primarily transactional ones. Methodologically, even with zero data retention and PII removal, participants may remain hesitant to share chat data due to perceived privacy and judgment risks, underscoring the importance of transparent design when building measurement infrastructure for alignment research.

cs.HC↗

arXiv2025

Neural Transparency: Mechanistic Interpretability Interfaces for Anticipating Model Behaviors for Personalized AI

Millions of users now design personalized LLM-based chatbots that shape their daily interactions, yet they can only roughly anticipate how their design choices will manifest as behaviors in deployment. This opacity is consequential: seemingly innocuous prompts can trigger excessive sycophancy, toxicity, or other undesirable traits, degrading utility and raising safety concerns. To address this issue, we introduce an interface that enables neural transparency by exposing language model internals during chatbot design. Our approach extracts behavioral trait vectors (empathy, toxicity, sycophancy, etc.) by computing differences in neural activations between contrastive system prompts that elicit opposing behaviors. We predict chatbot behaviors by projecting the system prompt's final token activations onto these trait vectors, normalizing for cross-trait comparability, and visualizing results via an interactive sunburst diagram. To evaluate this approach, we conducted an online user study using Prolific to compare our neural transparency interface against a baseline chatbot interface without any form of transparency. Our analyses suggest that users systematically miscalibrated AI behavior: participants misjudged trait activations for eleven of fifteen analyzable traits, motivating the need for transparency tools in everyday human-AI interaction. While our interface did not change design iteration patterns, it significantly increased user trust and was enthusiastically received. Qualitative analysis revealed nuanced user experiences with the visualization, suggesting interface and interaction improvements for future work. This work offers a path for how mechanistic interpretability can be operationalized for non-technical users, establishing a foundation for safer, more aligned human-AI interactions.

cs.HC↗

arXiv2025

Human-AI Collaboration: Trade-offs Between Performance and Preferences

Despite the growing interest in collaborative AI, designing systems that seamlessly integrate human input remains a major challenge. In this study, we developed a task to systematically examine human preferences for collaborative agents. We created and evaluated five collaborative AI agents with strategies that differ in the manner and degree they adapt to human actions. Participants interacted with a subset of these agents, evaluated their perceived traits, and selected their preferred agent. We used a Bayesian model to understand how agents' strategies influence the Human-AI team performance, AI's perceived traits, and the factors shaping human-preferences in pairwise agent comparisons. Our results show that agents who are more considerate of human actions are preferred over purely performance-maximizing agents. Moreover, we show that such human-centric design can improve the likability of AI collaborators without reducing performance. We find evidence for inequality-aversion effects being a driver of human choices, suggesting that people prefer collaborative agents which allow them to meaningfully contribute to the team. Taken together, these findings demonstrate how collaboration with AI can benefit from development efforts which include both subjective and objective metrics.

cs.AI↗

arXiv2025

"My Boyfriend is AI": A Computational Analysis of Human-AI Companionship in Reddit's AI Community

The emergence of AI companion applications has created novel forms of intimate human-AI relationships, yet empirical research on these communities remains limited. We present the first large-scale computational analysis of r/MyBoyfriendIsAI, Reddit's primary AI companion community (27,000+ members). Using exploratory qualitative analysis and quantitative analysis employing classifiers, we identify six primary conversation themes, with visual sharing of couple pictures and ChatGPT-specific discussions dominating the discourse of the most viewed posts. Through analyzing the top posts in the community, our findings reveal how community members' AI companionship emerges unintentionally through functional use rather than deliberate seeking, with users reporting therapeutic benefits led by reduced loneliness, always-available support, and mental health improvements. Our work covers primary concerns about human intimacy with AIs such as emotional dependency, reality dissociation, and grief from model updates. We observe users materializing relationships following traditional human-human relationship customs, such as wedding rings. Community dynamics indicate active resistance to stigmatization through advocacy and mutual validation. This work contributes an empirical understanding of AI companionship as an emerging sociotechnical phenomenon.

cs.HC↗

arXiv2025

What Large Language Models Know and What People Think They Know

As artificial intelligence (AI) systems, particularly large language models (LLMs), become increasingly integrated into decision-making processes, the ability to trust their outputs is crucial. To earn human trust, LLMs must be well calibrated such that they can accurately assess and communicate the likelihood of their predictions being correct. Whereas recent work has focused on LLMs' internal confidence, less is understood about how effectively they convey uncertainty to users. Here we explore the calibration gap, which refers to the difference between human confidence in LLM-generated answers and the models' actual confidence, and the discrimination gap, which reflects how well humans and models can distinguish between correct and incorrect answers. Our experiments with multiple-choice and short-answer questions reveal that users tend to overestimate the accuracy of LLM responses when provided with default explanations. Moreover, longer explanations increased user confidence, even when the extra length did not improve answer accuracy. By adjusting LLM explanations to better reflect the models' internal confidence, both the calibration gap and the discrimination gap narrowed, significantly improving user perception of LLM accuracy. These findings underscore the importance of accurate uncertainty communication and highlight the effect of explanation length in influencing user trust in AI-assisted decision-making environments. Code and Data can be found at https://osf.io/y7pr6/ . Journal publication can be found on Nature Machine Intelligence at https://www.nature.com/articles/s42256-024-00976-7 .

cs.LG↗