Searcharxiv⌕ Search

arXiv · 2611.21558

Benefits and Barriers to Human-AI Teaming for Crisis Counseling

Abstract

Background: AI systems are increasingly deployed to support mental health clinicians, yet real-world evidence remains limited. In crisis counseling, where clients often seek human connection, understanding whether AI augmentation preserves or undermines meaningful human engagement is essential. Methods: In a mixed-methods study of a counselor-facing AI copilot on a WhatsApp-based crisis hotline in India (Mar.-Oct. 2025), counselors could request, edit, and send AI messages. We analyzed 20 interviews, 8 shadowing sessions, 56 survey responses, and edits in 27,778 copilot messages from 108 counselors via thematic analysis (Technology Acceptance Model 2, plus inductive coding of emergent factors), and survey statistics. Results: Counselors reported high copilot acceptance (survey): 98.2% found it helpful; 89.3% likely to continue use; 80.4% reported efficiency gains. Interviews highlighted sentence scaffolding, empathetic phrasing, stress reduction, catching missed details, and efficiency. Barriers (survey) included repetitive outputs (23.2%), lack of context (17.9%), and authenticity concerns (10.7%). While 69.6% (survey) reported skill enhancement, interviews surfaced skill atrophy concerns. Counselors frequently accepted copilot suggestions, editing to remove questions, adjust tone, or personalize style. Conclusion: In one of the largest real-world deployments of a counselor-facing mental health AI copilot, counselors reported reduced cognitive load, empathetic communication, and greater efficiency, alongside concerns about skill atrophy, authenticity, and over-reliance. In settings like crisis counseling where clients seek human connection, AI tools must balance operational support with preserving clinical skill and genuine therapeutic engagement. Designs soliciting approval or editing of complete drafts may overweight efficiency at the expense of sustained human engagement.

Explore related subjects

Keep this discovery

Explore connections, maps & timelines

BibTeXRIS

Ivy Pham, Akshay Swaminathan, Sebastian Garcia, Jaimie Lim, Gloria Ye, Divyanjali Verma, Alan Pham, Chastin Chung, Shreya Shah, Shaked Azzam, Sharang Phadke, Ivan Lopez, William Wang, Angelina Wang, Nigam H. Shah. 2026-11-16. Benefits and Barriers to Human-AI Teaming for Crisis Counseling. https://searcharxiv.com/abs/2611.21558

Citations retain the original authors and DOI and include this site’s record URL. If this discovery service contributed to your research, you can also cite Searcharxiv by Indiciae: BibTeX · RIS.

KEEP EXPLORING

Related papers

Human-AI Writing as Distributed Cognition: Reassessing Collaborative Writing Theories in the Age of LLMs

In this paper, we conduct a structured synthesis of existing theories and frameworks on human-human collaborative writing to assess how co-writing with AI reshapes writing in knowledge work settings, and draw seven insights along with design implications for human-AI collaborative writing tools. Our main argument is that, as we delegate more writing to AI, our cognitive process shifts from the traditional plan/write/review process to a plan/delegate/review process, reshaping the distribution of the cognitive labor between human writers and AI writing tools. To ensure that humans cognitively stay in the loop, we discuss a ``prototyping'' approach, where the writing tool scaffolds a healthier distribution of cognition through rapid, small-scale iterations that keep writers mentally in the loop even when generation is delegated to AI. We aim to bring theoretical grounding and practical guidance to the interaction designs of human-AI collaborative writing, with the goal of balancing automation with cognitive integrity.

cs.HC↗

Shielded Trajectory Sampling for Personalized Relational Safety in Conversational AI

Anthropomorphic conversational systems can provide companionship and emotional support, but repeated, seemingly harmless interactions can encourage excessive dependence and reduced human contact. We formulate safety in user-AI relationships as a sequential decision problem, using conversational cues to track the relationship and selecting responses according to their anticipated effects over time. A conditional Generative Flow Network learns from sampled response plans to select replies that preserve the largest estimated mass of safe and useful continuations. Model-based checks screen plans during training and responses during interaction, requiring at least one sequence of future response strategies that meets a prescribed safety bound under every candidate user model, independently of the learned network. Strategies are selected directly from learned first-step probabilities, avoiding online plan sampling. We evaluate the approach in simulation against reactive safeguards, filters with and without memory, and alternative planners, using exact computation to validate the learned approximation. We also evaluate relational cue extraction from real user messages in WildChat. To our knowledge, this is the first approach to select conversational replies by anticipating their effects on the user-AI relationship while imposing an explicit model-based safety bound on future response sequences. Results show close agreement with exact planning, lower decision costs at longer horizons, and improved safety-utility trade-offs when protective responses are less effective. The approach balances immediate support with the longer-term effects of repeated interaction. The source code is available at https://github.com/benedettapicano/ANTHROPOMORPHIC_SAFETY_TRAJ.

cs.HC↗

Projected Route Cues for Bouldering: Effects of Circular, Linear, and Combined Highlighting on Route Observation and Climbing

Before attempting a bouldering problem, climbers inspect the route to find its holds and plan their moves, which novices find difficult on walls where many routes overlap. We built a projector-camera system that projects route cues onto the wall and compared three cues: circles on the route holds, limb-colored lines connecting the holds, and both combined. Ninety-six participants, 61 of them first-time climbers, each climbed four routes, one per cue and one without projection, in a Latin-square order. In mixed-effects models, all three cues shortened the time before the first attempt; with circles, its geometric mean fell from 83 s to 39 s. Circles also increased the number of moves made within each trial. Lines led to more use of off-route holds than circles, mainly on routes where such holds lay under the projected lines. Participants preferred the combined cue, although no behavioral advantage of it over circles was detected.

cs.HC↗