Searcharxiv⌕ Search

arXiv · 2611.14284

Ninety-Six Ways to Measure a Scribe: Fragmented Evaluation of Ambient Scribes in Healthcare

Abstract

Ambient artificial intelligence (AI) scribes have moved rapidly from experimental pilots to enterprise-wide deployment across health systems, promising reduced documentation burden and improved clinician experience. While research evaluating these tools has expanded quickly, there are no agreed-upon protocols or standardized metrics for assessing their performance, making it difficult to compare findings or cumulatively synthesize evidence across studies. Across 54 recent evaluation studies, we identify ten evaluation dimensions and 40 distinct metrics, operationalized in 96 different ways-35 of them (36%) used by only a single study. Assessment is dominated by documentation efficiency and time, measured in 44 of 54 studies. The other dimensions appear at broadly similar rates (16-24 studies each), and safety and clinical outcome least consistently of all. Ripple effects on the broader care system, such as impact on patients' care outcomes, redistribution of work, changes in billing and coding intensity, and downstream use of notes by other clinicians are rarely assessed. Moreover, the metrics used rely heavily on self-report and are fragmented and inconsistently operationalized, which limits comparability and hinders synthesis across studies. As an effort to support more rigorous and cumulative evaluation, in this paper we review the study designs, data sources, and metrics commonly used to evaluate ambient AI scribes in healthcare settings, discuss the limitations of current evaluation practice, and argue for a more holistic approach grounded in a standardized, multi-dimensional set of metrics.

Explore related subjects

Keep this discovery

Explore connections, maps & timelines

BibTeXRIS

Amama Mahmood, Shiye Cao, Laura Fleig, Junan Jiang, Theodore Jensen, Yee-Yin Choong, Kristen Greene, Chien-Ming Huang. 2026-11-11. Ninety-Six Ways to Measure a Scribe: Fragmented Evaluation of Ambient Scribes in Healthcare. https://searcharxiv.com/abs/2611.14284

Citations retain the original authors and DOI and include this site’s record URL. If this discovery service contributed to your research, you can also cite Searcharxiv by Indiciae: BibTeX · RIS.

KEEP EXPLORING

Related papers

Human-AI Writing as Distributed Cognition: Reassessing Collaborative Writing Theories in the Age of LLMs

In this paper, we conduct a structured synthesis of existing theories and frameworks on human-human collaborative writing to assess how co-writing with AI reshapes writing in knowledge work settings, and draw seven insights along with design implications for human-AI collaborative writing tools. Our main argument is that, as we delegate more writing to AI, our cognitive process shifts from the traditional plan/write/review process to a plan/delegate/review process, reshaping the distribution of the cognitive labor between human writers and AI writing tools. To ensure that humans cognitively stay in the loop, we discuss a ``prototyping'' approach, where the writing tool scaffolds a healthier distribution of cognition through rapid, small-scale iterations that keep writers mentally in the loop even when generation is delegated to AI. We aim to bring theoretical grounding and practical guidance to the interaction designs of human-AI collaborative writing, with the goal of balancing automation with cognitive integrity.

cs.HC↗

Shielded Trajectory Sampling for Personalized Relational Safety in Conversational AI

Anthropomorphic conversational systems can provide companionship and emotional support, but repeated, seemingly harmless interactions can encourage excessive dependence and reduced human contact. We formulate safety in user-AI relationships as a sequential decision problem, using conversational cues to track the relationship and selecting responses according to their anticipated effects over time. A conditional Generative Flow Network learns from sampled response plans to select replies that preserve the largest estimated mass of safe and useful continuations. Model-based checks screen plans during training and responses during interaction, requiring at least one sequence of future response strategies that meets a prescribed safety bound under every candidate user model, independently of the learned network. Strategies are selected directly from learned first-step probabilities, avoiding online plan sampling. We evaluate the approach in simulation against reactive safeguards, filters with and without memory, and alternative planners, using exact computation to validate the learned approximation. We also evaluate relational cue extraction from real user messages in WildChat. To our knowledge, this is the first approach to select conversational replies by anticipating their effects on the user-AI relationship while imposing an explicit model-based safety bound on future response sequences. Results show close agreement with exact planning, lower decision costs at longer horizons, and improved safety-utility trade-offs when protective responses are less effective. The approach balances immediate support with the longer-term effects of repeated interaction. The source code is available at https://github.com/benedettapicano/ANTHROPOMORPHIC_SAFETY_TRAJ.

cs.HC↗

Projected Route Cues for Bouldering: Effects of Circular, Linear, and Combined Highlighting on Route Observation and Climbing

Before attempting a bouldering problem, climbers inspect the route to find its holds and plan their moves, which novices find difficult on walls where many routes overlap. We built a projector-camera system that projects route cues onto the wall and compared three cues: circles on the route holds, limb-colored lines connecting the holds, and both combined. Ninety-six participants, 61 of them first-time climbers, each climbed four routes, one per cue and one without projection, in a Latin-square order. In mixed-effects models, all three cues shortened the time before the first attempt; with circles, its geometric mean fell from 83 s to 39 s. Circles also increased the number of moves made within each trial. Lines led to more use of off-route holds than circles, mainly on routes where such holds lay under the projected lines. Participants preferred the combined cue, although no behavioral advantage of it over circles was detected.

cs.HC↗