arXiv subjects

Lev Tankelevitch

Publications and source records attributed to Lev Tankelevitch.

15 recordsLinked to original sources

arXiv2026

Explaining Too Much? Understanding How Large Language Model Reasoning Traces Influence Performance and Metacognition

Large Language Model interfaces are increasingly verbose, exposing intermediate reasoning traces alongside final answers. Traces are framed as transparency mechanisms, yet it is unclear how people use them to solve problems. We report a preregistered between-subjects study (N = 559) in which participants solved ten LSAT-style reasoning problems under one of three conditions: an Answer-only baseline, a Full-trace revealed before the answer, and a Summary-trace presented alongside the answer. Summaries preserved task performance at the no-trace baseline while significantly elevating trust and hedonic appeal, establishing that trace exposure shifts subjective appraisal of the interaction without bringing performance benefits. Under an open-weight reasoning model exposing verbose intermediate output, full traces additionally impaired performance relative to the answer-only baseline. Across all conditions, participants substantially overestimated their performance, and no trace format supported calibrated self-evaluation. Further analysis indicates that hedonic appeal, not trust, carries the indirect path to overestimation, consistent with a processing-fluency account. Reasoning traces are best understood as user-facing interface artifacts rather than transparent windows into model cognition, and calibration is unlikely to emerge from the traces themselves and may best be scaffolded by interactions that elicit users' own reasoning first.

cs.HC↗

arXiv2026

Scaffolding Human-AI Collaboration: A Field Experiment on Behavioral Protocols and Cognitive Reframing

Organizations have widely deployed generative AI tools, yet productivity gains remain uneven, suggesting that how people use AI matters as much as whether they have access. We conducted a field experiment with 388 employees at a Fortune 500 retailer to test two scaffolding interventions for human-AI collaboration. All participants had access to the same AI tool; we varied only the structure surrounding its use. A behavioral scaffolding intervention (a structured protocol requiring joint AI use within pairs) was associated with lower document quality relative to unstructured use and substantially lower document production. A cognitive scaffolding intervention (partnership training that reframed AI as a thought partner) was associated with higher individual document quality at the top of the distribution. Treatment participants also showed greater positive belief change across the session, though sensitivity analyses suggest this likely reflects recovery from carry-over effects rather than genuine training-induced shifts. Both findings are subject to design limitations including an AM/PM session confound, differential attrition, and LLM grading sensitivity to document length.

econ.GN↗

arXiv2026

Nudging Attention to Workplace Meeting Goals: A Large-Scale, Preregistered Field Experiment

Ineffective meetings are pervasive. Thinking ahead explicitly about meeting goals may improve effectiveness, but current collaboration platforms lack integrated support. We tested a lightweight goal-reflection intervention in a preregistered field experiment in a global technology company (361 employees, 7196 meetings). Over two weeks, workers in the treatment group completed brief pre-meeting surveys in their collaboration platform, nudging attention to goals for upcoming meetings. To measure impact, both treatment and control groups completed post-meeting surveys about meeting effectiveness. While the intervention impact on meeting effectiveness was not statistically significant, mixed-methods findings revealed improvements in self-reported awareness and behaviour across both groups, with post-meeting surveys unintentionally functioning as an intervention. We highlight the promise of supporting goal reflection, while noting challenges of evaluating and supporting workplace reflection for meetings, including workflow and collaboration norms, and attitudes and behaviours around meeting preparation. We conclude with implications for designing technological support for meeting intentionality.

cs.HC↗

arXiv2025

Metacognition and Confidence Dynamics in Advice Taking from Generative AI

Generative Artificial Intelligence (GenAI) can aid humans in a wide range of tasks, but its effectiveness critically depends on users being able to evaluate the accuracy of GenAI outputs and their own expertise. Here we asked how confidence in self and GenAI contributes to decisions to seek and rely on advice from GenAI ('prospective confidence'), and how advice-taking in turn shapes this confidence ('retrospective confidence'). In a novel paradigm involving text generation, participants formulated plans for events, and could request advice from a GenAI (Study 1; N=200) or were randomly assigned to receive advice (Study 2; N=300), which they could rely on or ignore. Advice requests in Study 1 were related to higher prospective confidence in GenAI and lower confidence in self. Advice-seekers showed increased retrospective confidence in GenAI, while those who declined advice showed increased confidence in self. Random assignment in Study 2 revealed that advice exposure increases confidence in GenAI and in self, suggesting that GenAI advice-taking causally boosts retrospective confidence. These results were mirrored in advice reliance, operationalised as the textual similarity between GenAI advice and participants' responses, with reliance associated with increased retrospective confidence in both GenAI and self. Critically, participants who chose to obtain/rely on advice provided more detailed responses (likely due to the output's verbosity), but failed to check the output thoroughly, missing key information. These findings underscore a key role for confidence in interactions with GenAI, shaped by both prior beliefs about oneself and the reliability of AI, and context-dependent exposure to advice.

cs.HC↗

arXiv2025

Understanding, Protecting, and Augmenting Human Cognition with Generative AI: A Synthesis of the CHI 2025 Tools for Thought Workshop

Generative AI (GenAI) radically expands the scope and capability of automation for work, education, and everyday tasks, a transformation posing both risks and opportunities for human cognition. How will human cognition change, and what opportunities are there for GenAI to augment it? Which theories, metrics, and other tools are needed to address these questions? The CHI 2025 workshop on Tools for Thought aimed to bridge an emerging science of how the use of GenAI affects human thought, from metacognition to critical thinking, memory, and creativity, with an emerging design practice for building GenAI tools that both protect and augment human thought. Fifty-six researchers, designers, and thinkers from across disciplines as well as industry and academia, along with 34 papers and portfolios, seeded a day of discussion, ideation, and community-building. We synthesize this material here to begin mapping the space of research and design opportunities and to catalyze a multidisciplinary community around this pressing area of research.

cs.HC↗

arXiv2025

What Does Success Look Like? Catalyzing Meeting Intentionality with AI-Assisted Prospective Reflection

Despite decades of HCI and Meeting Science research, complaints about ineffective meetings are still pervasive. We argue that meeting technologies lack support for prospective reflection, that is, thinking about why a meeting is needed and what might happen. To explore this, we designed a Meeting Purpose Assistant (MPA) technology probe to coach users to articulate their meeting's purpose and challenges, and act accordingly. The MPA used Generative AI to support personalized and actionable prospective reflection across the diversity of meeting contexts. Using a participatory prompting methodology, 18 employees of a global technology company reflected with the MPA on upcoming meetings. Observed impacts were: clarifying meeting purposes, challenges, and success conditions; changing perspectives and flexibility; improving preparation and communication; and proposing changed plans. We also identify perceived social, temporal, and technological barriers to using the MPA. We present system and workflow design considerations for developing AI-assisted reflection support for meetings.

cs.HC↗

arXiv2025

Are We On Track? AI-Assisted Active and Passive Goal Reflection During Meetings

Meetings often suffer from a lack of intentionality, such as unclear goals and straying off-topic. Identifying goals and maintaining their clarity throughout a meeting is challenging, as discussions and uncertainties evolve. Yet meeting technologies predominantly fail to support meeting intentionality. AI-assisted reflection is a promising approach. To explore this, we conducted a technology probe study with 15 knowledge workers, integrating their real meeting data into two AI-assisted reflection probes: a passive and active design. Participants identified goal clarification as a foundational aspect of reflection. Goal clarity enabled people to assess when their meetings were off-track and reprioritize accordingly. Passive AI intervention helped participants maintain focus through non-intrusive feedback, while active AI intervention, though effective at triggering immediate reflection and action, risked disrupting the conversation flow. We identify three key design dimensions for AI-assisted reflection systems, and provide insights into design trade-offs, emphasizing the need to adapt intervention intensity and timing, balance democratic input with efficiency, and offer user control to foster intentional, goal-oriented behavior during meetings and beyond.

cs.HC↗

arXiv2025

The New Calculator? Practices, Norms, and Implications of Generative AI in Higher Education

Generative AI (GenAI) has introduced myriad opportunities and challenges for higher education. Anticipating this potential transformation requires understanding students' contextualised practices and norms around GenAI. We conducted semi-structured interviews with 26 students and 11 educators from diverse departments across two universities. Grounded in Strong Structuration Theory, we find diversity in students' uses and motivations for GenAI. Occurring in the context of unclear university guidelines, institutional fixation on plagiarism, and inconsistent educator communication, students' practices are informed by unspoken rules around appropriate use, GenAI limitations and reliance strategies, and consideration of agency and skills. Perceived impacts include changes in confidence, and concerns about skill development, relationships with educators, and plagiarism. Both groups envision changes in universities' attitude to GenAI, responsible use training, assessments, and integration of GenAI into education. We discuss socio-technical implications in terms of current and anticipated changes in the external and internal structures that contextualise students' GenAI use.

cs.HC↗

arXiv2024

Nods of Agreement: Webcam-Driven Avatars Improve Meeting Outcomes and Avatar Satisfaction Over Audio-Driven or Static Avatars in All-Avatar Work Videoconferencing

Avatars are edging into mainstream videoconferencing, but evaluation of how avatar animation modalities contribute to work meeting outcomes has been limited. We report a within-group videoconferencing experiment in which 68 employees of a global technology company, in 16 groups, used the same stylized avatars in three modalities (static picture, audio-animation, and webcam-animation) to complete collaborative decision-making tasks. Quantitatively, for meeting outcomes, webcam-animated avatars improved meeting effectiveness over the picture modality and were also reported to be more comfortable and inclusive than both other modalities. In terms of avatar satisfaction, there was a similar preference for webcam animation as compared to both other modalities. Our qualitative analysis shows participants expressing a preference for the holistic motion of webcam animation, and that meaningful movement outweighs realism for meeting outcomes, as evidenced through a systematic overview of ten thematic factors. We discuss implications for research and commercial deployment and conclude that webcam-animated avatars are a plausible alternative to video in work meetings.

cs.HC↗

arXiv2024

"It's like a rubber duck that talks back": Understanding Generative AI-Assisted Data Analysis Workflows through a Participatory Prompting Study

Generative AI tools can help users with many tasks. One such task is data analysis, which is notoriously challenging for non-expert end-users due to its expertise requirements, and where AI holds much potential, such as finding relevant data sources, proposing analysis strategies, and writing analysis code. To understand how data analysis workflows can be assisted or impaired by generative AI, we conducted a study (n=15) using Bing Chat via participatory prompting. Participatory prompting is a recently developed methodology in which users and researchers reflect together on tasks through co-engagement with generative AI. In this paper we demonstrate the value of the participatory prompting method. We found that generative AI benefits the information foraging and sensemaking loops of data analysis in specific ways, but also introduces its own barriers and challenges, arising from the difficulties of query formulation, specifying context, and verifying results.

cs.HC↗

arXiv2024

The CoExplorer Technology Probe: A Generative AI-Powered Adaptive Interface to Support Intentionality in Planning and Running Video Meetings

Effective meetings are effortful, but traditional videoconferencing systems offer little support for reducing this effort across the meeting lifecycle. Generative AI (GenAI) has the potential to radically redefine meetings by augmenting intentional meeting behaviors. CoExplorer, our novel adaptive meeting prototype, preemptively generates likely phases that meetings would undergo, tools that allow capturing attendees' thoughts before the meeting, and for each phase, window layouts, and appropriate applications and files. Using CoExplorer as a technology probe in a guided walkthrough, we studied its potential in a sample of participants from a global technology company. Our findings suggest that GenAI has the potential to help meetings stay on track and reduce workload, although concerns were raised about users' agency, trust, and possible disruption to traditional meeting norms. We discuss these concerns and their design implications for the development of GenAI meeting technology.

cs.HC↗

arXiv2024

Mental Models of Meeting Goals: Supporting Intentionality in Meeting Technologies

Ineffective meetings due to unclear goals are major obstacles to productivity, yet support for intentionality is surprisingly scant in our meeting and allied workflow technologies. To design for intentionality, we need to understand workers' attitudes and practices around goals. We interviewed 21 employees of a global technology company and identified contrasting mental models of meeting goals: meetings as a means to an end, and meetings as an end in themselves. We explore how these mental models impact how meeting goals arise, goal prioritization, obstacles to considering goals, and how lack of alignment around goals may create tension between organizers and attendees. We highlight the challenges in balancing preparation, constraining scope, and clear outcomes, with the need for intentional adaptability and discovery in meetings. Our findings have implications for designing systems which increase effectiveness in meetings by catalyzing intentionality and reducing tension in the organisation of meetings.

cs.HC↗

arXiv2024

Ironies of Generative AI: Understanding and mitigating productivity loss in human-AI interactions

Generative AI (GenAI) systems offer opportunities to increase user productivity in many tasks, such as programming and writing. However, while they boost productivity in some studies, many others show that users are working ineffectively with GenAI systems and losing productivity. Despite the apparent novelty of these usability challenges, these 'ironies of automation' have been observed for over three decades in Human Factors research on the introduction of automation in domains such as aviation, automated driving, and intelligence. We draw on this extensive research alongside recent GenAI user studies to outline four key reasons for productivity loss with GenAI systems: a shift in users' roles from production to evaluation, unhelpful restructuring of workflows, interruptions, and a tendency for automation to make easy tasks easier and hard tasks harder. We then suggest how Human Factors research can also inform GenAI system design to mitigate productivity loss by using approaches such as continuous feedback, system personalization, ecological interface design, task stabilization, and clear task allocation. Thus, we ground developments in GenAI system usability in decades of Human Factors research, ensuring that the design of human-AI interactions in this rapidly moving field learns from history instead of repeating it.

cs.HC↗

arXiv2023

The Metacognitive Demands and Opportunities of Generative AI

Generative AI (GenAI) systems offer unprecedented opportunities for transforming professional and personal work, yet present challenges around prompting, evaluating and relying on outputs, and optimizing workflows. We argue that metacognition$\unicode{x2013}$the psychological ability to monitor and control one's thoughts and behavior$\unicode{x2013}$offers a valuable lens to understand and design for these usability challenges. Drawing on research in psychology and cognitive science, and recent GenAI user studies, we illustrate how GenAI systems impose metacognitive demands on users, requiring a high degree of metacognitive monitoring and control. We propose these demands could be addressed by integrating metacognitive support strategies into GenAI systems, and by designing GenAI systems to reduce their metacognitive demand by targeting explainability and customizability. Metacognition offers a coherent framework for understanding the usability challenges posed by GenAI, and provides novel research and design directions to advance human-AI interaction.

cs.HC↗

arXiv2023

Meeting effectiveness and inclusiveness: large-scale measurement, identification of key features, and prediction in real-world remote meetings

Workplace meetings are vital to organizational collaboration, yet relatively little progress has been made toward measuring meeting effectiveness and inclusiveness at scale. The recent rise in remote and hybrid meetings represents an opportunity to do so via computer-mediated communication (CMC) systems. Here, we share the results of an effective and inclusive meetings survey embedded within a CMC system in a diverse set of companies and organizations. We correlate the survey results with objective metrics available from the CMC system to identify the generalizable attributes that characterize perceived effectiveness and inclusiveness in meetings. Additionally, we explore a predictive model of meeting effectiveness and inclusiveness based solely on objective meeting attributes. Lastly, we show challenges and discuss solutions around the subjective measurement of meeting experiences. To our knowledge, this is the largest data-driven study conducted after the pandemic peak to measure, understand, and predict effectiveness and inclusiveness in real-world meetings at an organizational scale.

cs.HC↗