SearcharxivSearch

arXiv subjects

Casper Harteveld

Publications and source records attributed to Casper Harteveld.

At least 19 recordsLinked to original sources

Assessing Learning Processes with Multimodal Data in Virtual Reality Learning Environments

Assessing learning in virtual reality (VR) environments typically relied on traditional pre-post content retention tests, revealing little about the process of learnng within such immersive environments. Multimodal data from player activity in VR is promising to better measure learning processes and higher-order skills, but little research in the learning sciences has explored how such data can be combined to provide meaningful measures. To address this, we explored multimodal sources of data from a VR escape room containing hands-on logic puzzles that require problem-solving strategies and engagement in verbal metacognitive reflections. We leverage VR's affordances to immerse users in sustained dialog and promote multimodal interactions in a digital environment to understand how logfile and verbal data reveal participants' reasoning skills versus guesswork.

cs.HC

Conversational Successes and Breakdowns in Everyday Smart Glasses Use

Non-Display Smart Glasses hold the potential to support everyday activities by combining continuous environmental sensing with voice-only interaction powered by large language models (LLMs). Understanding how conversational successes and breakdowns arise in everyday contexts can better inform the design of future voice-only interfaces. To investigate this, we conducted a month-long collaborative autoethnography (n=2) to identify patterns of successes and breakdowns when using such devices. We then compare these patterns with prior findings on voice-only interactions to highlight the unique affordances and opportunities offered by non-display smart glasses.

cs.HC

It Depends: Re_Authoring Play Through Clinical Reasoning in Wearable AR Rehab Games

Augmented reality games hold promise for rehabilitation, yet most remain confined to laboratory studies with limited clinical uptake. Recent advances in spatial computing, especially lightweight, glasses_form_factor AR, create a timely opportunity to embed rehabilitative play into clinical practice and daily contexts. To investigate this potential, we systematically reviewed 132 applications and conducted playtesting with 14 licensed physical therapists. Our analysis revealed three ways therapists re_authored AR games: co_authored play (reshaping movements, progressions, and difficulty), situated play (adapting across specialties, conditions, and contexts), and dual play (mediating both physical recovery and psychological support). We reframe therapists' frequent phrase_It depends_as a generative design principle. This study contributes a clinical reasoning_based framework and design principles and guidelines for creating personalized, situated forms of play that align with therapists' everyday workflows and inform future lab_to_clinic translation.

cs.HC

Reimagining Wearable AR Gesture Design: Physical Therapy Reasoning in Everyday Contexts

Lightweight augmented reality (AR) glasses are increasingly entering everyday use, extending interaction design beyond short, isolated sessions. However, most existing gesture vocabularies are inherited from VR headsets or early AR goggles. These systems tend to prioritize recognizer accuracy while overlooking fatigue, sustainability, and social legibility in daily contexts. To address this gap, we collaborated with physical therapists (PTs) to reimagine gesture design for everyday AR, drawing on their expertise in safe and sustainable movement. Through a review of 104 AR applications, we identified 15 common gesture intents and implemented an on-device gesture generator. Ten licensed physical therapists, with an average of 14.8 years of professional experience, then shaped these gesture intents through three iterative stages: unaided gesture performance, PT-guided gesture substitution, and stage-aware card sorting. This work contributes (1) a PT-informed gesture translation method, (2) the Everyday-AR Golden Ergonomic Canvas, and (3) a stage-aware social legibility framework that illustrates how gesture suitability shifts with social readability. Together, these contributions provide a recognizer-agnostic reference framework for designing sustainable and socially coherent gesture vocabularies for lightweight AR glasses.

cs.HC

Bridging Pedagogy and Play: Introducing a Language Mapping Interface for Human-AI Co-Creation in Educational Game Design

Educational games can foster critical thinking, problem-solving, and motivation, yet instructors often find it difficult to design games that reliably achieve specific learning outcomes. Existing authoring environments reduce the need for programming expertise, but they do not eliminate the underlying challenges of educational game design, and they can leave non-expert designers reliant on opaque suggestions from AI systems. We designed a controlled natural language framework-based web tool that positions language as the primary interface for LLM-assisted educational game design. In the tool, users and an LLM assistant collaboratively develop a structured language that maps pedagogy to gameplay through four linked components. We argue that, by making pedagogical intent explicit and editable in the interface, the tool has the potential to lower design barriers for non-expert designers, preserves human agency in critical decisions, and enables alignment and reflections between pedagogy and gameplay during and after co-creation.

cs.HC

Tangible Intangibles: Exploring Embodied Emotion in Mixed Reality for Art Therapy

This in-person studio explores how mixed reality (MR) and biometrics can make intangible emotional states tangible through embodied art practices. We begin with two well-established modalities, clay sculpting and free-form 2D drawing, to ground participants in somatic awareness and manual, reflective expression. Building on this baseline, we introduce an MR prototype that maps physiological signals (e.g., breath, heart rate variability, eye movement dynamics) to visual and spatial parameters (color saturation, pulsing, motion qualities), generating ''3D emotional artifacts.'' The full-day program balances theory (somatic psychology, embodied cognition, expressive biosignals), hands-on making, and comparative reflection to interrogate what analog and digital modalities respectively afford for awareness, expression, and meaning-making. Participants will (1) experience and compare analog and MR-based journaling of emotion; (2) prototype and critique mappings from biosignals to visual/spatial feedback; and (3) articulate design principles for trauma-informed, hybrid workflows that amplify interoceptive literacy without overwhelming the user. The expected contributions include a shared design vocabulary for biometric expressivity, a set of generative constraints for future TEI work on emotional archiving, and actionable insights into when automated translation supports or hinders embodied connection.

cs.HC

Supporting Aging Well through Accessible Digital Games: The Supplemental Role of AI in Game Design for Older Adults

As the population continues to age, and gaming continues to grow as a hobby for older people, heterogeneity among older adult gamers is increasing. We argue that traditional game-based accessibility features, such as simplified input schemes, redundant information channels, and increased legibility of digital user interfaces, are increasingly limited in the face of this heterogeneity. This is because such features affect all older adult players simultaneously and therefore are designed generically. We introduce artificial intelligence, although it has its own limitations and ethical concerns, as a method of creating player-based accessibility features, given the adaptive nature of the emerging technology. These accessibility features may help to address unique assemblage of accessibility needs an individual may accumulate through age. We adopt insights from gerontology, HCI, and disability studies into the digital game design discourse for older adults, and we contribute insight that can guide the integration of player-based accessibility features to supplement game-based counterparts. The accessibility of digital games for heterogenous older adult audience is paramount, as the medium offers short-term social, emotional, psychological, cognitive, and physical that support the long-term goal of aging well.

cs.HC

Integrating Artificial Intelligence as Assistive Technology for Older Adult Gamers: A Pilot Study

With respect to digital games, older adults are a demographic that is often underserved due to an industry-wide focus on younger audiences' preferences and skill sets. Meanwhile, as artificial intelligence (AI) continues to expand into everyday technologies, its assistive capabilities have been recognized, suggesting its potential in improving the gaming experience for older gamers. To study this potential, we iteratively developed a pilot survey aimed at understanding older adult gamers' current gameplay preference, challenges they are facing, and their perspectives of AI usage in gaming. This article contributes an overview of our iterative survey-design workflow, and pilot results from 39 participants. During each iteration, we analyzed the survey's efficacy and adjusted the content, language, and format to better capture meaningful data, and was able to create a refined survey for a larger, more representative future parent study. At the same time, preliminary findings suggest that for older adult gamers, usability issues in gaming remain key obstacles, while this demographic's perceptions of AI are shaped by both its practical benefits and concerns about autonomy and complexity. These findings also offer early insights for the design of age-inclusive, AI-supported gaming experiences.

cs.HC

Kaleidoscope Gallery: Exploring Ethics and Generative AI Through Art

Ethical theories and Generative AI (GenAI) models are dynamic concepts subject to continuous evolution. This paper investigates the visualization of ethics through a subset of GenAI models. We expand on the emerging field of Visual Ethics, using art as a form of critical inquiry and the metaphor of a kaleidoscope to invoke moral imagination. Through formative interviews with 10 ethics experts, we first establish a foundation of ethical theories. Our analysis reveals five families of ethical theories, which we then transform into images using the text-to-image (T2I) GenAI model. The resulting imagery, curated as Kaleidoscope Gallery and evaluated by the same experts, revealed eight themes that highlight how morality, society, and learned associations are central to ethical theories. We discuss implications for critically examining T2I models and present cautions and considerations. This work contributes to examining ethical theories as foundational knowledge that interrogates GenAI models as socio-technical systems.

cs.CY

MOSAAIC: Managing Optimization towards Shared Autonomy, Authority, and Initiative in Co-creation

Striking the appropriate balance between humans and co-creative AI is an open research question in computational creativity. Co-creativity, a form of hybrid intelligence where both humans and AI take action proactively, is a process that leads to shared creative artifacts and ideas. Achieving a balanced dynamic in co-creativity requires characterizing control and identifying strategies to distribute control between humans and AI. We define control as the power to determine, initiate, and direct the process of co-creation. Informed by a systematic literature review of 172 full-length papers, we introduce MOSAAIC (Managing Optimization towards Shared Autonomy, Authority, and Initiative in Co-creation), a novel framework for characterizing and balancing control in co-creation. MOSAAIC identifies three key dimensions of control: autonomy, initiative, and authority. We supplement our framework with control optimization strategies in co-creation. To demonstrate MOSAAIC's applicability, we analyze the distribution of control in six existing co-creative AI case studies and present the implications of using this framework.

cs.AI

GPT for Games: An Updated Scoping Review (2020-2024)

Due to GPT's impressive generative capabilities, its applications in games are expanding rapidly. To offer researchers a comprehensive understanding of the current applications and identify both emerging trends and unexplored areas, this paper introduces an updated scoping review of 177 articles, 122 of which were published in 2024, to explore GPT's potential for games. By coding and synthesizing the papers, we identify five prominent applications of GPT in current game research: procedural content generation, mixed-initiative game design, mixed-initiative gameplay, playing games, and game user research. Drawing on insights from these application areas and emerging research, we propose future studies should focus on expanding the technical boundaries of the GPT models and exploring the complex interaction dynamics between them and users. This review aims to illustrate the state of the art in innovative GPT applications in games, offering a foundation to enrich game development and enhance player experiences through cutting-edge AI innovations.

cs.AI

Exploring Eye Tracking to Detect Cognitive Load in Complex Virtual Reality Training

Virtual Reality (VR) has been a beneficial training tool in fields such as advanced manufacturing. However, users may experience a high cognitive load due to various factors, such as the use of VR hardware or tasks within the VR environment. Studies have shown that eye-tracking has the potential to detect cognitive load, but in the context of VR and complex spatiotemporal tasks (e.g., assembly and disassembly), it remains relatively unexplored. Here, we present an ongoing study to detect users' cognitive load using an eye-tracking-based machine learning approach. We developed a VR training system for cold spray and tested it with 22 participants, obtaining 19 valid eye-tracking datasets and NASA-TLX scores. We applied Multi-Layer Perceptron (MLP) and Random Forest (RF) models to compare the accuracy of predicting cognitive load (i.e., NASA-TLX) using pupil dilation and fixation duration. Our preliminary analysis demonstrates the feasibility of using eye tracking to detect cognitive load in complex spatiotemporal VR experiences and motivates further exploration.

cs.HC

Designing a Virtual Reality Training Apprenticeship for Cold Spray Advanced Manufacturing

Apprenticeship and training programs in advanced manufacturing frequently encounter safety and accessibility concerns due to using heavy machinery. Virtual Reality (VR) training addresses such constraints while maintaining the spatial and procedural learning requirements of such training. However, designing effective VR training is challenging because advanced manufacturing processes are complex and require experts to train novices for a long time. This paper presents a VR Training Apprenticeship (VRTA) tailored for cold spray, which we carefully designed to teach novices step-by-step this particular advanced manufacturing process. To assess its effectiveness, we conducted an exploratory study ($n = 22$). We evaluated user experience (UX) measures in the form of quantitative scales, users' qualitative insights, and task performance with real-world machinery after the VR training. We discuss how the VRTA design contributed to the effectiveness and the challenges of considering VR training for advanced manufacturing.

cs.HC

Exploring Dimensions of Expertise in AR-Guided Psychomotor Tasks

This study aimed to explore how novices and experts differ in performing complex psychomotor tasks guided by augmented reality (AR), focusing on decision-making and technical proficiency. Participants were divided into novice and expert groups based on a pre-questionnaire assessing their technical skills and theoretical knowledge of precision inspection. Participants completed a post-study questionnaire that evaluated cognitive load (NASA-TLX), self-efficacy, and experience with the HoloLens 2 and AR app, along with general feedback. We used multimodal data from AR devices and wearables, including hand tracking, galvanic skin response, and gaze tracking, to measure key performance metrics. We found that experts significantly outperformed novices in decision-making speed, efficiency, accuracy, and dexterity in the execution of technical tasks. Novices exhibited a positive correlation between perceived performance in the NASA-TLX and the GSR amplitude, indicating that higher perceived performance is associated with increased physiological stress responses. This study provides a foundation for designing multidimensional expertise estimation models to enable personalized industrial AR training systems.

cs.HC

GPT for Games: A Scoping Review (2020-2023)

This paper introduces a scoping review of 55 articles to explore GPT's potential for games, offering researchers a comprehensive understanding of the current applications and identifying both emerging trends and unexplored areas. We identify five key applications of GPT in current game research: procedural content generation, mixed-initiative game design, mixed-initiative gameplay, playing games, and game user research. Drawing from insights in each of these application areas, we propose directions for future research in each one. This review aims to lay the groundwork by illustrating the state of the art for innovative GPT applications in games, promising to enrich game development and enhance player experiences with cutting-edge AI innovations.

cs.HC

Snake Story: Exploring Game Mechanics for Mixed-Initiative Co-creative Storytelling Games

Mixed-initiative co-creative storytelling games have existed for some time as a way to merge storytelling with play. However, modern mixed-initiative co-creative storytelling games predominantly prioritize story creation over gameplay mechanics, which might not resonate with all players. As such, there is untapped potential for creating mixed-initiative games with more complex mechanics in which players can engage with both co-creation and gameplay goals. To explore the potential of more prominent gameplay in mixed-initiative co-creative storytelling games, we created Snake Story, a variation of the classic Snake game featuring a human-AI co-writing element. To explore how players interact with the mixed-initiative game, we conducted a qualitative playtest with 11 participants. Analysis of both think-aloud and interview data revealed that players' strategies and experiences were affected by their perception of Snake Story as either a collaborative tool, a traditional game, or a combination of both. Based on these findings, we present design considerations for future development in mixed-initiative co-creative gaming.

cs.HC

Thought Bubbles: A Proxy into Players' Mental Model Development

Studying mental models has recently received more attention, aiming to understand the cognitive aspects of human-computer interaction. However, there is not enough research on the elicitation of mental models in complex dynamic systems. We present Thought Bubbles as an approach for eliciting mental models and an avenue for understanding players' mental model development in interactive virtual environments. We demonstrate the use of Thought Bubbles in two experimental studies involving 250 participants playing a supply chain game. In our analyses, we rely on Situation Awareness (SA) levels, including perception, comprehension, and projection, and show how experimental manipulations such as disruptions and information sharing shape players' mental models and drive their decisions depending on their behavioral profile. Our results provide evidence for the use of thought bubbles in uncovering cognitive aspects of behavior by indicating how disruption location and availability of information affect people's mental model development and influence their decisions.

cs.HC

To Trust or to Stockpile: Modeling Human-Simulation Interaction in Supply Chain Shortages

Understanding decision-making in dynamic and complex settings is a challenge yet essential for preventing, mitigating, and responding to adverse events (e.g., disasters, financial crises). Simulation games have shown promise to advance our understanding of decision-making in such settings. However, an open question remains on how we extract useful information from these games. We contribute an approach to model human-simulation interaction by leveraging existing methods to characterize: (1) system states of dynamic simulation environments (with Principal Component Analysis), (2) behavioral responses from human interaction with simulation (with Hidden Markov Models), and (3) behavioral responses across system states (with Sequence Analysis). We demonstrate this approach with our game simulating drug shortages in a supply chain context. Results from our experimental study with 135 participants show different player types (hoarders, reactors, followers), how behavior changes in different system states, and how sharing information impacts behavior. We discuss how our findings challenge existing literature.

cs.HC