arXiv subjects

Jessica Van Brummelen

Publications and source records attributed to Jessica Van Brummelen.

11 recordsLinked to original sources

arXiv2026

NaviNote: Enabling In-situ Spatial Annotation Authoring to Support Exploration and Navigation for Blind and Low Vision People

GPS and smartphones enable users to place location-based annotations, capturing rich environmental context. Previous research demonstrates that blind and low vision (BLV) people can use annotations to explore unfamiliar areas. However, current commercial systems allowing BLV users to create annotations have never been evaluated, and current GPS-based systems can deviate several meters. Motivated by high-accuracy visual positioning technology, we first conducted a formative study with 24 BLV participants to envision a more accurate and inclusive annotation system. Surprisingly, many participants viewed the high-accuracy technology not just as an annotation system but also as a tool for precise last-few-meters navigation. Guided by participant feedback, we developed NaviNote, which combines vision-based high-precision localization with an agentic architecture to enable voice-based annotation authoring and navigation. Evaluating NaviNote with 18 BLV participants showed that it significantly improved navigation performance and supported users in understanding and annotating their surroundings. Based on these findings, we discuss design considerations for future accessible annotation authoring systems.

cs.HC↗

arXiv2025

ImaginateAR: AI-Assisted In-Situ Authoring in Augmented Reality

While augmented reality (AR) enables new ways to play, tell stories, and explore ideas rooted in the physical world, authoring personalized AR content remains difficult for non-experts, often requiring professional tools and time. Prior systems have explored AI-driven XR design but typically rely on manually defined VR environments and fixed asset libraries, limiting creative flexibility and real-world relevance. We introduce ImaginateAR, the first mobile tool for AI-assisted AR authoring to combine offline scene understanding, fast 3D asset generation, and LLMs -- enabling users to create outdoor scenes through natural language interaction. For example, saying "a dragon enjoying a campfire" (P7) prompts the system to generate and arrange relevant assets, which can then be refined manually. Our technical evaluation shows that our custom pipelines produce more accurate outdoor scene graphs and generate 3D meshes faster than prior methods. A three-part user study (N=20) revealed preferred roles for AI, how users create in freeform use, and design implications for future AR authoring tools. ImaginateAR takes a step toward empowering anyone to create AR experiences anywhere -- simply by speaking their imagination.

cs.HC↗

arXiv2025

AdjustAR: AI-Driven In-Situ Adjustment of Site-Specific Augmented Reality Content

Site-specific outdoor AR experiences are typically authored using static 3D models, but are deployed in physical environments that change over time. As a result, virtual content may become misaligned with its intended real-world referents, degrading user experience and compromising contextual interpretation. We present AdjustAR, a system that supports in-situ correction of AR content in dynamic environments using multimodal large language models (MLLMs). Given a composite image comprising the originally authored view and the current live user view from the same perspective, an MLLM detects contextual misalignments and proposes revised 2D placements for affected AR elements. These corrections are backprojected into 3D space to update the scene at runtime. By leveraging MLLMs for visual-semantic reasoning, this approach enables automated runtime corrections to maintain alignment with the authored intent as real-world target environments evolve.

cs.HC↗

arXiv2025

CoCreatAR: Enhancing Authoring of Outdoor Augmented Reality Experiences Through Asymmetric Collaboration

Authoring site-specific outdoor augmented reality (AR) experiences requires a nuanced understanding of real-world context to create immersive and relevant content. Existing ex-situ authoring tools typically rely on static 3D models to represent spatial information. However, in our formative study (n=25), we identified key limitations of this approach: models are often outdated, incomplete, or insufficient for capturing critical factors such as safety considerations, user flow, and dynamic environmental changes. These issues necessitate frequent on-site visits and additional iterations, making the authoring process more time-consuming and resource-intensive. To mitigate these challenges, we introduce CoCreatAR, an asymmetric collaborative mixed reality authoring system that integrates the flexibility of ex-situ workflows with the immediate contextual awareness of in-situ authoring. We conducted an exploratory study (n=32) comparing CoCreatAR to an asynchronous workflow baseline, finding that it enhances engagement, creativity, and confidence in the authored output while also providing preliminary insights into its impact on task load. We conclude by discussing the implications of our findings for integrating real-world context into site-specific AR authoring systems.

cs.HC↗

arXiv2023

What Do Children and Parents Want and Perceive in Conversational Agents? Towards Transparent, Trustworthy, Democratized Agents

Historically, researchers have focused on analyzing WEIRD, adult perspectives on technology. This means we may not have technology developed appropriately for children and those from non-WEIRD countries. In this paper, we analyze children and parents from various countries' perspectives on an emerging technology: conversational agents. We aim to better understand participants' trust of agents, partner models, and their ideas of "ideal future agents" such that researchers can better design for these users. Additionally, we empower children and parents to program their own agents through educational workshops, and present changes in perceptions as participants create and learn about agents. Results from the study (n=49) included how children felt agents were significantly more human-like, warm, and dependable than parents did, how participants trusted agents more than parents or friends for correct information, how children described their ideal agents as being more artificial than human-like than parents did, and how children tended to focus more on fun features, approachable/friendly features and addressing concerns through agent design than parents did, among other results. We also discuss potential agent design implications of the results, including how designers may be able to best foster appropriate levels of trust towards agents by focusing on designing agents' competence and predictability indicators, as well as increasing transparency in terms of agents' information sources.

cs.HC↗

arXiv2022

Learning Affects Trust: Design Recommendations and Concepts for Teaching Children -- and Nearly Anyone -- about Conversational Agents

Research has shown that human-agent relationships form in similar ways to human-human relationships. Since children do not have the same critical analysis skills as adults (and may over-trust technology, for example), this relationship-formation is concerning. Nonetheless, little research investigates children's perceptions of conversational agents in-depth, and even less investigates how education might change these perceptions. We present K-12 workshops with associated conversational AI concepts to encourage healthier understanding and relationships with agents. Through studies with the curriculum, and children and parents from various countries, we found participants' perceptions of agents -- specifically their partner models and trust -- changed. When participants discussed changes in trust of agents, we found they most often mentioned learning something. For example, they frequently mentioned learning where agents obtained information, what agents do with this information and how agents are programmed. Based on the results, we developed recommendations for teaching conversational agent concepts, including emphasizing the concepts students found most challenging, like training, turn-taking and terminology; supplementing agent development activities with related learning activities; fostering appropriate levels of trust towards agents; and fostering accurate partner models of agents. Through such pedagogy, students can learn to better understand conversational AI and what it means to have it in the world.

cs.HC↗

arXiv2021

"Alexa, Can I Program You?": Student Perceptions of Conversational Artificial Intelligence Before and After Programming Alexa

Growing up in an artificial intelligence-filled world, with Siri and Amazon Alexa often within arm's - or speech's - reach, could have significant impact on children. Conversational agents could influence how students anthropomorphize computer systems or develop a theory of mind. Previous research has explored how conversational agents are used and perceived by children within and outside of learning contexts. This study investigates how middle and high school students' perceptions of Alexa change through programming their own conversational agents in week-long AI education workshops. Specifically, we investigate the workshops' influence on student perceptions of Alexa's intelligence, friendliness, aliveness, safeness, trustworthiness, human-likeness, and feelings of closeness. We found that students felt Alexa was more intelligent and felt closer to Alexa after the workshops. We also found strong correlations between students' perceptions of Alexa's friendliness and trustworthiness, and safeness and trustworthiness. Finally, we explored how students tended to more frequently use computer science-related diction and ideas after the workshops. Based on our findings, we recommend designers carefully consider personification, transparency, playfulness and utility when designing CAs for learning contexts.

cs.HC↗

arXiv2020

Designing AI Learning Experiences for K-12: Emerging Works, Future Opportunities and a Design Framework

Artificial intelligence (AI) literacy is a rapidly growing research area and a critical addition to K-12 education. However, support for designing tools and curriculum to teach K-12 AI literacy is still limited. There is a need for additional interdisciplinary human-computer interaction and education research investigating (1) how general AI literacy is currently implemented in learning experiences and (2) what additional guidelines are required to teach AI literacy in specifically K-12 learning contexts. In this paper, we analyze a collection of K-12 AI and education literature to show how core competencies of AI literacy are applied successfully and organize them into an educator-friendly chart to enable educators to efficiently find appropriate resources for their classrooms. We also identify future opportunities and K-12 specific design guidelines, which we synthesized into a conceptual framework to support researchers, designers, and educators in creating K-12 AI learning experiences.

cs.CY↗

arXiv2020

Engaging Teachers to Co-Design Integrated AI Curriculum for K-12 Classrooms

Artificial Intelligence (AI) education is an increasingly popular topic area for K-12 teachers. However, little research has investigated how AI education can be designed to be more accessible to all learners. We organized co-design workshops with 15 K-12 teachers to identify opportunities to integrate AI education into core curriculum to leverage learners' interests. During the co-design workshops, teachers and researchers co-created lesson plans where AI concepts were embedded into various core subjects. We found that K-12 teachers need additional scaffolding in the curriculum to facilitate ethics and data discussions, and value supports for learner engagement, collaboration, and reflection. We identify opportunities for researchers and teachers to collaborate to make AI education more accessible, and present an exemplar lesson plan that shows entry points for teaching AI in non-computing subjects. We also reflect on co-designing with K-12 teachers in a remote setting.

physics.ed-ph↗

arXiv2020

Teaching Tech to Talk: K-12 Conversational Artificial Intelligence Literacy Curriculum and Development Tools

With children talking to smart-speakers, smart-phones and even smart-microwaves daily, it is increasingly important to educate students on how these agents work-from underlying mechanisms to societal implications. Researchers are developing tools and curriculum to teach K-12 students broadly about artificial intelligence (AI); however, few studies have evaluated these tools with respect to AI-specific learning outcomes, and even fewer have addressed student learning about AI-based conversational agents. We evaluate our Conversational Agent Interface for MIT App Inventor and workshop curriculum with respect to eight AI competencies from the literature. Furthermore, we analyze teacher (n=9) and student (n=47) feedback from workshops with the interface and recommend that future work leverages design considerations from the literature to optimize engagement, collaborates with teachers, and addresses a range of student abilities through pacing and opportunities for extension. We found students struggled most with the concepts of AI ethics and learning, and recommend emphasizing these topics when teaching. The appendix, including a demo video, can be found here: https://gist.github.com/jessvb/1cd959e32415a6ad4389761c49b54bbf

cs.CY↗

arXiv2020

Convo: What does conversational programming need? An exploration of machine learning interface design

Vast improvements in natural language understanding and speech recognition have paved the way for conversational interaction with computers. While conversational agents have often been used for short goal-oriented dialog, we know little about agents for developing computer programs. To explore the utility of natural language for programming, we conducted a study ($n$=45) comparing different input methods to a conversational programming system we developed. Participants completed novice and advanced tasks using voice-based, text-based, and voice-or-text-based systems. We found that users appreciated aspects of each system (e.g., voice-input efficiency, text-input precision) and that novice users were more optimistic about programming using voice-input than advanced users. Our results show that future conversational programming tools should be tailored to users' programming experience and allow users to choose their preferred input mode. To reduce cognitive load, future interfaces can incorporate visualizations and possess custom natural language understanding and speech recognition models for programming.

cs.HC↗