arXiv subjects

Seon Gyeom Kim

Publications and source records attributed to Seon Gyeom Kim.

7 recordsLinked to original sources

arXiv2026

AnnoSketch: Evaluating and Collecting Human Sketches for MLLM-assisted Chart Annotation

As multimodal large language models (MLLMs) support a growing range of input modalities, increasing work explores how to incorporate rough sketches to convey user intent. For annotated chart generation, it remains unclear what annotation sketches people provide and when such visual input helps MLLMs generate more useful annotations. In this study, we examine when sketch input is useful for MLLM-generated chart annotations across variation in chart type and caption type. In addition, we qualitatively analyze participants' explanations of their output preferences to characterize what made generated annotations more or less helpful. To further document participants' annotation sketches, we present AnnoSketch, comprising 1,600 annotation sketches collected across 160 chart-caption pairs from the conditions in which sketch guidance proved most beneficial, together with participants' annotation intents, perceived comprehension difficulty, and self-reported expressive limitations. We also label these sketches with structured metadata describing how each sketch relates to its caption and how participants express annotations through visual marks. Together, our study and AnnoSketch help determine when to solicit sketch input and provide empirical source for how people sketch chart annotations to support captions. The dataset and supplemental materials are available in our OSF repository.

cs.HC↗

arXiv2026

Student-ChatGPT Interaction Visible: Designing a Teacher Dashboard for EFL Writing Education

We present a Prompt Analytics Dashboard (PAD) for teachers that can traces student-LLM interactions from EFL writing classes. PAD can show student prompt-response exchanges with LLM chatbot and English essay writing revision histories to support data-informed instruction and visibility in classes. Through two iterative co-design sessions with six EFL instructors, we distilled a compact trace taxonomy (misuse signals, goal-alignment cues, revision effort) and instantiated three interface views (overview, week/outcome filter, drill-down with evidence snippets). This pipeline summarizes potential misuse and alignment at class/cohort levels and attaches micro-explanations to reduce over-surveillance. Instructors reported reduced scanning burden and clearer timing for interventions.

cs.HC↗

arXiv2026

Designing and Evaluating In-Vehicle Temporal Decoupling Pointing System for Selecting External Object

As In-Vehicle Infotainment Systems (IVIS) grow in complexity, selecting external points of interest (POIs) using traditional touchscreens significantly increases driver cognitive load. Recent evidence indicates that this visual-motor overload induces dangerous "hand-before-eye" behaviors, degrading primary driving tasks. To address this, we propose Point and Select, a novel in-vehicle interaction paradigm that introduces temporal decoupling to spatial gestures. By dividing the interaction into a rapid, ballistic spatial anchoring phase ("Rough Pointing") and a deferred, tactile confirmation phase ("Fine Selection"), our design aligns with the driver's cognitive-motor sequence. We evaluated this temporally decoupled approach in a high-fidelity driving simulator under urban speed conditions. Results indicate that Point and Select effectively minimizes perceived cognitive workload while seamlessly maintaining primary driving performance. This study demonstrates that decoupling spatial identification from confirmation successfully mitigates cognitive friction, offering a safer behavioral design strategy for non-autonomous driving environments.

cs.HC↗

arXiv2026

Evaluating Visual Prompts with Eye-Tracking Data for MLLM-Based Human Activity Recognition

Large Language Models (LLMs) have emerged as foundation models for IoT applications such as human activity recognition (HAR). However, directly applying high-frequency and multi-dimensional sensor data, such as eye-tracking data, leads to information loss and high token costs. To mitigate this, we investigate a visual prompting strategy that transforms sensor signals into data visualization images as an input to multimodal LLMs (MLLMs) using eye-tracking data. We conducted a systematic evaluation of MLLM-based HAR across three public eye-tracking datasets using three visualization types of timeline, heatmap, and scanpath, under varying temporal window sizes. Our findings suggest that visual prompting provides a token-efficient and scalable representation for eye-tracking data, highlighting its potential to enable MLLMs to effectively reason over high-frequency sensor signals in IoT contexts.

cs.HC↗

arXiv2025

Chart-to-Experience: Benchmarking Multimodal LLMs for Predicting Experiential Impact of Charts

The field of Multimodal Large Language Models (MLLMs) has made remarkable progress in visual understanding tasks, presenting a vast opportunity to predict the perceptual and emotional impact of charts. However, it also raises concerns, as many applications of LLMs are based on overgeneralized assumptions from a few examples, lacking sufficient validation of their performance and effectiveness. We introduce Chart-to-Experience, a benchmark dataset comprising 36 charts, evaluated by crowdsourced workers for their impact on seven experiential factors. Using the dataset as ground truth, we evaluated capabilities of state-of-the-art MLLMs on two tasks: direct prediction and pairwise comparison of charts. Our findings imply that MLLMs are not as sensitive as human evaluators when assessing individual charts, but are accurate and reliable in pairwise comparisons.

cs.HC↗

arXiv2025

Understanding the Impact of Spatial Immersion in Web Data Stories

An increasing number of web articles engage the reader with the feeling of being immersed in the data space. However, the exact characteristics of spatial immersion in the context of visual storytelling remain vague. For example, what are the common design patterns of data stories with spatial immersion? How do they affect the reader's experience? To gain a deeper understanding of the subject, we collected 23 distinct data stories with spatial immersion, and identified six design patterns, such as cinematic camera shots and transitions, intuitive data representations, realism, naturally moving elements, direct manipulation of camera or visualization, and dynamic dimension. Subsequently, we designed four data stories and conducted a crowdsourced user study comparing three design variations (static, animated, and immersive). Our results suggest that data stories with the design patterns for spatial immersion are more interesting and persuasive than static or animated ones, but no single condition was deemed more understandable or trustworthy.

cs.HC↗

arXiv2024

Optimizing Data Delivery: Insights from User Preferences on Visuals, Tables, and Text

In this work, we research user preferences to see a chart, table, or text given a question asked by the user. This enables us to understand when it is best to show a chart, table, or text to the user for the specific question. For this, we conduct a user study where users are shown a question and asked what they would prefer to see and used the data to establish that a user's personal traits does influence the data outputs that they prefer. Understanding how user characteristics impact a user's preferences is critical to creating data tools with a better user experience. Additionally, we investigate to what degree an LLM can be used to replicate a user's preference with and without user preference data. Overall, these findings have significant implications pertaining to the development of data tools and the replication of human preferences using LLMs. Furthermore, this work demonstrates the potential use of LLMs to replicate user preference data which has major implications for future user modeling and personalization research.

cs.HC↗