arXiv subjects

Yunge Wen

Publications and source records attributed to Yunge Wen.

14 recordsLinked to original sources

arXiv2026

Text-Driven Artistic Staging: 3D Posing, Lighting, and Camera References from Paintings

Artists coordinate human pose, illumination, and camera placement to convey narrative and emotion, but existing generative methods typically model these elements independently. We introduce text-to-editable 3D staging, a task that jointly generates human poses, a dominant light, and a camera configuration from an affective description. We construct 11,911 text--staging pairs from 2,328 figurative paintings by reconstructing SMPL bodies, estimating low-frequency illumination, recovering camera parameters, and pairing each scene with ArtEmis descriptions. We train a flow-matching transformer that supports variable numbers of figures and produces multiple staging alternatives for each prompt. On held-out descriptions, the model achieves 32.2\% retrieval R@1, compared with 16.6\% for CLIP-based nearest-neighbor retrieval, while approximately preserving corpus-level diversity. These results demonstrate the feasibility of generating editable, emotionally conditioned 3D staging references from text.

cs.CV↗

arXiv2026

NephoCodex: Exploring Bounded Material Agency in Weather Data Physicalization

Weather is a complex, continuously changing system in which uncertainty is intrinsic. Physicalizing this uncertainty introduces further variation because computational outputs cannot fully determine material behavior. We distinguish computational uncertainty from material variability and introduce bounded material agency: computation constrains material realization without fixing its exact appearance. We present NephoCodex, a data physicalization system informed by a formative study that constructs five artistic weather states and predicts probability distributions over them. Probability-weighted mappings translate these distributions into material control proposals, while entropy-based regulation, local sensing, and safety constraints bound their execution through mist, airflow, light, and transparent displays. A within-participant study found increased spatial presence and physical demand, while perceived data comprehensibility remained inconclusive after correction. These findings contribute to hybrid data physicalization by showing how variable material expression can be paired with stable digital annotations and how embodied experience can be evaluated separately from data comprehension.

cs.HC↗

arXiv2026

The Garden of Forking Paths: Threading Narrative Archetype as a Semantic Signal Through Gameplay Planning

Generative models can produce individual game facets, but whole-game generation remains an orchestration problem: narrative, level structure, encounters, objectives, rewards, and visuals must express shared intent. We present Forking Garden, a branching game generation system that uses narrative archetype as a persistent semantic signal across the generation pipeline. Narrative progression is represented as soft Rise/Fall states; candidate plot nodes are generated before structural constraints assemble archetype-conforming graphs. The same state then conditions encounter composition, objectives, rewards, and runtime difficulty adaptation, while a shared symbolic schema preserves narrative entities and gameplay configurations through content instantiation. Across 10 storylines, generated paths exhibit distinguishable archetypal trajectories, narrative threat predicts damage taken by a threat-blind combat agent, and generate-first-constrain-later yields 2.6 times the entity diversity of a hierarchical baseline. A 16-participant study further suggests that propagated Rise/Fall distinctions can remain meaningful during play, while also supporting narrative understanding and creator-oriented interpretation.

cs.HC↗

arXiv2026

PaintCopilot: Modeling Painting as Autonomous Artistic Continuation

Existing neural painting methods are target-driven: given a reference image, strokes are optimized to reconstruct it, fixing the outcome before painting begins. We instead ask whether a model can predict plausible painting actions when the final image is unknown, a problem we call autonomous artistic continuation. PaintCopilot realizes this through three lightweight, task-specific models: a Target Predictor that infers a provisional visual target directly from the evolving canvas, a Stroke Predictor that formulates brushstroke generation as flow matching to capture multiple plausible continuations, and a Region Sampler that generates stroke sequences within artist-specified regions via a conditional VAE, trained on a constructed dataset of 3000 portraits with stroke-level supervision. This lets AI autonomously act on an artwork whose direction is still open, through interruptible, revisable delegation mechanisms spanning individual strokes, extended sequences, bounded regions, and prior stroke history. A two-week deployment with ten artists suggests that creative agency was shaped not simply by how much work artists delegated, but by their ability to continually negotiate the direction, scope, and persistence of AI participation.

cs.CV↗

arXiv2026

Feelium: A Touchable Blimp Body for Aerial Telepresence

Floating things invite touch. We present Feelium, a blimp-based telepresence platform that enables visual embodiment and touch interaction through its inflatable skin. Through a VR headset, a remote person inhabits the blimp, looking out of it first-person, appearing on its skin as a face or avatar, and steering it through the room. Partners in the room pat it, press a palm against it, draw on it, or lean into it; the skin senses each contact, renders it into the wearer's view in VR spaces. Touch thus provides a physical interaction channel for remote presence, turning the skin into a shared surface between remote and co-located partners.

cs.HC↗

arXiv2026

SyncSpace: Layout-Conditioned 3D Gaussian Splatting for Space Reskinning in Mixed Reality

We present SyncSpace, a system that achieves both spatial alignment and visual consistency between a generated 3DGS world and physical space. We first scan the space via depth sensing to extract 3D bounding boxes, which we render into a layout-only panorama and feed as a geometric prior to a generative world model, producing a Gaussian splat scene in which objects are re-semantized to fit a target style without per-object control. We then align the generated scene to physical space with a coarse-to-fine registration algorithm, refined manually via pinch gestures when automatic registration does not converge. We demonstrate a hand-tracked engulfment interaction in which the virtual world rises to replace the physical space, and show a single space reskinned into multiple stylistically distinct worlds with its layout preserved.

cs.HC↗

arXiv2026

Whispering Water: Materializing Human-AI Dialogue as Interactive Ripples

Water has long served as a recipient of human confession across cultures. We present \textit{Whispering Water}, an interactive installation that materializes human-AI dialogue through cymatic patterns on water. Participants confess to a water surface, triggering a four-phase ritual: confession, contemplation, response, and release. Speech sentiment is translated into excitation frequencies that prime the water's physical state, while semantic content enters a multi-agent system of heterogeneous LLMs whose identities emerge through situated discourse. A novel algorithm decomposes synthesized speech into harmonic components via logarithmic spacing and Bark-scale mapping, reconstructing machine voices as physical wave superpositions. The installation explores emotional self-exploration through sensory-rich, ritually framed human-AI interaction.

cs.HC↗

arXiv2026

AromaGen: Interactive Generation of Rich Olfactory Experiences with Multimodal Language Models

Smell's deep connection with food, memory, and social experience has long motivated researchers to bring olfaction into interactive systems. Yet most olfactory interfaces remain limited to fixed scent cartridges and pre-defined generation patterns, and the scarcity of large-scale olfactory datasets has further constrained AI-based approaches. We present AromaGen, an AI-powered wearable interface capable of real-time, general-purpose aroma generation from free-form text or visual inputs. AromaGen is powered by a multimodal LLM that leverages latent olfactory knowledge to map semantic inputs to structured mixtures of 12 carefully selected base odorants, released through a neck-worn dispenser. Users can iteratively refine generated aromas through natural language feedback via in-context learning. Through a controlled user study ($N = 26$), AromaGen matches human-composed mixtures in zero-shot generation and significantly surpasses them after iterative refinement, achieving a median similarity of 8/10 to real food aromas and reducing perceived artificiality to levels comparable to real food. AromaGen is a step towards real-world interactive aroma generation, opening new possibilities for communication, wellbeing, and immersive technologies.

cs.HC↗

arXiv2026

SmellNet: A Large-scale Dataset for Real-world Smell Recognition

The ability of AI to sense and identify various substances based on their smell alone can have profound impacts on allergen detection (e.g. smelling gluten or peanuts in a cake), monitoring the manufacturing process, and sensing hormones that indicate emotional states, stress levels, and diseases. Despite these broad impacts, there are few standardized datasets, and therefore little progress, for training and evaluating AI systems' ability to `smell' in the real-world. In this paper, we use small gas and chemical sensors to create SmellNet, a comparatively large dataset for sensor-based machine olfaction that digitizes a diverse range of smells in the natural world. SmellNet contains about 828,000 time-series data points across 50 substances, spanning nuts, spices, herbs, fruits, and vegetables, and 43 mixtures among them with fixed ingredient volumetric ratios, with 68 hours of data collected. Using SmellNet, we developed ScentFormer, a Transformer-based architecture combining temporal differencing and sliding-window augmentation for smell data. For the SmellNet-Base classification tasks, ScentFormer achieves 63.3% Top-1 accuracy with GC-MS supervision, and for the SmellNet-Mixture distribution prediction tasks, ScentFormer achieves 50.2% Top-1@0.1 on the test-seen split. ScentFormer's ability to generalize across conditions and capture transient chemical dynamics demonstrates the promise of temporal modeling in sensor-based olfactory AI. SmellNet and ScentFormer lay the groundwork for sensor-based olfactory applications across healthcare, food and beverage, environmental monitoring, manufacturing, and entertainment.

cs.AI↗

arXiv2026

Smell with Genji: Rediscovering Human Perception through an Olfactory Game with AI

Olfaction plays an important role in human perception, yet its subjective and ephemeral nature makes it difficult to articulate, compare, and share across individuals. Traditional practices like the Japanese incense game Genji-ko offer one way to structure olfactory experience through shared interpretation. In this work, we present Smell with Genji, an AI-mediated olfactory interaction system that reinterprets Genji-ko as a collaborative human-AI sensory experience. By integrating a game setup, a mobile application, and an LLM-powered co-smelling partner equipped with olfactory sensing and LLM-based conversation, the system invites participants to compare scents and construct Genji-mon patterns, fostering reflection through a dialogue that highlights the alignment and discrepancies between human and machine perception. This work illustrates how sensing-enabled AI can participate in olfactory experience alongside users, pointing toward new possibilities for AI-supported sensory interaction and reflection in HCI.

cs.HC↗

arXiv2025

DuoZone: A User-Centric, LLM-Guided Mixed-Initiative XR Window Management System

Mixed reality (XR) environments offer vast spatial possibilities, but current window management systems require users to manually place, resize, and organize multiple applications across large 3D spaces. This creates cognitive and interaction burdens that limit productivity. We introduce DuoZone, a mixed-initiative XR window management system that combines user-defined spatial layouts with LLM-guided automation. DuoZone separates window management into two complementary zones. The Recommendation Zone enables fast setup by providing spatial layout templates and automatically recommending relevant applications based on user tasks and high-level goals expressed through voice or text. The Arrangement Zone supports precise refinement through direct manipulation, allowing users to adjust windows using natural spatial actions such as dragging, resizing, and snapping. Through this dual-zone approach, DuoZone promotes efficient organization while reducing user cognitive load. We conducted a user study comparing DuoZone with a baseline manual XR window manager. Results show that DuoZone improves task completion speed, reduces mental effort, and increases sense of control when working with multiple applications in XR. We discuss design implications for future mixed-initiative systems and outline opportunities for integrating adaptive, goal-aware intelligence into spatial computing workflows.

cs.HC↗

arXiv2025

ClassMind: Scaling Classroom Observation and Instructional Feedback with Multimodal AI

Classroom observation -- one of the most effective methods for teacher development -- remains limited due to high costs and a shortage of expert coaches. We present ClassMind, an AI-driven classroom observation system that integrates generative AI and multimodal learning to analyze classroom artifacts (e.g., class recordings) and deliver timely, personalized feedback aligned with pedagogical practices. At its core is AVA-Align, an agent framework that analyzes long classroom video recordings to generate temporally precise, best-practice-aligned feedback to support teacher reflection and improvement. Our three-phase study involved participatory co-design with educators, development of a full-stack system, and field testing with teachers at different stages of practice. Teachers highlighted the system's usefulness, ease of use, and novelty, while also raising concerns about privacy and the role of human judgment, motivating deeper exploration of future human--AI coaching partnerships. This work illustrates how multimodal AI can scale expert coaching and advance teacher development.

cs.HC↗

arXiv2025

All Stories Are One Story: Emotional Arc Guided Procedural Game Level Generation

The emotional arc is a universal narrative structure underlying stories across cultures and media -- an idea central to structuralist narratology, often encapsulated in the phrase "all stories are one story." We present a framework for procedural game narrative generation that incorporates emotional arcs as a structural backbone for both story progression and gameplay dynamics. Leveraging established narratological theories and large-scale empirical analyses, we focus on two core emotional patterns -- Rise and Fall -- to guide the generation of branching story graphs. Each story node is automatically populated with characters, items, and gameplay-relevant attributes (e.g., health, attack), with difficulty adjusted according to the emotional trajectory. Implemented in a prototype action role-playing game (ARPG), our system demonstrates how emotional arcs can be operationalized using large language models (LLMs) and adaptive entity generation. Evaluation through player ratings, interviews, and sentiment analysis shows that emotional arc integration significantly enhances engagement, narrative coherence, and emotional impact. These results highlight the potential of emotionally structured procedural generation for advancing interactive storytelling for games.

cs.AI↗

arXiv2025

"See What I Imagine, Imagine What I See": Human-AI Co-Creation System for 360$^\circ$ Panoramic Video Generation in VR

The emerging field of panoramic video generation from text and image prompts unlocks new creative possibilities in virtual reality (VR), addressing the limitations of current immersive experiences, which are constrained by pre-designed environments that restrict user creativity. To advance this frontier, we present Imagine360, a proof-of-concept prototype that integrates co-creation principles with AI agents. This system enables refined speech-based text prompts, egocentric perspective adjustments, and real-time customization of virtual surroundings based on user perception and intent. An eight-participant pilot study comparing non-AI and linear AI-driven workflows demonstrates that Imagine360's co-creative approach effectively integrates temporal and spatial creative controls. This introduces a transformative VR paradigm, allowing users to seamlessly transition between 'seeing' and 'imagining,' thereby shaping virtual reality through the creations of their minds.

cs.HC↗