arXiv subjects

Sitong Wang

Publications and source records attributed to Sitong Wang.

At least 19 recordsLinked to original sources

arXiv2026

ZORO: Active Rules for Reliable Vibe Coding

Rules files (e.g., AGENTS\.md, CLAUDE\.md) are the primary mechanism for human-agent alignment when developers vibe code. However, they remain passive: it is not immediately apparent when rules are being used or followed, or how to improve them. To transform rules from passive text into active controls, we introduce ZORO, an interactive interface that integrates directly with a coding agent and anchors rules to every step of the coding process. After an agent generates an initial plan, ZORO enriches the plan with rules, enforces the rules during implementation by requiring the agent prove that each rule was followed, and allows users to provide in-situ feedback when they are unsatisfied with a rule application to evolve the ruleset. A technical evaluation shows that coding agents follow rules more with ZORO than without. A user study demonstrates a change in people's behavior and cognitive strategies when rules are at the forefront of vibe coding. We discuss how making rules active in agentic systems unlocks broader opportunities for human-agent alignment in coding settings and beyond.

cs.HC↗

arXiv2026

Rewriting Video: Text-Driven Reauthoring of Video Footage

Video is a powerful medium for communication and storytelling, yet reauthoring existing footage remains challenging. Even simple edits often demand expertise, time, and careful planning, constraining how creators envision and shape their narratives. Recent advances in generative AI suggest a new paradigm: what if editing a video were as straightforward as rewriting text? To investigate this, we present a tech probe and a study on text-driven video reauthoring. Our approach involves two technical contributions: (1) a generative reconstruction algorithm that reverse-engineers video into an editable text prompt, and (2) an interactive probe, Rewrite Kit, that allows creators to manipulate these prompts. A technical evaluation of the algorithm reveals a critical human-AI perceptual gap. A probe study with 12 creators surfaced novel use cases such as virtual reshooting, synthetic continuity, and aesthetic restyling. It also highlighted key tensions around coherence, control, and creative alignment in this new paradigm. Our work contributes empirical insights into the opportunities and challenges of text-driven video reauthoring, offering design implications for future co-creative video tools.

cs.HC↗

arXiv2025

DoubleAgents: Human-Agent Alignment in a Socially Embedded Workflow

Aligning agentic AI with user intent is critical for delegating complex, socially embedded tasks, yet user preferences are often implicit, evolving, and difficult to specify upfront. We present DoubleAgents, a system for human-agent alignment in coordination tasks, grounded in distributed cognition. DoubleAgents integrates three components: (1) a coordination agent that maintains state and proposes plans and actions, (2) a dashboard visualization that makes the agent's reasoning legible for user evaluation, and (3) a policy module that transforms user edits into reusable alignment artifacts, including coordination policies, email templates, and stop hooks, which improve system behavior over time. We evaluate DoubleAgents through a two-day lab study (n=10), three real-world deployments, and a technical evaluation. Participants' comfort in offloading tasks and reliance on DoubleAgents both increased over time, correlating with the three distributed cognition components. Participants still required control at points of uncertainty - edge-case flagging and context-dependent actions. We contribute a distributed cognition approach to human-agent alignment in socially embedded tasks.

cs.HC↗

arXiv2025

Facilitating Longitudinal Interaction Studies of AI Systems

UIST researchers develop tools to address user challenges. However, user interactions with AI evolve over time through learning, adaptation, and repurposing, making one time evaluations insufficient. Capturing these dynamics requires longer-term studies, but challenges in deployment, evaluation design, and data collection have made such longitudinal research difficult to implement. Our workshop aims to tackle these challenges and prepare researchers with practical strategies for longitudinal studies. The workshop includes a keynote, panel discussions, and interactive breakout groups for discussion and hands-on protocol design and tool prototyping sessions. We seek to foster a community around longitudinal system research and promote it as a more embraced method for designing, building, and evaluating UIST tools.

cs.HC↗

arXiv2025

Human-AI Schema Discovery and Application for Creative Problem Solving

Humans often rely on underlying structural patterns-schemas-to create, whether by writing stories, designing software, or composing music. Schemas help organize ideas and guide exploration, but they are often difficult to discover and apply, especially in complex or unfamiliar domains. My Ph.D. research develops a framework for human-AI schema discovery and application to support creative problem solving. I design systems that support users in sensemaking over examples to abstract schemas, and in operationalizing schemas into human-AI co-creative workflows for application. This research offers insights into how schema-guided interaction can make implicit knowledge more accessible and actionable, advancing more transparent and collaborative human-AI systems.

cs.HC↗

arXiv2025

TraGraph-GS: Trajectory Graph-based Gaussian Splatting for Arbitrary Large-Scale Scene Rendering

High-quality novel view synthesis for large-scale scenes presents a challenging dilemma in 3D computer vision. Existing methods typically partition large scenes into multiple regions, reconstruct a 3D representation using Gaussian splatting for each region, and eventually merge them for novel view rendering. They can accurately render specific scenes, yet they do not generalize effectively for two reasons: (1) rigid spatial partition techniques struggle with arbitrary camera trajectories, and (2) the merging of regions results in Gaussian overlap to distort texture details. To address these challenges, we propose TraGraph-GS, leveraging a trajectory graph to enable high-precision rendering for arbitrarily large-scale scenes. We present a spatial partitioning method for large-scale scenes based on graphs, which incorporates a regularization constraint to enhance the rendering of textures and distant objects, as well as a progressive rendering strategy to mitigate artifacts caused by Gaussian overlap. Experimental results demonstrate its superior performance both on four aerial and four ground datasets and highlight its remarkable efficiency: our method achieves an average improvement of 1.86 dB in PSNR on aerial datasets and 1.62 dB on ground datasets compared to state-of-the-art approaches.

cs.CV↗

arXiv2025

SSR-Zero: Simple Self-Rewarding Reinforcement Learning for Machine Translation

Large language models (LLMs) have recently demonstrated remarkable capabilities in machine translation (MT). However, most advanced MT-specific LLMs heavily rely on external supervision signals during training, such as human-annotated reference data or trained reward models (RMs), which are often expensive to obtain and challenging to scale. To overcome this limitation, we propose a Simple Self-Rewarding (SSR) Reinforcement Learning (RL) framework for MT that is reference-free, fully online, and relies solely on self-judging rewards. Training with SSR using 13K monolingual examples and Qwen-2.5-7B as the backbone, our model SSR-Zero-7B outperforms existing MT-specific LLMs, e.g., TowerInstruct-13B and GemmaX-28-9B, as well as larger general LLMs like Qwen2.5-32B-Instruct in English $\leftrightarrow$ Chinese translation tasks from WMT23, WMT24, and Flores200 benchmarks. Furthermore, by augmenting SSR with external supervision from COMET, our strongest model, SSR-X-Zero-7B, achieves state-of-the-art performance in English $\leftrightarrow$ Chinese translation, surpassing all existing open-source models under 72B parameters and even outperforming closed-source models. Our analysis highlights the effectiveness of the self-rewarding mechanism compared to the external LLM-as-a-judge approach in MT and demonstrates its complementary benefits when combined with trained RMs. Our findings provide valuable insight into the potential of self-improving RL methods. We have publicly released our code, data and models.

cs.CL↗

arXiv2025

Schemex: Discovering Structural Abstractions from Examples

Creative and communicative work is often underpinned by implicit structures, such as the Hero's Journey in storytelling, design patterns in software, or chord progressions in music. People often learn these structures from examples - a process known as schema induction. However, because schemas are abstract and implicit, they are difficult to discover: shared structural patterns are obscured by surface-level variation, and balancing generality with specificity is challenging. We present Schemex, an interactive AI workflow that systematically supports schema induction by decomposing it into three tractable stages: clustering examples, abstracting candidate schemas, and contrastively refining them by generating new instances and comparing against originals. Studies show that Schemex produces more actionable schemas than a frontier baseline without sacrificing generalizability, with participants uncovering deep and nuanced structural patterns. We also discuss design implications for the cognitive role of interactive process in structure discovery.

cs.HC↗

arXiv2025

Schemex: Discovering Design Patterns from Examples through Iterative Abstraction and Refinement

Expertise is often built by learning from examples. This process, known as schema induction, helps us identify patterns from examples. Despite its importance, schema induction remains a challenging cognitive task. Recent advances in generative AI reasoning capabilities offer new opportunities to support schema induction through human-AI collaboration. We present Schemex, an AI-powered workflow that enhances human schema induction through three stages: clustering, abstraction, and refinement via contrasting examples. We conducted an initial evaluation of Schemex through two real-world case studies: writing abstracts for HCI papers and creating news TikToks. Qualitative analysis demonstrates the high accuracy and usefulness of the generated schemas. We also discuss future work on developing more flexible methods for workflow construction to help humans focus on high-level thinking.

cs.HC↗

arXiv2025

The Role of Human Creativity in the Presence of AI Creativity Tools at Work: A Case Study on AI-Driven Content Transformation in Journalism

As AI becomes more capable, it is unclear how human creativity will remain essential in jobs that incorporate AI. We conducted a 14-week study of a student newsroom using an AI tool to convert web articles into social media videos. Most creators treated the tool as a creative springboard, not as a completion mechanism. They edited the AI outputs. The tool enabled the team to publish successful content that received over 500,000 views. Human creativity remained essential: after AI produced templated outputs, creators took ownership of the task, injecting their own creativity, especially when AI failed to create appropriate content. AI was initially seen as an authority, due to creators' lack of experience, but they ultimately learned to assert their own authority.

cs.HC↗

arXiv2024

JumpStarter: Human-AI Planning with Task-Structured Context Curation

Human-AI collaboration on complex planning goals is bottlenecked by how LLM interfaces handle context: users must manually curate and re-surface relevant information across long and unstructured chat histories. Despite advances in long-context prompting and memory-augmented retrieval, this burden remains unresolved: users still have to identify and supply the right context at each decision point, regardless of how much the model can store or surface. We propose JumpStarter, a system that enables LLMs to collaborate with humans on complex goals by dynamically decomposing tasks to help users manage context. We specifically introduce task-structured context curation, a framework that breaks down a user's goal into a hierarchy of actionable subtasks and scopes context to localized decision points, enabling finer-grained personalization and reuse. The framework is realized through three core mechanisms: context elicitation, selection, and reuse. In a within-subjects user study, plans produced with JumpStarter were rated substantially higher in quality than those produced with ChatGPT. A complementary automatic simulation study shows that JumpStarter consistently outperforms ChatGPT baselines, planning and memory agents, and workflow ablations. These findings show that effective human-AI planning depends not on the volume of context provided, but on attaching the right context to the right subtask at the right time.

cs.HC↗

arXiv2024

DynEx: Dynamic Code Synthesis with Structured Design Exploration for Accelerated Exploratory Programming

Recent advancements in large language models have significantly expedited the process of generating front-end code. This allows users to rapidly prototype user interfaces and ideate through code, a process known as exploratory programming. However, existing LLM code generation tools focus more on technical implementation details rather than finding the right design given a particular problem. We present DynEx, an LLM-based method for design exploration in accelerated exploratory programming. DynEx introduces a technique to explore the design space through a structured Design Matrix before creating the prototype with a modular, stepwise approach to LLM code generation. Code is generated sequentially, and users can test and approve each step before moving onto the next. A user study of 10 experts found that DynEx increased design exploration and enabled the creation of more complex and varied prototypes compared to a Claude Artifact baseline. We conclude with a discussion of the implications of design exploration for exploratory programming.

cs.HC↗

arXiv2024

Copying style, Extracting value: Illustrators' Perception of AI Style Transfer and its Impact on Creative Labor

Generative text-to-image models are disrupting the lives of creative professionals. Specifically, illustrators are threatened by models that claim to extract and reproduce their style. Yet, research on style transfer has rarely focused on their perspectives. We provided four illustrators with a model fine-tuned to their style and conducted semi-structured interviews about the model's successes, limitations, and potential uses. Evaluating their output, artists reported that style transfer successfully copies aesthetic fragments but is limited by content-style disentanglement and lacks the crucial emergent quality of their style. They also deemed the others' copies more successful. Understanding the results of style transfer as "boundary objects," we analyze how they can simultaneously be considered unsuccessful by artists and poised to replace their work by others. We connect our findings to critical HCI frameworks, demonstrating that style transfer, rather than merely a Creativity Support Tool, should also be understood as a supply chain optimization one.

cs.HC↗

arXiv2024

DIDUP: Dynamic Iterative Development for UI Prototyping

Large language models (LLMs) are remarkably good at writing code. A particularly valuable case of human-LLM collaboration is code-based UI prototyping, a method for creating interactive prototypes that allows users to view and fully engage with a user interface. We conduct a formative study of GPT Pilot, a leading LLM-generated code-prototyping system, and find that its inflexibility towards change once development has started leads to weaknesses in failure prevention and dynamic planning; it closely resembles the linear workflow of the waterfall model. We introduce DIDUP, a system for code-based UI prototyping that follows an iterative spiral model, which takes changes and iterations that come up during the development process into account. We propose three novel mechanisms for LLM-generated code-prototyping systems: (1) adaptive planning, where plans should be dynamic and reflect changes during implementation, (2) code injection, where the system should write a minimal amount of code and inject it instead of rewriting code so users have a better mental model of the code evolution, and (3) lightweight state management, a simplified version of source control so users can quickly revert to different working states. Together, this enables users to rapidly develop and iterate on prototypes.

cs.HC↗

arXiv2024

A Design Space for Intelligent and Interactive Writing Assistants

In our era of rapid technological advancement, the research landscape for writing assistants has become increasingly fragmented across various research communities. We seek to address this challenge by proposing a design space as a structured way to examine and explore the multidimensional space of intelligent and interactive writing assistants. Through a large community collaboration, we explore five aspects of writing assistants: task, user, technology, interaction, and ecosystem. Within each aspect, we define dimensions (i.e., fundamental components of an aspect) and codes (i.e., potential options for each dimension) by systematically reviewing 115 papers. Our design space aims to offer researchers and designers a practical tool to navigate, comprehend, and compare the various possibilities of writing assistants, and aid in the envisioning and design of new writing assistants.

cs.HC↗

arXiv2024

MoodSmith: Enabling Mood-Consistent Multimedia for AI-Generated Advocacy Campaigns

Emotion is vital to information and message processing, playing a key role in attitude formation. Consequently, creating a mood that evokes an emotional response is essential to any compelling piece of outreach communication. Many nonprofits and charities, despite having established messages, face challenges in creating advocacy campaign videos for social media. It requires significant creative and cognitive efforts to ensure that videos achieve the desired mood across multiple dimensions: script, visuals, and audio. We introduce MoodSmith, an AI-powered system that helps users explore mood possibilities for their message and create advocacy campaigns that are mood-consistent across dimensions. To achieve this, MoodSmith uses emotive language and plotlines for scripts, artistic style and color palette for visuals, and positivity and energy for audio. Our studies show that MoodSmith can effectively achieve a variety of moods, and the produced videos are consistent across media dimensions.

cs.HC↗

arXiv2024

ReelFramer: Human-AI Co-Creation for News-to-Video Translation

Short videos on social media are the dominant way young people consume content. News outlets aim to reach audiences through news reels -- short videos conveying news -- but struggle to translate traditional journalistic formats into short, entertaining videos. To translate news into social media reels, we support journalists in reframing the narrative. In literature, narrative framing is a high-level structure that shapes the overall presentation of a story. We identified three narrative framings for reels that adapt social media norms but preserve news value, each with a different balance of information and entertainment. We introduce ReelFramer, a human-AI co-creative system that helps journalists translate print articles into scripts and storyboards. ReelFramer supports exploring multiple narrative framings to find one appropriate to the story. AI suggests foundational narrative details, including characters, plot, setting, and key information. ReelFramer also supports visual framing; AI suggests character and visual detail designs before generating a full storyboard. Our studies show that narrative framing introduces the necessary diversity to translate various articles into reels, and establishing foundational details helps generate scripts that are more relevant and coherent. We also discuss the benefits of using narrative framing and foundational details in content retargeting.

cs.HC↗

arXiv2023

Tweetorial Hooks: Generative AI Tools to Motivate Science on Social Media

Communicating science and technology is essential for the public to understand and engage in a rapidly changing world. Tweetorials are an emerging phenomenon where experts explain STEM topics on social media in creative and engaging ways. However, STEM experts struggle to write an engaging "hook" in the first tweet that captures the reader's attention. We propose methods to use large language models (LLMs) to help users scaffold their process of writing a relatable hook for complex scientific topics. We demonstrate that LLMs can help writers find everyday experiences that are relatable and interesting to the public, avoid jargon, and spark curiosity. Our evaluation shows that the system reduces cognitive load and helps people write better hooks. Lastly, we discuss the importance of interactivity with LLMs to preserve the correctness, effectiveness, and authenticity of the writing.

cs.HC↗