Searcharxiv⌕ Search

arXiv subjects

Yiliu Tang

Publications and source records attributed to Yiliu Tang.

4 recordsLinked to original sources

RiskLens: Human-Informed Incident Transfer for AI Risk Evaluation

As AI capabilities advance, early-stage teams need better ways to judge whether historical incidents transfer to new products, demanding support beyond static risk checklists or one-off ideation tools. We present RiskLens, a human-AI workflow with an interactive system built around a live crawl, refreshable ranker, and extensible, human-informed schema. RiskLens analyzes 2,034 AI incidents and 10,443 AI products to rank transferable incidents using a feature schema spanning technical, operational, and user-facing attributes and supports an interface that presents ranked incident evidence, generated scenarios, risk metrics, and mitigation suggestions. In a study with 21 AI-familiar participants, users surfaced evaluation criteria that we added as 13 human-informed dimensions, including harm categories, failure mechanisms, and vulnerable groups. The feature-based model outperformed a generic baseline with an 86.2% win rate, and remained stable on a twenty-month future holdout; after human-informed schema expansion, the updated model outperformed the same baseline on 91.4% of pairs.

cs.HC↗

EvAlignUX: Advancing UX Evaluation through LLM-Supported Metrics Exploration

Evaluating UX in the context of AI's complexity, unpredictability, and generative nature presents unique challenges. How can we support HCI researchers to create comprehensive UX evaluation plans? In this paper, we introduce EvAlignUX, a system powered by large language models and grounded in scientific literature, designed to help HCI researchers explore evaluation metrics and their relationship to research outcomes. A user study with 19 HCI scholars showed that EvAlignUX improved the perceived quality and confidence in UX evaluation plans while prompting deeper consideration of research impact and risks. The system enhanced participants' thought processes, leading to the creation of a ``UX Question Bank'' to guide UX evaluation development. Findings also highlight how researchers' backgrounds influence their inspiration and concerns about AI over-reliance, pointing to future research on AI's role in fostering critical thinking. In a world where experience defines impact, we discuss the importance of shifting UX evaluation from a ``method-centric'' to a ``mindset-centric'' approach as the key to meaningful and lasting design evaluation.

cs.HC↗

CoQuest: Exploring Research Question Co-Creation with an LLM-based Agent

Developing novel research questions (RQs) often requires extensive literature reviews, especially in interdisciplinary fields. To support RQ development through human-AI co-creation, we leveraged Large Language Models (LLMs) to build an LLM-based agent system named CoQuest. We conducted an experiment with 20 HCI researchers to examine the impact of two interaction designs: breadth-first and depth-first RQ generation. The findings revealed that participants perceived the breadth-first approach as more creative and trustworthy upon task completion. Conversely, during the task, participants considered the depth-first generated RQs as more creative. Additionally, we discovered that AI processing delays allowed users to reflect on multiple RQs simultaneously, leading to a higher quantity of generated RQs and an enhanced sense of control. Our work makes both theoretical and practical contributions by proposing and evaluating a mental model for human-AI co-creation of RQs. We also address potential ethical issues, such as biases and over-reliance on AI, advocating for using the system to improve human research creativity rather than automating scientific inquiry.

cs.HC↗

UX Research on Conversational Human-AI Interaction: A Literature Review of the ACM Digital Library

Early conversational agents (CAs) focused on dyadic human-AI interaction between humans and the CAs, followed by the increasing popularity of polyadic human-AI interaction, in which CAs are designed to mediate human-human interactions. CAs for polyadic interactions are unique because they encompass hybrid social interactions, i.e., human-CA, human-to-human, and human-to-group behaviors. However, research on polyadic CAs is scattered across different fields, making it challenging to identify, compare, and accumulate existing knowledge. To promote the future design of CA systems, we conducted a literature review of ACM publications and identified a set of works that conducted UX (user experience) research. We qualitatively synthesized the effects of polyadic CAs into four aspects of human-human interactions, i.e., communication, engagement, connection, and relationship maintenance. Through a mixed-method analysis of the selected polyadic and dyadic CA studies, we developed a suite of evaluation measurements on the effects. Our findings show that designing with social boundaries, such as privacy, disclosure, and identification, is crucial for ethical polyadic CAs. Future research should also advance usability testing methods and trust-building guidelines for conversational AI.

cs.HC↗