arXiv subjects

Michael S. Bernstein

Publications and source records attributed to Michael S. Bernstein.

At least 19 recordsLinked to original sources

arXiv2026

The Interface of Theseus: The Rise of Just-In-Time Interfaces

Our community creates interfaces---pre-constructed, static artifacts. These interfaces are effortful to create, so we invest great effort and care into each of them. Yet the equilibrium is shifting: we can now construct nearly any software, bespoke, within moments. When construction is cheap, we may soon live in a future in which interfaces are constructed in real time. This is a world of just-in-time interfaces: rapid, discardable, on-demand interfaces tailored to our needs in the moment. Will our community still create traditional interactive systems in this world? Or will we create interface generators, ``interface.md'' guides for AI consumption? I argue that we ought to lean into a future of rapid, bespoke, on-demand interfaces.

cs.HC↗

arXiv2026

Art Card Game (ACG): Embedding Illustration in Gameplay to Mitigate Artist Self-Criticism

Persistent self-criticism--harsh evaluative self-talk--can undermine illustrators' performance and well-being. Traditional interventions draw on psychotherapeutic approaches (e.g., compassion training) but sit outside the illustration workflow, requiring time, facilitation, and skill transfer. We propose an in-workflow alternative: evaluative off-centering, a mechanism redirecting self-critical evaluation away from an inherently self-evaluative task (like illustration) by embedding it in an alternative activity. We instantiate evaluative off-centering in Art Card Game (ACG) that integrates illustration into a card customization game: players illustrate cards that become playable assets in a head-to-head battle. In a four-day randomized controlled study with hobbyist and professional illustrators (N=38), ACG outperformed a control condition with identical illustration constraints but no evaluative off-centering mechanisms (e.g. multiplayer, gameplay), yielding significantly higher pride in produced artwork and activity enjoyment. Pride and enjoyment--positive affect states linked to lower self-criticism--help explain how ACG reduces self-criticism. We discuss design implications for creativity support tools that apply evaluative off-centering across creative domains.

cs.HC↗

arXiv2026

LLM Agents Grounded in Self-Reports Enable General-Purpose Simulation of Individuals

Machine learning can predict human behavior well when substantial structured data are available for well-defined outcomes. Such models are typically outcome-specific, however, requiring training data for each target outcome, limiting their applicability to new domains. We test whether large language models (LLMs) can relax these requirements by using self-report data to build attitudinal and behavioral simulations, or "generative agents," that can predict responses across outcomes without outcome-specific training data. Using data from a diverse national sample of 1,052 Americans, we built agents from (i) two-hour, semi-structured interviews elicited using the American Voices Project interview schedule, (ii) structured surveys including General Social Survey items and the Big Five personality inventory, or (iii) both sources combined. On held-out General Social Survey items, interview-only, survey-only, and combined agents achieved accuracies equal to 83%, 82%, and 86% of participants' own two-week test-retest consistency benchmark, respectively, compared with 74% for demographics-only agents. Combining interviews and surveys produced the highest accuracy, though gains over either source alone were modest, suggesting that predictive benefits from data begin to asymptote once the model has observed sufficient evidence within a domain. We find that these agents also predict personality traits, economic-game behavior, and experimental responses, while reducing accuracy disparities across racial and ideological groups relative to demographics-only agents. Together, these results show that LLM agents grounded in qualitative or quantitative self-reports can support general-purpose simulation of individuals across outcomes, without requiring task-specific training data.

cs.AI↗

arXiv2026

Behavior Latticing: Inferring User Motivations from Unstructured Interactions

A long-standing vision of computing is the personal AI system: one that understands us well enough to address our underlying needs. Today's AI focuses on what users do, ignoring why they might be doing such things in the first place. As a result, AI systems default to optimizing or repeating existing behaviors (e.g., user has ChatGPT complete their homework) even when they run counter to users' needs (e.g., gaining subject expertise). Instead we require systems that can make connections across observations, synthesizing them into insights about the motivations underlying these behaviors (e.g., user's ongoing commitments make it difficult to prioritize learning despite expressed desire to do so). We introduce an architecture for building user understanding through behavior latticing, connecting seemingly disparate behaviors, synthesizing them into insights, and repeating this process over long spans of interaction data. Doing so affords new capabilities, including being able to infer users' needs rather than just their tasks and connecting subtle patterns to produce conclusions that users themselves may not have previously realized. In an evaluation, we validate that behavior latticing produces accurate insights about the user with significantly greater interpretive depth compared to state-of-the-art approaches. To demonstrate the new interactive capabilities that behavior lattices afford, we instantiate a personal AI agent steered by user insights, finding that our agent is significantly better at addressing users' needs while still providing immediate utility.

cs.HC↗

arXiv2026

Comparing Design Metaphors and User-Driven Metaphors for Interaction Design

Metaphors enable designers to communicate their ideal user experience for platforms. Yet, we often do not know if these design metaphors match users' actual experiences. In this work, we compare design and user metaphors across three different platforms: ChatGPT, Twitter, and YouTube. We build on prior methods to elicit 554 user metaphors, as well as ratings on how well each metaphor describes users' experiences. We then identify 21 design metaphors by analyzing each platform's historical web presence since their launch date. We find that design metaphors often do not match the metaphors that users use to describe their experiences. Even when design and user metaphors do match, the metaphors do not always resonate universally. Through these findings, we highlight how comparing design and user metaphors can help to evaluate and refine metaphors for user experience.

cs.HC↗

arXiv2026

Mapping the Spiral of Silence: Surveying Unspoken Opinions in Online Communities

We often treat social media as a lens onto society. How might that lens distort the popularity of political and social viewpoints? We examine discrepancies between publicly posted and privately surveyed opinions within communities, contributing a measurement of the "spiral of silence" theory; the theory posits people are less likely to voice opinions when they believe they hold minority views, creating a reinforcing cycle where these opinions are expressed less. We surveyed members of politically-oriented Reddit communities about their willingness to post on contentious topics, yielding 439 responses across twelve subreddits. 72.1% of participants who perceive themselves in the minority remain silent and are half as likely to post compared to those who believe their opinion is in the majority. Community design factors, such as perceived diversity, are associated with less self-silencing. We provide recommendations for counteracting self-silencing at the community level (e.g., positive reinforcement, more transparent moderation). Overall, these results reveal gaps between online discourse and broader public opinion.

cs.SI↗

arXiv2026

Value Alignment of Social Media Ranking Algorithms

While social media feed rankings are primarily driven by engagement signals rather than any explicit value system, the resulting algorithmic feeds are not value-neutral: engagement may prioritize specific individualistic values. This paper presents an approach for social media feed value alignment. We adopt Schwartz's theory of Basic Human Values -- a broad set of human values that articulates complementary and opposing values forming the building blocks of many cultures -- and we implement an algorithmic approach that models and then ranks feeds by expressions of Schwartz's values in social media posts. Our approach enables controls where users can express weights on their desired values, combining these weights and post value expressions into a ranking that respects users' articulated trade-offs. Through controlled experiments (N=141 and N=250), we demonstrate that users can use these controls to architect feeds reflecting their desired values. Across users, value-ranked feeds align with personal values, diverging substantially from existing engagement-driven feeds.

cs.HC↗

arXiv2026

Learning Next Action Predictors from Human-Computer Interaction

Truly proactive AI systems must anticipate what we will do next. This foresight demands far richer information than the sparse signals we type into our prompts -- it demands reasoning over the entire context of what we see and do. We formalize this as next action prediction (NAP): given a sequence of a user's multimodal interactions with a computer (screenshots, clicks, sensor data), predict that user's next action. Progress on this task requires both new data and modeling approaches. To scale data, we annotate longitudinal, naturalistic computer use with vision-language models. We release an open-source pipeline for performing this labeling on private infrastructure, and label over 360K actions across one month of continuous phone usage from 20 users, amounting to 1,800 hours of screen time. We then introduce LongNAP, a user model that combines parametric and in-context learning to reason over long interaction histories. LongNAP is trained via policy gradient methods to generate user-specific reasoning traces given some context; retrieve relevant traces from a library of past traces; and then apply retrieved traces in-context to predict future actions. Using an LLM-as-judge evaluation metric (0-1 similarity to ground truth), LongNAP significantly outperforms supervised finetuning and prompted baselines on held-out data (by 79% and 39% respectively). Additionally, LongNAP generalizes to held out users when trained across individuals. The space of next actions a user might take at any moment is unbounded, spanning thousands of possible outcomes. Despite this, 17.1% of LongNAP's predicted trajectories are well-aligned with what a user does next (LLM-judge score $\geq$ 0.5). This rises to 26% when we filter to highly confident predictions. In sum, we argue that learning from the full context of user behavior to anticipate user needs is now a viable task with substantial opportunity.

cs.CL↗

arXiv2026

Just-In-Time Objectives: A General Approach for Specialized AI Interactions

Large language models promise a broad set of functions, but when not given a specific objective, they default to generic results. We demonstrate that inferring the user's in-the-moment objective, then rapidly optimizing for that singular objective, enables LLMs to produce specialized tools, interfaces, and responses. Our work introduces just-in-time objectives, which model a user's goals to specialize LLM systems on the fly. We contribute an architecture for automatically inducing such objectives by passively observing user behavior, then steering downstream AI systems through generation and evaluation against this objective. Inducing just-in-time objectives (e.g., "Clarify the abstract's research contribution") enables automatic generation of tools, e.g., those that critique a draft based on relevant HCI methodologies, anticipate related researchers' reactions, or surface ambiguous terminology. In a series of experiments on participants' own tasks, JIT objectives enable LLM outputs that achieve 66-86% win rates over typical LLMs. In-person use sessions confirm that JIT objectives produce specialized tools that are unique to each participant and are rated as significantly higher quality than a standard LLM chat tool.

cs.HC↗

arXiv2026

Social Media Feed Elicitation

Social media users have repeatedly advocated for control over the currently opaque operations of feed algorithms. Large language models (LLMs) now offer the promise of custom-defined feeds--but users often fail to foresee the gaps and edge cases in how they define their custom feed. We introduce feed elicitation interviews, an interactive method that guides users through identifying these gaps and articulating their preferences to better author custom social media feeds. We deploy this approach in an online study to create custom BlueSky feeds and find that participants significantly prefer the feeds produced from their elicited preferences to those produced by users manually describing their feeds. Through feed elicitation interviews, we advance users' ability to control their social media experience, empowering them to describe and implement their desired feeds.

cs.HC↗

arXiv2026

Reranking Social Media Feeds: A Practical Guide for Field Experiments

Social media plays a central role in shaping public opinion and behavior, yet performing experiments on these platforms and, in particular, on feed algorithms is becoming increasingly challenging. This guide offers practical recommendations for researchers developing and deploying field experiments focused on real-time reranking of social media feeds. The article is organized around two contributions. First, we provide an overview of an experimental method using web browser extensions that intercepts and reranks content in real time, enabling naturalistic reranking field experiments. We then describe feed interventions and measurements that this paradigm enables on participants' actual feeds, without requiring the involvement of social media platforms. Second, we offer concrete technical recommendations for intercepting and reranking social media feeds with minimal user-facing delay, and provide an open-source implementation. This document aims to summarize lessons learned in running field experiments on social media, provide concrete implementation details, and foster the ecosystem of independent social media research. Finally, we release the source code that serves as a blueprint for implementing future feed-ranking experiments.

cs.SI↗

arXiv2026

People Can Accurately Predict Behavior of Complex Algorithms That Are Available, Compact, and Aligned

Users trust algorithms more when they can predict the algorithms' behavior. Simple algorithms trivially yield predictively accurate mental models, but modern AI algorithms have often been assumed too complex for people to build predictive mental models, especially in the social media domain. In this paper, we describe conditions under which even complex algorithms can yield predictive mental models, opening up opportunities for a broader set of human-centered algorithms. We theorize that users will form an accurate predictive mental model of an algorithm's behavior if and only if the algorithm simultaneously satisfies three criteria: (1) cognitive availability of the underlying concepts being modeled, (2) concept compactness (does it form a single cognitive construct?), and (3) high alignment between the person's and algorithm's execution of the concept. We evaluate this theory through a pre-registered experiment (N=1250) where users predict behavior of 25 social media feed ranking algorithms that vary on these criteria. We find that even complex (e.g., LLM-based) algorithms enjoy accurate prediction rates when they meet all criteria, and even simple (e.g., basic term count) algorithms fail to be predictable when a single criterion fails. We also find that these criteria determine outcomes beyond prediction accuracy, such as which mental models users deploy to make their predictions.

cs.HC↗

arXiv2025

Finetuning LLMs for Human Behavior Prediction in Social Science Experiments

Large language models (LLMs) offer a powerful opportunity to simulate the results of social science experiments. In this work, we demonstrate that finetuning LLMs directly on individual-level responses from past experiments meaningfully improves the accuracy of such simulations across diverse social science domains. We construct SocSci210 via an automatic pipeline, a dataset comprising 2.9 million responses from 400,491 participants in 210 open-source social science experiments. Through finetuning, we achieve multiple levels of generalization. In completely unseen studies, our strongest model, Socrates-Qwen-14B, produces predictions that are 26% more aligned with distributions of human responses to diverse outcome questions under varying conditions relative to its base model (Qwen2.5-14B), outperforming GPT-4o by 13%. By finetuning on a subset of conditions in a study, generalization to new unseen conditions is particularly robust, improving by 71%. Since SocSci210 contains rich demographic information, we reduce demographic parity difference, a measure of bias, by 10.6% through finetuning. Because social sciences routinely generate rich, topic-specific datasets, our findings indicate that finetuning on such data could enable more accurate simulations for experimental hypothesis screening. We release our data, models and finetuning code at stanfordhci.github.io/socrates.

cs.LG↗

arXiv2025

Creating General User Models from Computer Use

Human-computer interaction has long imagined technology that understands us-from our preferences and habits, to the timing and purpose of our everyday actions. Yet current user models remain fragmented, narrowly tailored to specific apps, and incapable of the flexible reasoning required to fulfill these visions. This paper presents an architecture for a general user model (GUM) that learns about you by observing any interaction you have with your computer. The GUM takes as input any unstructured observation of a user (e.g., device screenshots) and constructs confidence-weighted propositions that capture user knowledge and preferences. GUMs can infer that a user is preparing for a wedding they're attending from messages with a friend. Or recognize that a user is struggling with a collaborator's feedback on a draft by observing multiple stalled edits and a switch to reading related work. GUMs introduce an architecture that infers new propositions about a user from multimodal observations, retrieves related propositions for context, and continuously revises existing propositions. To illustrate the breadth of applications that GUMs enable, we demonstrate how they augment chat-based assistants with context, manage OS notifications to selectively surface important information, and enable interactive agents that adapt to preferences across apps. We also instantiate proactive assistants (GUMBOs) that discover and execute useful suggestions on a user's behalf using their GUM. In our evaluations, we find that GUMs make calibrated and accurate inferences about users, and that assistants built on GUMs proactively identify and perform actions that users wouldn't think to request explicitly. Altogether, GUMs introduce methods that leverage multimodal models to understand unstructured context, enabling long-standing visions of HCI and entirely new interactive systems that anticipate user needs.

cs.HC↗

arXiv2025

Knoll: Creating a Knowledge Ecosystem for Large Language Models

Large language models are designed to encode general purpose knowledge about the world from Internet data. Yet, a wealth of information falls outside this scope -- ranging from personal preferences to organizational policies, from community-specific advice to up-to-date news -- that users want models to access but remains unavailable. In this paper, we propose a knowledge ecosystem in which end-users can create, curate, and configure custom knowledge modules that are utilized by language models, such as ChatGPT and Claude. To support this vision, we introduce Knoll, a software infrastructure that allows users to make modules by clipping content from the web or authoring shared documents on Google Docs and GitHub, add modules that others have made, and rely on the system to insert relevant knowledge when interacting with an LLM. We conduct a public deployment of Knoll reaching over 200 users who employed the system for a diverse set of tasks including personalized recommendations, advice-seeking, and writing assistance. In our evaluation, we validate that using Knoll improves the quality of generated responses.

cs.HC↗

arXiv2025

Burst: Collaborative Curation in Connected Social Media Communities

Positive social interactions can occur in groups of many shapes and sizes, spanning from small and private to large and open. However, social media tends to binarize our experiences into either isolated small groups or into large public squares. In this paper, we introduce Burst, a social media design that allows users to share and curate content between many spaces of varied size and composition. Users initially post content to small trusted groups, who can then burst that content, routing it to the groups that would be the best audience. We instantiate this approach into a mobile phone application, and demonstrate through a ten-day field study (N=36) that Burst enabled a participatory curation culture. With this work, we aim to articulate potential new design directions for social media sharing.

cs.HC↗

arXiv2025

Alexandria: A Library of Pluralistic Values for Realtime Re-Ranking of Social Media Feeds

Social media feed ranking algorithms fail when they too narrowly focus on engagement as their objective. The literature has asserted a wide variety of values that these algorithms should account for as well -- ranging from well-being to productive discourse -- far more than can be encapsulated by a single topic or theory. In response, we present a $\textit{library of values}$ for social media algorithms: a pluralistic set of 78 values as articulated across the literature, implemented into LLM-powered content classifiers that can be installed individually or in combination for real-time re-ranking of social media feeds. We investigate this approach by developing a browser extension, $\textit{Alexandria}$, that re-ranks the X/Twitter feed in real time based on the user's desired values. Through two user studies, both qualitative (N=12) and quantitative (N=257), we found that diverse user needs require a large library of values, enabling more nuanced preferences and greater user control. With this work, we argue that the values criticized as missing from social media ranking algorithms can be operationalized and deployed today through end-user tools.

cs.HC↗

arXiv2025

Aligning Language Models with Demonstrated Feedback

Language models are aligned to emulate the collective voice of many, resulting in outputs that align with no one in particular. Steering LLMs away from generic output is possible through supervised finetuning or RLHF, but requires prohibitively large datasets for new ad-hoc tasks. We argue that it is instead possible to align an LLM to a specific setting by leveraging a very small number (< 10) of demonstrations as feedback. Our method, Demonstration ITerated Task Optimization (DITTO), directly aligns language model outputs to a user's demonstrated behaviors. Derived using ideas from online imitation learning, DITTO cheaply generates online comparison data by treating users' demonstrations as preferred over output from the LLM and its intermediate checkpoints. Concretely, DITTO operates by having an LLM generate examples that are presumed to be inferior to expert demonstrations. The method iteratively constructs pairwise preference relationships between these LLM-generated samples and expert demonstrations, potentially including comparisons between different training checkpoints. These constructed preference pairs are then used to train the model using a preference optimization algorithm (e.g. DPO). We evaluate DITTO's ability to learn fine-grained style and task alignment across domains such as news articles, emails, and blog posts. Additionally, we conduct a user study soliciting a range of demonstrations from participants (N = 16). Across our benchmarks and user study, we find that win-rates for DITTO outperform few-shot prompting, supervised fine-tuning, and other self-play methods by an avg. of 19% points. By using demonstrations as feedback directly, DITTO offers a novel method for effective customization of LLMs.

cs.CL↗