arXiv subjects

Joseph A. Konstan

Publications and source records attributed to Joseph A. Konstan.

13 recordsLinked to original sources

arXiv2026

Balancing Domestic and Global Perspectives: Evaluating Dual-Calibration and LLM-Generated Nudges for Diverse News Recommendation

In this study, we applied the ``personalized diversity nudge framework'' with the goal of expanding user reading coverage in terms of news locality (i.e., domestic and world news). We designed a novel topic-locality dual calibration algorithmic nudge and a large language model-based news personalization presentation nudge, then launched a 5-week real-user study with 120 U.S. news readers on the news recommendation experiment platform POPROX. With user interaction logs and survey responses, we found that algorithmic nudges can successfully increase exposure and consumption diversity, while the impact of LLM-based presentation nudges varied. User-level topic interest is a strong predictor of user clicks, while highlighting the relevance of news articles to prior read articles outperforms generic topic-based and no personalization. We also demonstrate that longitudinal exposure to calibrated news may shift readers' reading habits to value a balanced news digest from both domestic and world articles. Our results provide direction for future work on nudging for diverse consumption in news recommendation systems.

cs.IR↗

arXiv2025

Co-Authoring the Self: A Human-AI Interface for Interest Reflection in Recommenders

Natural language-based user profiles in recommender systems have been explored for their interpretability and potential to help users scrutinize and refine their interests, thereby improving recommendation quality. Building on this foundation, we introduce a human-AI collaborative profile for a movie recommender system that presents editable personalized interest summaries of a user's movie history. Unlike static profiles, this design invites users to directly inspect, modify, and reflect on the system's inferences. In an eight-week online field deployment with 1775 active movie recommender users, we find persistent gaps between user-perceived and system-inferred interests, show how the profile encourages engagement and reflection, and identify design directions for leveraging imperfect AI-powered user profiles to stimulate more user intervention and build more transparent and trustworthy recommender experiences.

cs.HC↗

arXiv2025

Beyond the Individual: A Community-Engaged Framework for Ethical Online Community Research

Online community research routinely poses minimal risk to individuals, but does the same hold true for online communities? In response to high-profile breaches of online community trust and increased debate in the social computing research community on the ethics of online community research, this paper investigates community-level harms and benefits of research. Through 9 participatory-inspired workshops with four critical online communities (Wikipedia, InTheRooms, CaringBridge, and r/AskHistorians) we found researchers should engage more directly with communities' primary purpose by rationalizing their methods and contributions in the context of community goals to equalize the beneficiaries of community research. To facilitate deeper alignment of these expectations, we present the FACTORS (Functions for Action with Communities: Teaching, Overseeing, Reciprocating, and Sustaining) framework for ethical online community research. Finally, we reflect on our findings by providing implications for researchers and online communities to identify and implement functions for navigating community-level harms and benefits.

cs.HC↗

arXiv2024

What Are We Optimizing For? A Human-centric Evaluation of Deep Learning-based Movie Recommenders

In the past decade, deep learning (DL) models have gained prominence for their exceptional accuracy on benchmark datasets in recommender systems (RecSys). However, their evaluation has primarily relied on offline metrics, overlooking direct user perception and experience. To address this gap, we conduct a human-centric evaluation case study of four leading DL-RecSys models in the movie domain. We test how different DL-RecSys models perform in personalized recommendation generation by conducting survey study with 445 real active users. We find some DL-RecSys models to be superior in recommending novel and unexpected items and weaker in diversity, trustworthiness, transparency, accuracy, and overall user satisfaction compared to classic collaborative filtering (CF) methods. To further explain the reasons behind the underperformance, we apply a comprehensive path analysis. We discover that the lack of diversity and too much serendipity from DL models can negatively impact the consequent perceived transparency and personalization of recommendations. Such a path ultimately leads to lower summative user satisfaction. Qualitatively, we confirm with real user quotes that accuracy plus at least one other attribute is necessary to ensure a good user experience, while their demands for transparency and trust can not be neglected. Based on our findings, we discuss future human-centric DL-RecSys design and optimization strategies.

cs.IR↗

arXiv2024

Large Language Models as Conversational Movie Recommenders: A User Study

This paper explores the effectiveness of using large language models (LLMs) for personalized movie recommendations from users' perspectives in an online field experiment. Our study involves a combination of between-subject prompt and historic consumption assessments, along with within-subject recommendation scenario evaluations. By examining conversation and survey response data from 160 active users, we find that LLMs offer strong recommendation explainability but lack overall personalization, diversity, and user trust. Our results also indicate that different personalized prompting techniques do not significantly affect user-perceived recommendation quality, but the number of movies a user has watched plays a more significant role. Furthermore, LLMs show a greater ability to recommend lesser-known or niche movies. Through qualitative analysis, we identify key conversational patterns linked to positive and negative user interaction experiences and conclude that providing personal context and examples is crucial for obtaining high-quality recommendations from LLMs.

cs.IR↗

arXiv2024

COVID-19 as Reflected in University President Bulk Email

E-mail ``Messages From the President'' to university students, staff, and faculty have long been used to keep campus communities aware of the latest policies, events, and news. But during the COVID-19 pandemic, as universities quickly closed facilities, sent students home, and canceled travel, these messages took on even greater importance. We report on a content analysis of bulk emails from different universities' presidents to their students and employees before and in three stages of the pandemic. We find that these messages change as universities move towards and through closure. During the pandemic, 1) presidential bulk emails tend to be more informative, positive, clearer than before; 2) they tend to use more personal and collective language; 3) university presidents tend to mention more local political leaders and fewer other university leaders. Our results can inform research on digital crisis communication and may be useful for researchers interested in automatically identifying crisis situations from communication streams.

cs.HC↗

arXiv2023

Interactive Content Diversity and User Exploration in Online Movie Recommenders: A Field Experiment

Recommender systems often struggle to strike a balance between matching users' tastes and providing unexpected recommendations. When recommendations are too narrow and fail to cover the full range of users' preferences, the system is perceived as useless. Conversely, when the system suggests too many items that users don't like, it is considered impersonal or ineffective. To better understand user sentiment about the breadth of recommendations given by a movie recommender, we conducted interviews and surveys and found out that many users considered narrow recommendations to be useful, while a smaller number explicitly wanted greater breadth. Additionally, we designed and ran an online field experiment with a larger user group, evaluating two new interfaces designed to provide users with greater access to broader recommendations. We looked at user preferences and behavior for two groups of users: those with higher initial movie diversity and those with lower diversity. Among our findings, we discovered that different level of exploration control and users' subjective preferences on interfaces are more predictive of their satisfaction with the recommender.

cs.HC↗

arXiv2023

Organizational Bulk Email Systems: Their Role and Performance in Remote Work

The COVID-19 pandemic has forced many employees to work from home. Organizational bulk emails now play a critical role to reach employees with central information in this work-from-home environment. However, we know from our own recent work that organizational bulk email has problems: recipients fail to retain the bulk messages they received from the organization; recipients and senders have different opinions on which bulk messages were important; and communicators lack technology support to better target and design messages. In this position paper, first we review the prior work on evaluating, designing, and prototyping organizational communication systems. Second we review our recent findings and some research techniques we found useful in studying organizational communication. Last we propose a research agenda to study organizational communications in remote work environment and suggest some key questions and potential study directions.

cs.HC↗

arXiv2023

Multi-Objective Personalization in Multi-Stakeholder Organizational Bulk E-mail: A Field Experiment

Bulk email is often used in organizations to communicate ``important-to-organization'' messages such as policy changes, organizational plans, and administrative updates. However, normal employees may prefer messages more relevant to their jobs or interests. Organizations face the challenge of balancing prioritizing the messages they prefer employees to know (tactical goals) while maintaining employees' positive experiences with these bulk emails, then they continue to read these emails in the future (strategic goals). Could personalization help organizations achieve these tactical and strategic goals? In an 8-week field experiment with a university newsletter, we implemented a 4x5x5 factorial design on personalizing subject lines, top news, and message order based on both the employees' and the organization's preferences. We measured these designs' influences on the open/interest/recognition/read-in-detail rate of the whole newsletter and the single messages within it. We found that ``important-to-organization'' messages only got higher recognition rates when being put on subject lines / top news (tactical goal). Mixing them with employee-preferred messages in top news did not bring further improvement to their own recognition rates but could improve the whole newsletter's recognition rate. Only when the top news solely contained the employee-preferred messages were the employees slightly more interested in the newsletter (strategic goal). We further analyze on which topics the employees and the organization's preferences conflicted. Finally, we discuss the design suggestions on personalization and recommendation for organizational bulk email.

cs.HC↗

arXiv2023

Less Can Be More: Exploring Population Rating Dispositions with Partitioned Models in Recommender Systems

In this study, we partition users by rating disposition - looking first at their percentage of negative ratings, and then at the general use of the rating scale. We hypothesize that users with different rating dispositions may use the recommender system differently and therefore the agreement with their past ratings may be less predictive of the future agreement. We use data from a large movie rating website to explore whether users should be grouped by disposition, focusing on identifying their various rating distributions that may hurt recommender effectiveness. We find that such partitioning not only improves computational efficiency but also improves top-k performance and predictive accuracy. Though such effects are largest for the user-based KNN CF, smaller for item-based KNN CF, and smallest for latent factor algorithms such as SVD.

cs.LG↗

arXiv2023

Getting the Most from Eye-Tracking: User-Interaction Based Reading Region Estimation Dataset and Models

A single digital newsletter usually contains many messages (regions). Users' reading time spent on, and read level (skip/skim/read-in-detail) of each message is important for platforms to understand their users' interests, personalize their contents, and make recommendations. Based on accurate but expensive-to-collect eyetracker-recorded data, we built models that predict per-region reading time based on easy-to-collect Javascript browser tracking data. With eye-tracking, we collected 200k ground-truth datapoints on participants reading news on browsers. Then we trained machine learning and deep learning models to predict message-level reading time based on user interactions like mouse position, scrolling, and clicking. We reached 27\% percentage error in reading time estimation with a two-tower neural network based on user interactions only, against the eye-tracking ground truth data, while the heuristic baselines have around 46\% percentage error. We also discovered the benefits of replacing per-session models with per-timestamp models, and adding user pattern features. We concluded with suggestions on developing message-level reading estimation techniques based on available data.

cs.HC↗

arXiv2021

Learning to Ignore: A Case Study of Organization-Wide Bulk Email Effectiveness

Bulk email is a primary communication channel within organizations, with all-company emails and regular newsletters serving as a mechanism for making employees aware of policies and events. Ineffective communication could result in wasted employee time and a lack of compliance or awareness. Previous studies on organizational emails focused mostly on recipients. However, organizational bulk email system is a multi-stakeholder problem including recipients, communicators, and the organization itself. We studied the effectiveness, practice, and assessments of the organizational bulk email system of a large university from multi-stakeholders' perspectives. We conducted a qualitative study with the university's communicators, recipients, and managers. We delved into the organizational bulk email's distributing mechanisms of the communicators, the reading behaviors of recipients, and the perspectives on emails' values of communicators, managers, and recipients. We found that the organizational bulk email system as a whole was strained, and communicators are caught in the middle of this multi-stakeholder problem. First, though the communicators had an interest in preserving the effectiveness of channels in reaching employees, they had high-level clients whose interests might outweigh judgment about whether a message deserves widespread circulation. Second, though communicators thought they were sending important information, recipients viewed most of the organizational bulk emails as not relevant to them. Third, this disagreement was amplified by the success metric used by communicators. They viewed their bulk emails as successful if they had a high open rate. But recipients often opened and then rapidly discarded emails without reading the details. Last, while the communicators in general understood the challenge, they had a limited set of targeting and feedback tools to support their task.

cs.HC↗

arXiv2019

Recommender Systems Notation: Proposed Common Notation for Teaching and Research

As the field of recommender systems has developed, authors have used a myriad of notations for describing the mathematical workings of recommendation algorithms. These notations ap-pear in research papers, books, lecture notes, blog posts, and software documentation. The dis-ciplinary diversity of the field has not contributed to consistency in notation; scholars whose home base is in information retrieval have different habits and expectations than those in ma-chine learning or human-computer interaction. In the course of years of teaching and research on recommender systems, we have seen the val-ue in adopting a consistent notation across our work. This has been particularly highlighted in our development of the Recommender Systems MOOC on Coursera (Konstan et al. 2015), as we need to explain a wide variety of algorithms and our learners are not well-served by changing notation between algorithms. In this paper, we describe the notation we have adopted in our work, along with its justification and some discussion of considered alternatives. We present this in hope that it will be useful to others writing and teaching about recommender systems. This notation has served us well for some time now, in research, online education, and traditional classroom instruction. We feel it is ready for broad use.

cs.IR↗