arXiv subjects

Christopher Clarke

Publications and source records attributed to Christopher Clarke.

15 recordsLinked to original sources

arXiv2026

Characterizing In-the-Wild Personal Listening Device Use to Inform Earable Application Design

Ear-worn devices are evolving from audio-playback tools into sensing platforms for health, interaction, and context-awareness. Yet, earable systems are typically designed and evaluated under strong assumptions about how long, how often, and in which situations people actually wear personal listening devices (PLDs). To ground these assumptions in-the-wild behavior, we combine a survey of 330 adults with multi-year, passively logged headphone audio-exposure records donated via Apple Health by 90 of them. We characterize where and when people use PLDs, how logged use has changed in recent years, and how psychological traits and social context associate with PLD usage. Our results show that logged mean daily use has increased from 37 minutes in 2020 to 64 minutes in 2024. Listening was intermittent: no listening was logged on 48% of participant-days in 2024, and sessions were fewer but longer on weekends. Sensation seeking, particularly disinhibition, showed small to medium positive associations with self-reported PLD use. Younger adults listened at higher volumes than the 25-34 group. High-volume exposure was uncommon, with only 4% of participant-weeks exceeding World Health Organization (WHO) safe-listening limits. Finally, most participants also reported avoiding PLD use in social situations. We translate these findings into implications for earable computing: realistic expectations of intermittent rather than continuous wear, contextual coverage that anticipates systematic gaps, targeted safe-listening interventions, and personalization grounded in psychosocial and demographic profiles rather than assumptions of uniform use.

cs.HC↗

arXiv2025

EarXplore: An Open Research Database on Earable Interaction

Interaction with sensor-augmented earphones, referred to as earables or hearables, represents a major area of earable research. Proximate to the head and reachable by hand, earables support diverse interactions and can detect multiple inputs simultaneously. Yet this diversity has fragmented research, complicating the tracking of developments. To address this, we introduce EarXplore, a curated, interactive online database on earable interaction research. Designed through a question-centered approach that guided the development of 33 criteria applied to annotate 118 studies and the structure of the platform, EarXplore comprises four integrated views: a Tabular View for structured exploration, a Graphical View for visual overviews, a Similarity View for conceptual links, and a Timeline View for scholarly trends. We demonstrate how the platform supports tailored exploration and filtering, and we leverage its capabilities to discuss gaps and opportunities. With community update mechanisms, EarXplore evolves with the field, serving as a living resource to accelerate future research

cs.HC↗

arXiv2025

earEOG via Periauricular Electrodes to Facilitate Eye Tracking in a Natural Headphone Form Factor

Eye tracking technology is frequently utilized to diagnose eye and neurological disorders, assess sleep and fatigue, study human visual perception, and enable novel gaze-based interaction methods. However, traditional eye tracking methodologies are constrained by bespoke hardware that is often cumbersome to wear, complex to apply, and demands substantial computational resources. To overcome these limitations, we investigated Electrooculography (EOG) eye tracking using 14 electrodes positioned around the ears, integrated into a custom-built headphone form factor device. In a controlled experiment, 16 participants tracked stimuli designed to induce smooth pursuits and saccades. Data analysis identified optimal electrode pairs for vertical and horizontal eye movement tracking, benchmarked against gold-standard EOG and camera-based methods. The electrode montage nearest the eyes yielded the best horizontal results. Horizontal smooth pursuits via earEOG showed high correlation with gold-standard measures ($r_{\mathrm{EOG}} = 0.81, p = 0.01$; $r_{\mathrm{CAM}} = 0.56, p = 0.02$), while vertical pursuits were weakly correlated ($r_{\mathrm{EOG}} = 0.28, p = 0.04$; $r_{\mathrm{CAM}} = 0.35, p = 0.05$). Voltage deflections when performing saccades showed strong correlation in the horizontal direction ($r_{\mathrm{left}} = 0.99, p = 0.0$; $r_{\mathrm{right}} = 0.99, p = 0.0$) but low correlation in the vertical direction ($r_{\mathrm{up}} = 0.6, p = 0.23$; $r_{\mathrm{down}} = 0.19, p = 0.73$). Overall, horizontal earEOG demonstrated strong performance, indicating its potential effectiveness, while vertical earEOG results were poor, suggesting limited feasibility in our current setup.

cs.HC↗

arXiv2025

SLMEval: Entropy-Based Calibration for Human-Aligned Evaluation of Large Language Models

The LLM-as-a-Judge paradigm offers a scalable, reference-free approach for evaluating language models. Although several calibration techniques have been proposed to better align these evaluators with human judgment, prior studies focus primarily on narrow, well-structured benchmarks. As a result, it remains unclear whether such calibrations generalize to real-world, open-ended tasks. In this work, we show that SOTA calibrated evaluators often fail in these settings, exhibiting weak or even negative correlation with human judgments. To address this, we propose SLMEval, a novel and efficient calibration method based on entropy maximization over a small amount of human preference data. By estimating a latent distribution over model quality and reweighting evaluator scores accordingly, SLMEval achieves strong correlation with human evaluations across two real-world production use cases and the public benchmark. For example, on one such task, SLMEval achieves a Spearman correlation of 0.57 with human judgments, while G-Eval yields a negative correlation. In addition, SLMEval reduces evaluation costs by 5-30x compared to GPT-4-based calibrated evaluators such as G-eval.

cs.CL↗

arXiv2024

Ranking Unraveled: Recipes for LLM Rankings in Head-to-Head AI Combat

Deciding which large language model (LLM) to use is a complex challenge. Pairwise ranking has emerged as a new method for evaluating human preferences for LLMs. This approach entails humans evaluating pairs of model outputs based on a predefined criterion. By collecting these comparisons, a ranking can be constructed using methods such as Elo. However, applying these algorithms as constructed in the context of LLM evaluation introduces several challenges. In this paper, we explore the effectiveness of ranking systems for head-to-head comparisons of LLMs. We formally define a set of fundamental principles for effective ranking and conduct a series of extensive evaluations on the robustness of several ranking algorithms in the context of LLMs. Our analysis uncovers key insights into the factors that affect ranking accuracy and efficiency, offering guidelines for selecting the most appropriate methods based on specific evaluation contexts and resource constraints.

cs.CL↗

arXiv2024

PEFT-U: Parameter-Efficient Fine-Tuning for User Personalization

The recent emergence of Large Language Models (LLMs) has heralded a new era of human-AI interaction. These sophisticated models, exemplified by Chat-GPT and its successors, have exhibited remarkable capabilities in language understanding. However, as these LLMs have undergone exponential growth, a crucial dimension that remains understudied is the personalization of these models. Large foundation models such as GPT-3 etc. focus on creating a universal model that serves a broad range of tasks and users. This approach emphasizes the model's generalization capabilities, treating users as a collective rather than as distinct individuals. While practical for many common applications, this one-size-fits-all approach often fails to address the rich tapestry of human diversity and individual needs. To explore this issue we introduce the PEFT-U Benchmark: a new dataset for building and evaluating NLP models for user personalization. \datasetname{} consists of a series of user-centered tasks containing diverse and individualized expressions where the preferences of users can potentially differ for the same input. Using PEFT-U, we explore the challenge of efficiently personalizing LLMs to accommodate user-specific preferences in the context of diverse user-centered tasks.

cs.CL↗

arXiv2024

MTP: A Meaning-Typed Language Abstraction for AI-Integrated Programming

Software development is shifting from traditional programming to AI-integrated applications that leverage generative AI and large language models (LLMs) during runtime. However, integrating LLMs remains complex, requiring developers to manually craft prompts and process outputs. Existing tools attempt to assist with prompt engineering, but often introduce additional complexity. This paper presents Meaning-Typed Programming (MTP), a novel paradigm that abstracts LLM integration through intuitive language-level constructs. By leveraging the inherent semantic richness of code, MTP automates prompt generation and response handling without additional developer effort. We introduce the (1) by operator for seamless LLM invocation, (2) MT-IR, a meaning-based intermediate representation for semantic extraction, and (3) MT-Runtime, an automated system for managing LLM interactions. We implement MTP in Jac, a programming language that supersets Python, and find that MTP significantly reduces coding complexity while maintaining accuracy and efficiency. MTP significantly reduces development complexity, lines of code modifications needed, and costs while improving run-time performance and maintaining or exceeding the accuracy of existing approaches. Our user study shows that developers using MTP completed tasks 3.2x faster with 45% fewer lines of code compared to existing frameworks. Moreover, MTP demonstrates resilience even when up to 50% of naming conventions are degraded, demonstrating robustness to suboptimal code. MTP is developed as part of the Jaseci open-source project, and is available under the module byLLM.

cs.PL↗

arXiv2024

Guylingo: The Republic of Guyana Creole Corpora

While major languages often enjoy substantial attention and resources, the linguistic diversity across the globe encompasses a multitude of smaller, indigenous, and regional languages that lack the same level of computational support. One such region is the Caribbean. While commonly labeled as "English speaking", the ex-British Caribbean region consists of a myriad of Creole languages thriving alongside English. In this paper, we present Guylingo: a comprehensive corpus designed for advancing NLP research in the domain of Creolese (Guyanese English-lexicon Creole), the most widely spoken language in the culturally rich nation of Guyana. We first outline our framework for gathering and digitizing this diverse corpus, inclusive of colloquial expressions, idioms, and regional variations in a low-resource language. We then demonstrate the challenges of training and evaluating NLP models for machine translation in Creole. Lastly, we discuss the unique opportunities presented by recent NLP advancements for accelerating the formal adoption of Creole languages as official languages in the Caribbean.

cs.CL↗

arXiv2024

One Agent Too Many: User Perspectives on Approaches to Multi-agent Conversational AI

Conversational agents have been gaining increasing popularity in recent years. Influenced by the widespread adoption of task-oriented agents such as Apple Siri and Amazon Alexa, these agents are being deployed into various applications to enhance user experience. Although these agents promote "ask me anything" functionality, they are typically built to focus on a single or finite set of expertise. Given that complex tasks often require more than one expertise, this results in the users needing to learn and adopt multiple agents. One approach to alleviate this is to abstract the orchestration of agents in the background. However, this removes the option of choice and flexibility, potentially harming the ability to complete tasks. In this paper, we explore these different interaction experiences (one agent for all) vs (user choice of agents) for conversational AI. We design prototypes for each, systematically evaluating their ability to facilitate task completion. Through a series of conducted user studies, we show that users have a significant preference for abstracting agent orchestration in both system usability and system performance. Additionally, we demonstrate that this mode of interaction is able to provide quality responses that are rated within 1% of human-selected answers.

cs.HC↗

arXiv2023

Rule By Example: Harnessing Logical Rules for Explainable Hate Speech Detection

Classic approaches to content moderation typically apply a rule-based heuristic approach to flag content. While rules are easily customizable and intuitive for humans to interpret, they are inherently fragile and lack the flexibility or robustness needed to moderate the vast amount of undesirable content found online today. Recent advances in deep learning have demonstrated the promise of using highly effective deep neural models to overcome these challenges. However, despite the improved performance, these data-driven models lack transparency and explainability, often leading to mistrust from everyday users and a lack of adoption by many platforms. In this paper, we present Rule By Example (RBE): a novel exemplar-based contrastive learning approach for learning from logical rules for the task of textual content moderation. RBE is capable of providing rule-grounded predictions, allowing for more explainable and customizable predictions compared to typical deep learning-based approaches. We demonstrate that our approach is capable of learning rich rule embedding representations using only a few data examples. Experimental results on 3 popular hate speech classification datasets show that RBE is able to outperform state-of-the-art deep learning classifiers as well as the use of rules in both supervised and unsupervised settings while providing explainable model predictions via rule-grounding.

cs.CL↗

arXiv2023

Label Agnostic Pre-training for Zero-shot Text Classification

Conventional approaches to text classification typically assume the existence of a fixed set of predefined labels to which a given text can be classified. However, in real-world applications, there exists an infinite label space for describing a given text. In addition, depending on the aspect (sentiment, topic, etc.) and domain of the text (finance, legal, etc.), the interpretation of the label can vary greatly. This makes the task of text classification, particularly in the zero-shot scenario, extremely challenging. In this paper, we investigate the task of zero-shot text classification with the aim of improving the ability of pre-trained language models (PLMs) to generalize to both seen and unseen data across varying aspects and domains. To solve this we introduce two new simple yet effective pre-training strategies, Implicit and Explicit pre-training. These methods inject aspect-level understanding into the model at train time with the goal of conditioning the model to build task-level understanding. To evaluate this, we construct and release UTCD, a new benchmark dataset for evaluating text classification in zero-shot settings. Experimental results on UTCD show that our approach achieves improved zero-shot generalization on a suite of challenging datasets across an array of zero-shot formalizations.

cs.CL↗

arXiv2022

One Agent To Rule Them All: Towards Multi-agent Conversational AI

The increasing volume of commercially available conversational agents (CAs) on the market has resulted in users being burdened with learning and adopting multiple agents to accomplish their tasks. Though prior work has explored supporting a multitude of domains within the design of a single agent, the interaction experience suffers due to the large action space of desired capabilities. To address these problems, we introduce a new task BBAI: Black-Box Agent Integration, focusing on combining the capabilities of multiple black-box CAs at scale. We explore two techniques: question agent pairing and question response pairing aimed at resolving this task. Leveraging these techniques, we design One For All (OFA), a scalable system that provides a unified interface to interact with multiple CAs. Additionally, we introduce MARS: Multi-Agent Response Selection, a new encoder model for question response pairing that jointly encodes user question and agent response pairs. We demonstrate that OFA is able to automatically and accurately integrate an ensemble of commercially available CAs spanning disparate domains. Specifically, using the MARS encoder we achieve the highest accuracy on our BBAI task, outperforming strong baselines.

cs.CL↗

arXiv2022

Towards Personalized Intelligence at Scale

Personalized Intelligence (PI) is the problem of providing customized AI experiences tailored to each individual user. In many applications, PI is preferred or even required. Existing personalization approaches involve fine-tuning pre-trained models to create new customized models. However, these approaches require a significant amount of computation to train, scaling with model size and the number of users, inhibiting PI to be realized widely. In this work, we introduce a novel model architecture and training/inference framework to enable Personalized Intelligence at scale. We achieve this by attaching a Personalization Head (PH) to pre-trained language models (LM). During training, the base LMs are frozen and only the parameters in PH are updated and are unique per user. This results in significantly smaller overall model sizes and training cost than traditional fine-tuning approaches when scaled across many users. We evaluate PHs on academia and industry-focused datasets and show that the PHs outperform zeroshot baseline in F1 score and are significantly more scalable than traditional fine-tuning approaches. We identify key factors required for effective PH design and training.

cs.CL↗

arXiv2019

An Evaluation Dataset for Intent Classification and Out-of-Scope Prediction

Task-oriented dialog systems need to know when a query falls outside their range of supported intents, but current text classification corpora only define label sets that cover every example. We introduce a new dataset that includes queries that are out-of-scope---i.e., queries that do not fall into any of the system's supported intents. This poses a new challenge because models cannot assume that every query at inference time belongs to a system-supported intent class. Our dataset also covers 150 intent classes over 10 domains, capturing the breadth that a production task-oriented agent must handle. We evaluate a range of benchmark classifiers on our dataset along with several different out-of-scope identification schemes. We find that while the classifiers perform well on in-scope intent classification, they struggle to identify out-of-scope queries. Our dataset and evaluation fill an important gap in the field, offering a way of more rigorously and realistically benchmarking text classification in task-driven dialog systems.

cs.CL↗

arXiv2011

Optimal Inverter VAR Control in Distribution Systems with High PV Penetration

The intent of the study detailed in this paper is to demonstrate the benefits of inverter var control on a fast timescale to mitigate rapid and large voltage fluctuations due to the high penetration of photovoltaic generation and the resulting reverse power flow. Our approach is to formulate the volt/var control as a radial optimal power flow (OPF) problem to minimize line losses and energy consumption, subject to constraints on voltage magnitudes. An efficient solution to the radial OPF problem is presented and used to study the structure of optimal inverter var injection and the net benefits, taking into account the additional cost of inverter losses when operating at non-unity power factor. This paper will illustrate how, depending on the circuit topology and its loading condition, the inverter's optimal reactive power injection is not necessarily monotone with respect to their real power output. The results are demonstrated on a distribution feeder on the Southern California Edison system that has a very light load and a 5 MW photovoltaic (PV) system installed away from the substation.

cs.ET↗