arXiv subjects

Jessica Irons

Publications and source records attributed to Jessica Irons.

4 recordsLinked to original sources

arXiv2026

Using a Human-AI Teaming Approach to Create and Curate Scientific Datasets with the SCILIRE System

The rapid growth of scientific literature has made manual extraction of structured knowledge increasingly impractical. To address this challenge, we introduce SCILIRE, a system for creating datasets from scientific literature. SCILIRE has been designed around Human-AI teaming principles centred on workflows for verifying and curating data. It facilitates an iterative workflow in which researchers can review and correct AI outputs. Furthermore, this interaction is used as a feedback signal to improve future LLM-based inference. We evaluate our design using a combination of intrinsic benchmarking outcomes together with real-world case studies across multiple domains. The results demonstrate that SCILIRE improves extraction fidelity and facilitates efficient dataset creation.

cs.CL↗

arXiv2026

To Believe or Not To Believe: Comparing Supporting Information Tools to Aid Human Judgments of AI Veracity

With increasing awareness of the hallucination risks of generative artificial intelligence (AI), we see a growing shift toward providing information tooling to help users determine the veracity of AI-generated answers for themselves. User responsibility for assessing veracity is particularly critical for certain sectors that rely on on-demand, AI-generated data extraction, such as biomedical research and the legal sector. While prior work offers us a variety of ways in which systems can provide such support, there is a lack of empirical evidence on how this information is actually incorporated into the user's decision-making process. Our user study takes a step toward filling this knowledge gap. In the context of a generative AI data extraction tool, we examine the relationship between the type of supporting information (full source text, passage retrieval, and Large Language Model (LLM) explanations) and user behavior in the veracity assessment process, examined through the lens of efficiency, effectiveness, reliance and trust. We find that passage retrieval offers a reasonable compromise between accuracy and speed, with judgments of veracity comparable to using the full source text. LLM explanations, while also enabling rapid assessments, fostered inappropriate reliance and trust on the data extraction AI, such that participants were less likely to detect errors. In additiona, we analyzed the impacts of the complexity of the information need, finding preliminary evidence that inappropriate reliance is worse for complex answers. We demonstrate how, through rigorous user evaluation, we can better develop systems that allow for effective and responsible human agency in veracity assessment processes.

cs.HC↗

arXiv2025

A Conceptual Framework for Human-AI Collaborative Genome Annotation

Genome annotation is essential for understanding the functional elements within genomes. While automated methods are indispensable for processing large-scale genomic data, they often face challenges in accurately predicting gene structures and functions. Consequently, manual curation by domain experts remains crucial for validating and refining these predictions. These combined outcomes from automated tools and manual curation highlight the importance of integrating human expertise with AI capabilities to improve both the accuracy and efficiency of genome annotation. However, the manual curation process is inherently labor-intensive and time-consuming, making it difficult to scale for large datasets. To address these challenges, we propose a conceptual framework, Human-AI Collaborative Genome Annotation (HAICoGA), which leverages the synergistic partnership between humans and artificial intelligence to enhance human capabilities and accelerate the genome annotation process. Additionally, we explore the potential of integrating Large Language Models (LLMs) into this framework to support and augment specific tasks. Finally, we discuss emerging challenges and outline open research questions to guide further exploration in this area.

q-bio.GN↗

arXiv2024

Towards a criteria-based approach to selecting human-AI interaction mode

Artificial intelligence (AI) tools are now prevalent in many knowledge work industries. As AI becomes more capable and interactive, there is a growing need for guidance on how to employ AI most effectively. The A2C framework (Tariq, Chhetri, Nepal & Paris, 2024) distinguishes three decision-making modes for engaging AI: automation (AI completes a task, including decision/action), augmentation (AI supports human to decide) and collaboration (iterative interaction between human and AI). However, selecting the appropriate mode for a specific application is not always straightforward. The goal of the present study was to compile and trial a simple set of criteria to support recommendations about appropriate A2C mode for a given application. Drawing on human factors and computer science literature, we identified key criteria related to elements of the task, impacts on worker and support needs. From these criteria we built a scoring rubric with recommendation for A2C mode. As a preliminary test of this approach, we applied the criteria to cognitive task analysis (CTA) outputs from three tasks in the science domain - genome annotation, biological collections curation and protein crystallization - which provided insights into worker decision points, challenges and expert strategies. This paper describes the method for connecting CTA to A2C, reflecting on the challenges and future directions.

cs.HC↗