arXiv subjects

Shana O. Kelley

Publications and source records attributed to Shana O. Kelley.

4 recordsLinked to original sources

arXiv2026

BioEVAL: A global, multi-institutional benchmark of large language and multimodal models for bioengineering

Large Language Models (LLMs) have demonstrated historic breakthroughs in general reasoning with early successes in biomedical science. However, existing LLM benchmarking emphasizes factual recall, offering limited insight into model performance on frontier and multimodal tasks. We assembled BioEVAL (BioEngineering Validation of AI and LLMs), a global, multi-institutional initiative designed to assess experimental reasoning capability across bioengineering (BE) subfields. BioEVAL spans 11 major BE subfields plus a set of uncategorized items, bringing together 22 research groups to create a PhD-level benchmark comprising 608 evaluation items: 1) 380 multiple-choice questions (MCQs, 359 retained after audit), 2) 218 literature synthesis tasks, and 3) 10 multimodal problems with experimental image interpretation. Benchmark items underwent authoring-group expert review and centralized quality control before evaluation. Following evaluation, a blinded cross-group consensus audit of the highest- and lowest-accuracy MCQ items flagged 21 questions for revision or removal; these were withheld, and all reported MCQ results are computed on the 359 retained items. We evaluated diverse cloud-scale foundation/multimodal models (e.g., ChatGPT, Gemini, and Grok) and locally deployable models suitable for inference on consumer-grade GPUs. Models achieved the highest accuracy of up to 90% on MCQs, similarity score of 0.72 on literature synthesis, and accuracy of 80% on a small sample of multimodal reasoning questions, with substantial performance variation across subfields. Leaderboard rankings characterize current capabilities, limitations, and development priorities across the evaluated BE task categories. BioEVAL is maintained as an extensible benchmark with standardized protocols for continuing expert item contribution and model evaluation.

cs.AI↗

arXiv2025

A Roadmap for Predictive Human Immunology

For over a century, immunology has masterfully discovered and dissected the components of our immune system, yet its collective behavior remains fundamentally unpredictable. In this perspective, we argue that building on the learnings of reductionist biology and systems immunology, the field is poised for a third revolution. This new era will be driven by the convergence of purpose-built, large-scale causal experiments and predictive, generalizable AI models. Here, we propose the Predictive Immunology Loop as the unifying engine to harness this convergence. This closed loop iteratively uses AI to design maximally informative experiments and, in turn, leverages the resulting data to improve dynamic, in silico models of the human immune system across biological scales, culminating in a Virtual Immune System. This engine provides a natural roadmap for addressing immunology's grand challenges, from decoding molecular recognition to engineering tissue ecosystems. It also offers a framework to transform immunology from a descriptive discipline into one capable of forecasting and, ultimately, engineering human health.

q-bio.OT↗

arXiv2025

A path towards AI-scale, interoperable biological data

Biology is at the precipice of a new era where AI accelerates and amplifies the ability to study how cells operate, organize, and work as systems, revealing why disease happens and how to correct it. Organizations globally are prioritizing AI to accelerate basic research, drug discovery, personalized medicine, and synthetic biology. However, despite these opportunities, scientific data have proven a bottleneck, and progress has been slow and fragmented. Unless the scientific community takes a technology-led, community-focused approach to scaling and harnessing data, we will fail to capture this opportunity to drive new insights and biological discovery. The data bottleneck presents a unique paradox. It is increasingly simple to generate huge data volumes, thanks to expanding imaging datasets and plummeting sequencing costs, but scientists lack standards and tooling for large biological datasets, preventing integration into a multimodal foundational dataset that unlocks generalizable models of cellular and tissue function. This contradiction highlights two interrelated problems: abundant data that's difficult to manage, and a lack of data resources with necessary quality and utility to realize AI's potential in biology. Science must forge a collective approach enabling distributed contributions to combine into cohesive, powerful datasets transcending individual purposes. Here, we present a technological and data generation roadmap for scaling scientific impact. We outline AI's opportunity, mechanisms to scale data generation, the need for multi-modal measurements, and means to pool resources, standardize approaches, and collectively build the foundation enabling AI's full potential in biological discovery.

q-bio.OT↗

arXiv2024

How to Build the Virtual Cell with Artificial Intelligence: Priorities and Opportunities

The cell is arguably the most fundamental unit of life and is central to understanding biology. Accurate modeling of cells is important for this understanding as well as for determining the root causes of disease. Recent advances in artificial intelligence (AI), combined with the ability to generate large-scale experimental data, present novel opportunities to model cells. Here we propose a vision of leveraging advances in AI to construct virtual cells, high-fidelity simulations of cells and cellular systems under different conditions that are directly learned from biological data across measurements and scales. We discuss desired capabilities of such AI Virtual Cells, including generating universal representations of biological entities across scales, and facilitating interpretable in silico experiments to predict and understand their behavior using virtual instruments. We further address the challenges, opportunities and requirements to realize this vision including data needs, evaluation strategies, and community standards and engagement to ensure biological accuracy and broad utility. We envision a future where AI Virtual Cells help identify new drug targets, predict cellular responses to perturbations, as well as scale hypothesis exploration. With open science collaborations across the biomedical ecosystem that includes academia, philanthropy, and the biopharma and AI industries, a comprehensive predictive understanding of cell mechanisms and interactions has come into reach.

q-bio.QM↗