Searcharxiv⌕ Search

arXiv · 2611.11796

Publications as benchmarks: auditing AI agents on spatial transcriptomics alignment

Abstract

AI agents are increasingly automating data analysis, but responsible adoption requires reliable performance evaluation, which is costly in specialized scientific domains. We introduce a framework that translates scientific publications into auditable benchmarks for AI agents and demonstrate its use on spatial transcriptomics alignment tasks. Across 121 tasks, 16 baseline methods, and six AI agents, the best agents (GPT-6-Astra and Gemini-4-Argon) matched or outperformed all baselines on 80% of tasks, often using surprisingly simple heuristics. Agent performance was robust to prompting and access to pre-installed packages. Benchmark construction also uncovered errors in prior work and shortcuts in current evaluation metrics. Finally, we show that agent-generated solutions can be distilled into a reusable program that ranks in the top 10% of baseline methods on unseen data. Our framework offers a scalable way to evaluate AI agents in specialized scientific domains by leveraging and stress-testing existing papers and metrics.

Explore related subjects

Keep this discovery

Explore connections, maps & timelines

BibTeXRIS

Yiqun T. Chen, Jan Matthias, Jianing Yao, Hongyu Eric Zhao, Stephanie C. Hicks. 2026-11-09. Publications as benchmarks: auditing AI agents on spatial transcriptomics alignment. https://searcharxiv.com/abs/2611.11796

Citations retain the original authors and DOI and include this site’s record URL. If this discovery service contributed to your research, you can also cite Searcharxiv by Indiciae: BibTeX · RIS.

KEEP EXPLORING

Related papers

ST-FateFlow: Continuous Gene-Spatial Reconstruction with Differentiation-Potential Learning from Time-Resolved Spatial Transcriptomics

Time-resolved spatial transcriptomics provides a powerful view of dynamic biological processes by measuring cellular states together with their spatial organization across time. However, destructive profiling and experimental cost restrict observations to a few discrete time points collected from independent samples, with no known cell pairing across time. This makes it difficult to reconstruct gene expression and tissue organization at unobserved stages and to characterize differentiation-associated dynamics. Here, we introduce ST-FateFlow, a flow-matching framework for continuous gene-spatial dynamics and differentiation-potential learning. ST-FateFlow jointly learns gene-expression and spatial-coordinate velocity fields, enabling interpolation of missing time points and continuous modeling of transcriptional evolution and tissue remodeling. In parallel, it independently learns a time-conditioned differentiation potential, whose values organize cell states along inferred differentiation progressions and support the analysis of differentiation-associated transcriptional changes. Across multiple time-resolved spatial transcriptomics datasets, ST-FateFlow effectively captures coordinated gene-expression and spatial dynamics. Case studies further show that the learned potential landscapes are consistent with known developmental and regenerative trajectories and help identify candidate genes associated with differentiation progression.

q-bio.GN↗

Ten simple rules for non-visual, reproducible and accessible bioinformatics

Bioinformatics workflows rely heavily on visual representations. Quality-control plots, cell embeddings, heatmaps, genome-browser tracks, and interactive dashboards are not merely illustrations, but instruments for making analytical decisions. For blind and low-vision researchers who use screen readers, braille displays, or audio-based interfaces, these create a barrier: the evidence used to justify an analysis is often encoded in visual form, while the underlying decision remains undocumented. We argue that non-visual accessibility and computational reproducibility are closely aligned, as they both require analyses to be transparent and to record why decisions were made. We present ten simple rules for non-visual bioinformatics, covering plots as decision records, cautious use of AI-generated figure descriptions, accessible computing environments, text-first literate programming, structured data and metadata, compact object summaries, accessible publication formats, collaboration practices, shared community infrastructure, and accessibility as part of FAIR research. The intended audience is computational biologists and developers. Using single-cell RNA-seq as a running example, we show that the accessible equivalent of a plot is a structured decision record. That is, a plot companion that goes beyond storing the underlying data by also stating the purpose of the analysis and the resulting quantitative evidence and uncertainty. We argue that treating accessibility in this way makes bioinformatics more inclusive and also more transparent and auditable.

q-bio.GN↗

How Private Are DNA Embeddings? Inverting Foundation Model Representations of Genomic Sequences

Genomic foundation models are increasingly used to share dense representations rather than raw DNA sequences. This is particularly pursued for mean-pooled representations that are later used for downstream prediction tasks. Whether this practice protects the privacy of the underlying sequence is largely unknown. We evaluated model-inversions against zero-shot embeddings from DNABERT-2, Evo 2, and Nucleotide Transformer v2 (NTv2). Substantial reconstruction remains possible: a single multi-length Length-Conditioned Query Decoder reconstructed NTv2 sequences with mean Levenshtein similarity from 0.98 at 10 nucleotides to 0.68 at 100, and Evo 2 sequences from nearly 1.00 at 15 nucleotides to 0.67 at 100. DNABERT-2 showed higher resistance (0.56 to 0.51), although it remained above the random baseline at short lengths. Therefore, we further evaluated, whether a linkage attack similar to Homer's attack is possible based on the mean-pooled representations of human DNA sequences. At SNP-enriched length-25 loci and known coordinates, Evo 2 and NTv2 reached 99\% identification at 32 loci and DNABERT-2 at 256 loci. With alignment-based coordinate recovery, the corresponding 99\% budgets were 64 and 128 loci for Evo 2 and NTv2, while DNABERT-2 did not reach the reported identification thresholds. Random loci required larger budgets. These findings show that mean-pooled embeddings can leak sequence and identity, with risk depending on sequence length, representation, locus informativeness, and coordinate information.

q-bio.GN↗