arXiv subjects

Mehrdad Shoeibi

Publications and source records attributed to Mehrdad Shoeibi.

4 recordsLinked to original sources

arXiv2026

Can We Trust In-Distribution Success? Locked Evaluation Reveals Transfer Failure and Sampling-Depth Entanglement in CRISPRi Perturbation Prediction

AI evaluation can support the wrong inference when an in-domain benchmark success does not survive distribution shift, or when the benchmark endpoint is entangled with a design factor. We study this problem in CRISPRi perturbation-effect prediction, evaluating a frozen Geneformer representation under a locked, pre-registered protocol: heads and model selection were frozen before test evaluation; the protocol required external outcome labels to remain withheld until final unblinding; and analysis-governing decisions were fixed before the evaluations they govern. In-distribution on the Virtual Cell Challenge (VCC), the frozen representation carries measurable predictive information beyond a dimension-matched random-feature control (Delta R^2 = +0.1645, 95% CI [+0.1375, +0.1920]), satisfying the pre-registered informativeness gate required before interpreting transfer. It then fails zero-shot transfer on both external screens (Spearman rho = -0.139 and -0.267), lying below that control on each. Adding a predefined magnitude block improves the representation externally (Delta rho = +0.032 and +0.143) but, under the frozen primary head, does not rescue transfer: both remain negative. A pre-registered, count-adjusted max-response secondary is positively associated with the outcome on both screens; we report it as correlational and secondary, not as a recovered magnitude signal. Finally, the VCC endpoint is strongly sample-size associated: a count-only linear model reaches R^2 = +0.4325, versus +0.2589 for the four magnitude scalars; adding those scalars to cell count improves R^2 by only +0.0017, so much of the aggregate-magnitude signal overlaps with cell count. This case study shows how locking the evaluation, harmonizing the measured endpoint, and separating primary from secondary evidence can change the inference supported by an AI benchmark.

cs.LG↗

arXiv2026

Pre-Registered External Evaluation Yields a Consistent Partial-Replication Category across Three Transcriptomic Foundation Models

Transcriptomic foundation models are increasingly used as reusable cell and gene representations, but validating them on new data under weak supervision and distribution shift is hard: standard comparisons conflate genuine representation signal with model capacity, row-identity artifacts, gains over strong task-specific baselines, and outcome rules chosen after seeing the test set. We introduce a pre-registered, final-test-once evaluation framework that locks the outcome rule, seeds, and target-gene-grouped splits before any test data are seen, and scores each frozen representation against a strong expression baseline, a matched-capacity Gaussian control, and a within-split row-identity (shuffle) control; only the per-cell embedding-extraction step is model-specific. Applying it to three architecturally distinct models-Geneformer, scGPT, and UCE-across two external Replogle Perturb-seq datasets (RPE1 and K562), all three clear the capacity and row-identity controls by a wide margin, yet none reliably beats the expression baseline: the strongest (Geneformer) exceeds it by at most about $0.03$ test $R^2$ and clears the pre-registered four-of-five-seed threshold in neither dataset, while scGPT and UCE fall below it. All three therefore land in the same pre-registered partial-replication category-a consistent cross-architecture outcome, even though the baseline-relative gap differs in sign and magnitude across models. These representations carry real structure beyond trivial controls but, under this weak magnitude label, do not transfer past a simple strong baseline; the locked framework is reusable for any frozen transcriptomic representation by swapping only the extraction step.

q-bio.OT↗

arXiv2026

Learning Stable Predictors from Weak Supervision under Distribution Shift

Learning from weak, proxy, or relative supervision is common when ground-truth labels are unavailable, but robustness under distribution shift remains poorly understood because the supervision mechanism itself may change across environments. We formalize this phenomenon as supervision drift, defined as changes in $P(y \mid x, c)$ across contexts, and study it in CRISPR-Cas13d transcriptomic perturbation experiments where guide efficacy is inferred indirectly from RNA-seq responses. Using publicly available data spanning two human cell lines and multiple post-induction timepoints, we construct a controlled non-IID benchmark with explicit domain (cell line) and temporal shifts, while reusing a fixed weak-label construction across all contexts to avoid changing targets. Across linear and tree-based models, weak supervision supports meaningful learning in-domain (ridge $R^2 = 0.356$, Spearman $\rho = 0.442$) and partial cross-cell-line transfer ($\rho \approx 0.40$). In contrast, temporal transfer collapses across all model classes considered, yielding negative $R^2$ and weak or near-zero $\rho$ (ridge $R^2 = -0.145$, $\rho = 0.008$; XGBoost $R^2 = -0.155$, $\rho = 0.056$; random forest $R^2 = -0.322$, $\rho = 0.139$). Additional robustness analyses using externally recomputed weak labels, shift-score quantification, and simple mitigation baselines preserve the same qualitative pattern. Feature-label association and feature-importance analyses remain relatively stable across cell lines but change sharply over time, indicating that failures arise from supervision drift rather than model capacity or simple covariate shift. These results show that strong in-domain performance under weak supervision can be misleading and motivate feature stability as a lightweight diagnostic for non-transferability before deployment.

cs.LG↗

arXiv2025

BIOGEN: Evidence-Grounded Multi-Agent Reasoning Framework for Transcriptomic Interpretation in Antimicrobial Resistance

Interpreting gene clusters from RNA sequencing (RNA-seq) remains challenging, especially in antimicrobial resistance studies where mechanistic insight is important for hypothesis generation. Existing pathway enrichment methods can summarize co-expressed modules, but they often provide limited cluster-specific explanations and weak connections to supporting literature. We present BIOGEN, an evidence-grounded multi-agent framework for post hoc interpretation of RNA-seq transcriptional modules. BIOGEN combines biomedical retrieval, structured reasoning, and multi-critic verification to generate traceable cluster-level explanations with explicit evidence and confidence labels. On a primary Salmonella enterica dataset, BIOGEN achieved strong biological grounding, including BERTScore 0.689, Semantic Alignment Score 0.715, KEGG Functional Similarity 0.342, and a hallucination rate of 0.000, compared with 0.100 for an LLM-only baseline. Across four additional bacterial RNA-seq datasets, BIOGEN also maintained zero hallucination under the same fixed pipeline. In comparisons with representative open-source agentic AI baselines, BIOGEN was the only framework that consistently preserved zero hallucination across all five datasets. These findings suggest that retrieval alone is not enough for reliable biological interpretation, and that evidence-grounded orchestration is important for transparent and source-traceable transcriptomic reasoning.

q-bio.QM↗