Devyani Singh | Searcharxiv

arXiv2025

Radiology's Last Exam (RadLE): Benchmarking Frontier Multimodal AI Against Human Experts and a Taxonomy of Visual Reasoning Errors in Radiology

Generalist multimodal AI systems such as large language models (LLMs) and vision language models (VLMs) are increasingly accessed by clinicians and patients alike for medical image interpretation through widely available consumer-facing chatbots. Most evaluations claiming expert level performance are on public datasets containing common pathologies. Rigorous evaluation of frontier models on difficult diagnostic cases remains limited. We developed a pilot benchmark of 50 expert-level "spot diagnosis" cases across multiple imaging modalities to evaluate the performance of frontier AI models against board-certified radiologists and radiology trainees. To mirror real-world usage, the reasoning modes of five popular frontier AI models were tested through their native web interfaces, viz. OpenAI o3, OpenAI GPT-5, Gemini 2.5 Pro, Grok-4, and Claude Opus 4.1. Accuracy was scored by blinded experts, and reproducibility was assessed across three independent runs. GPT-5 was additionally evaluated across various reasoning modes. Reasoning quality errors were assessed and a taxonomy of visual reasoning errors was defined. Board-certified radiologists achieved the highest diagnostic accuracy (83%), outperforming trainees (45%) and all AI models (best performance shown by GPT-5: 30%). Reliability was substantial for GPT-5 and o3, moderate for Gemini 2.5 Pro and Grok-4, and poor for Claude Opus 4.1. These findings demonstrate that advanced frontier models fall far short of radiologists in challenging diagnostic cases. Our benchmark highlights the present limitations of generalist AI in medical imaging and cautions against unsupervised clinical use. We also provide a qualitative analysis of reasoning traces and propose a practical taxonomy of visual reasoning errors by AI models for better understanding their failure modes, informing evaluation standards and guiding more robust model development.

cs.AI↗

arXiv2024

Is methane the 'climate culprit'? The dangers of using imprecise, long-term GWP for methane to address the climate emergency

The United Nations Environmental Program's (UNEP) Emissions Gap Report, 2023, Temperatures hit new highs, yet world fails to cut emissions (again)'', and in 2024, No more hot air, emissions' massive gap between rhetoric and reality''. A climate emergency has been declared yet policies and emission reductions continue to fail. Global temperature anomalies in recent years have not been modelled well. Methane (CH4) is a potent greenhouse gas (GHG) with a short atmospheric half-life (~8.4 years), and a perturbation lifetime of 11.8 $\pm$ 1.8 yrs (IPCC AR6). It has a high, short-term impact on global warming: substantially greater than CO2. Traditional metrics such as the 100-year Global Warming Potential (GWP100) obscure the short-term, negative climatic effects of CH4, potentially leading to inadequate policy responses. This study examines the limitations of GWP100 in capturing the true, immediate climate impact of CH4 and its inability to incorporate varying emissions, explores alternative metrics, and discusses the multi-faceted implications of this under-reporting of CH4 emissions. Recalculation of 2024 Emissions Gap Report using a ten-year GWP of 105 increased CH4's warming effect to almost 90% of CO2, rather than 25% using a GWP100 of 28. We highlight the necessity of adopting a more immediate time horizon for CH4's warming effects, accelerating climate emergency action, while recognizing the adverse effects of the rapid growth rate of CH4 emissions on reduction efforts. To overcome the limitations of GWP100, a static constant, we propose GWPEFF(t) which dynamically represents warming across various time periods. It is a novel, physically realistic measure that is simple to understand, and effective for policies in reducing short-term emissions such as CH4.

physics.ao-ph↗