arXiv subjects

Thomas Finnie

Publications and source records attributed to Thomas Finnie.

4 recordsLinked to original sources

arXiv2025

Healthy LLMs? Benchmarking LLM Knowledge of UK Government Public Health Information

As Large Language Models (LLMs) become widely accessible, a detailed understanding of their knowledge within specific domains becomes necessary for successful real world use. This is particularly critical in the domains of medicine and public health, where failure to retrieve relevant, accurate, and current information could significantly impact UK residents. However, while there are a number of LLM benchmarks in the medical domain, currently little is known about LLM knowledge within the field of public health. To address this issue, this paper introduces a new benchmark, PubHealthBench, with over 8000 questions for evaluating LLMs' Multiple Choice Question Answering (MCQA) and free form responses to public health queries. To create PubHealthBench we extract free text from 687 current UK government guidance documents and implement an automated pipeline for generating MCQA samples. Assessing 24 LLMs on PubHealthBench we find the latest proprietary LLMs (GPT-4.5, GPT-4.1 and o1) have a high degree of knowledge, achieving >90% accuracy in the MCQA setup, and outperform humans with cursory search engine use. However, in the free form setup we see lower performance with no model scoring >75%. Therefore, while there are promising signs that state of the art (SOTA) LLMs are an increasingly accurate source of public health information, additional safeguards or tools may still be needed when providing free form responses.

cs.CL↗

arXiv2024

Real-time modelling of the SARS-CoV-2 pandemic in England 2020-2023: a challenging data integration

A central pillar of the UK's response to the SARS-CoV-2 pandemic was the provision of up-to-the moment nowcasts and short term projections to monitor current trends in transmission and associated healthcare burden. Here we present a detailed deconstruction of one of the 'real-time' models that was key contributor to this response, focussing on the model adaptations required over three pandemic years characterised by the imposition of lockdowns, mass vaccination campaigns and the emergence of new pandemic strains. The Bayesian model integrates an array of surveillance and other data sources including a novel approach to incorporating prevalence estimates from an unprecedented large-scale household survey. We present a full range of estimates of the epidemic history and the changing severity of the infection, quantify the impact of the vaccination programme and deconstruct contributing factors to the reproduction number. We further investigate the sensitivity of model-derived insights to the availability and timeliness of prevalence data, identifying its importance to the production of robust estimates.

stat.AP↗

arXiv2024

Evaluating Large Language Models for Public Health Classification and Extraction Tasks

Advances in Large Language Models (LLMs) have led to significant interest in their potential to support human experts across a range of domains, including public health. In this work we present automated evaluations of LLMs for public health tasks involving the classification and extraction of free text. We combine six externally annotated datasets with seven new internally annotated datasets to evaluate LLMs for processing text related to: health burden, epidemiological risk factors, and public health interventions. We evaluate eleven open-weight LLMs (7-123 billion parameters) across all tasks using zero-shot in-context learning. We find that Llama-3.3-70B-Instruct is the highest performing model, achieving the best results on 8/16 tasks (using micro-F1 scores). We see significant variation across tasks with all open-weight LLMs scoring below 60% micro-F1 on some challenging tasks, such as Contact Classification, while all LLMs achieve greater than 80% micro-F1 on others, such as GI Illness Classification. For a subset of 11 tasks, we also evaluate three GPT-4 and GPT-4o series models and find comparable results to Llama-3.3-70B-Instruct. Overall, based on these initial results we find promising signs that LLMs may be useful tools for public health experts to extract information from a wide variety of free text sources, and support public health surveillance, research, and interventions.

cs.CL↗

arXiv2020

Shut and re-open: the role of schools in the spread of COVID-19 in Europe

We investigate the effect of school closure and subsequent reopening on the transmission of COVID-19, by considering Denmark, Norway, Sweden, and German states as case studies. By comparing the growth rates in daily hospitalisations or confirmed cases under different interventions, we provide evidence that the effect of school closure is visible as a reduction in the growth rate approximately 9 days after implementation. Limited school attendance, such as older students sitting exams or the partial return of younger year groups, does not appear to significantly affect community transmission. A large-scale reopening of schools while controlling or suppressing the epidemic appears feasible in countries such as Denmark or Norway, where community transmission is generally low. However, school reopening can contribute to significant increases in the growth rate in countries like Germany, where community transmission is relatively high. Our findings underscore the need for a cautious evaluation of reopening strategies that ensure low classroom occupancy and a solid infrastructure to quickly identify and isolate new infections.

q-bio.PE↗