arXiv · 2611.13652
How Well Do Benchmarks Evaluate LLMs' Use for Personal Health Informatics?
Abstract
Benchmarking is a valuable technique for advancing the capabilities of Large Language Models (LLMs), including in personal health informatics. There are growing efforts on benchmark creation for personal health data activities. However, it is unclear whether and how well these efforts align with needs and considerations of personal health informatics process. Addressing this need, we first analyzed 53,587 tasks across eight benchmarks related to personal health informatics and characterized the space of benchmark design. We then contrasted our observations of design choices with personal informatics theories and empirical knowledge. We find significant mismatches between existing benchmark design and domain guidelines and understanding. This lack of alignment limits the utility of the evidence current benchmarks offer about LLM capabilities. We use domain knowledge to highlight capabilities benchmarks should include and discuss opportunities our work enables for taking a human-centered and domain-driven approach to benchmark design.
Explore related subjects
Keep this discovery
Explore connections, maps & timelines
Fatemeh Mirhosseini, Yasaman S. Sefidgar. 2026-11-10. How Well Do Benchmarks Evaluate LLMs' Use for Personal Health Informatics?. https://searcharxiv.com/abs/2611.13652
Citations retain the original authors and DOI and include this site’s record URL. If this discovery service contributed to your research, you can also cite Searcharxiv by Indiciae: BibTeX · RIS.