arXiv subjects

Guillaume Chabot-Couture

Publications and source records attributed to Guillaume Chabot-Couture.

3 recordsLinked to original sources

arXiv2026

From Guidelines to Guarantees: A Graph-Based Evaluation Harness for Domain-Specific Evaluation of LLMs

Rigorous evaluation of domain-specific language models requires benchmarks that are comprehensive, contamination-resistant, and maintainable. Static, manually curated datasets do not satisfy these properties. We present a graph-based evaluation harness that transforms structured clinical guidelines into a queryable knowledge graph and dynamically instantiates evaluation queries via graph traversal. The framework provides three guarantees: (1) complete coverage of guideline relationships; (2) surface-form contamination resistance through combinatorial variation; and (3) validity inherited from expert-authored graph structure. Applied to the WHO IMCI guidelines, the harness generates clinically grounded multiple-choice questions spanning symptom recognition, treatment, severity classification, and follow-up care. Evaluation across five language models reveals systematic capability gaps. Models perform well on symptom recognition but show lower accuracy on treatment protocols and clinical management decisions. The framework supports continuous regeneration of evaluation data as guidelines evolve and generalizes to domains with structured decision logic. This provides a scalable foundation for evaluation infrastructure.

cs.AI↗

arXiv2024

Bridging the Gap: Enhancing LLM Performance for Low-Resource African Languages with New Benchmarks, Fine-Tuning, and Cultural Adjustments

Large Language Models (LLMs) have shown remarkable performance across various tasks, yet significant disparities remain for non-English languages, and especially native African languages. This paper addresses these disparities by creating approximately 1 million human-translated words of new benchmark data in 8 low-resource African languages, covering a population of over 160 million speakers of: Amharic, Bambara, Igbo, Sepedi (Northern Sotho), Shona, Sesotho (Southern Sotho), Setswana, and Tsonga. Our benchmarks are translations of Winogrande and three sections of MMLU: college medicine, clinical knowledge, and virology. Using the translated benchmarks, we report previously unknown performance gaps between state-of-the-art (SOTA) LLMs in English and African languages. Finally, using results from over 400 fine-tuned models, we explore several methods to reduce the LLM performance gap, including high-quality dataset fine-tuning (using an LLM-as-an-Annotator), cross-lingual transfer, and cultural appropriateness adjustments. Key findings include average mono-lingual improvements of 5.6% with fine-tuning (with 5.4% average mono-lingual improvements when using high-quality data over low-quality data), 2.9% average gains from cross-lingual transfer, and a 3.0% out-of-the-box performance boost on culturally appropriate questions. The publicly available benchmarks, translations, and code from this study support further research and development aimed at creating more inclusive and effective language technologies.

cs.CL↗

arXiv2008

Demonstrating the model nature of the high-temperature superconductor HgBa$_2$CuO$_{4+Δ}$

The compound HgBa$_2$CuO$_{4+Δ}$ (Hg1201) exhibits a simple tetragonal crystal structure and the highest superconducting transition temperature (T$_c$) among all single Cu-O layer cuprates, with T$_c$ = 97 K (onset) at optimal doping. Due to a lack of sizable single crystals, experimental work on this very attractive system has been significantly limited. Thanks to a recent breakthrough in crystal growth, such crystals have now become available. Here, we demonstrate that it is possible to identify suitable heat treatment conditions to systematically and uniformly tune the hole concentration of Hg1201 crystals over a wide range, from very underdoped (T$_c$ = 47 K, hole concentration p ~ 0.08) to overdoped (T$_c$ = 64 K, p ~ 0.22). We then present quantitative magnetic susceptibility and DC charge transport results that reveal the very high-quality nature of the studied crystals. Using XPS on cleaved samples, we furthermore demonstrate that it is possible to obtain large surfaces of good quality. These characterization measurements demonstrate that Hg1201 should be viewed as a model high-temperature superconductor, and they provide the foundation for extensive future experimental work.

cond-mat.mtrl-sci↗