arXiv subjects

Pierre-Andre Noel

Publications and source records attributed to Pierre-Andre Noel.

3 recordsLinked to original sources

arXiv2024

RepLiQA: A Question-Answering Dataset for Benchmarking LLMs on Unseen Reference Content

Large Language Models (LLMs) are trained on vast amounts of data, most of which is automatically scraped from the internet. This data includes encyclopedic documents that harbor a vast amount of general knowledge (e.g., Wikipedia) but also potentially overlap with benchmark datasets used for evaluating LLMs. Consequently, evaluating models on test splits that might have leaked into the training set is prone to misleading conclusions. To foster sound evaluation of language models, we introduce a new test dataset named RepLiQA, suited for question-answering and topic retrieval tasks. RepLiQA is a collection of five splits of test sets, four of which have not been released to the internet or exposed to LLM APIs prior to this publication. Each sample in RepLiQA comprises (1) a reference document crafted by a human annotator and depicting an imaginary scenario (e.g., a news article) absent from the internet; (2) a question about the document's topic; (3) a ground-truth answer derived directly from the information in the document; and (4) the paragraph extracted from the reference document containing the answer. As such, accurate answers can only be generated if a model can find relevant content within the provided document. We run a large-scale benchmark comprising several state-of-the-art LLMs to uncover differences in performance across models of various types and sizes in a context-conditional language modeling setting. Released splits of RepLiQA can be found here: https://huggingface.co/datasets/ServiceNow/repliqa.

cs.CL↗

arXiv2023

Constraining Representations Yields Models That Know What They Don't Know

A well-known failure mode of neural networks is that they may confidently return erroneous predictions. Such unsafe behaviour is particularly frequent when the use case slightly differs from the training context, and/or in the presence of an adversary. This work presents a novel direction to address these issues in a broad, general manner: imposing class-aware constraints on a model's internal activation patterns. Specifically, we assign to each class a unique, fixed, randomly-generated binary vector - hereafter called class code - and train the model so that its cross-depths activation patterns predict the appropriate class code according to the input sample's class. The resulting predictors are dubbed Total Activation Classifiers (TAC), and TACs may either be trained from scratch, or used with negligible cost as a thin add-on on top of a frozen, pre-trained neural network. The distance between a TAC's activation pattern and the closest valid code acts as an additional confidence score, besides the default unTAC'ed prediction head's. In the add-on case, the original neural network's inference head is completely unaffected (so its accuracy remains the same) but we now have the option to use TAC's own confidence and prediction when determining which course of action to take in an hypothetical production workflow. In particular, we show that TAC strictly improves the value derived from models allowed to reject/defer. We provide further empirical evidence that TAC works well on multiple types of architectures and data modalities and that it is at least as good as state-of-the-art alternative confidence scores derived from existing models.

cs.LG↗

arXiv2007

Time Evolution of Disease Spread on Networks with Degree Heterogeneity

Two crucial elements facilitate the understanding and control of communicable disease spread within a social setting. These components are, the underlying contact structure among individuals that determines the pattern of disease transmission; and the evolution of this pattern over time. Mathematical models of infectious diseases, which are in principle analytically tractable, use two general approaches to incorporate these elements. The first approach, generally known as compartmental modeling, addresses the time evolution of disease spread at the expense of simplifying the pattern of transmission. On the other hand, the second approach uses network theory to incorporate detailed information pertaining to the underlying contact structure among individuals. However, while providing accurate estimates on the final size of outbreaks/epidemics, this approach, in its current formalism, disregards the progression of time during outbreaks. So far, the only alternative that enables the integration of both aspects of disease spread simultaneously has been to abandon the analytical approach and rely on computer simulations. We offer a new analytical framework based on percolation theory, which incorporates both the complexity of contact network structure and the time progression of disease spread. Furthermore, we demonstrate that this framework is equally effective on finite- and "infinite"-size networks. Application of this formalism is not limited to disease spread; it can be equally applied to similar percolation phenomena on networks in other areas in science and technology.

q-bio.PE↗