arXiv subjects

James Barry

Publications and source records attributed to James Barry.

At least 19 recordsLinked to original sources

arXiv2026

SemEval-2026 Task 7: Everyday Knowledge Across Diverse Languages and Cultures

We present our shared task on evaluating the adaptability of LLMs and NLP systems across multiple languages and cultures. The task data consist of an extended version of our manually constructed BLEnD benchmark (Myung et al. 2024), covering more than 30 language-culture pairs, predominantly representing low-resource languages spoken across multiple continents. As the task is designed strictly for evaluation, participants were not permitted to use the data for training, fine-tuning, few-shot learning, or any other form of model modification. Our task includes two tracks: (a) Short-Answer Questions (SAQ) and (b) Multiple-Choice Questions (MCQ). Participants were required to predict labels and were allowed to submit any NLP system and adopt diverse modelling strategies, provided that the benchmark was used solely for evaluation. The task attracted more than 140 registered participants, and we received final submissions from 62 teams, along with 19 system description papers. We report the results and present an analysis of the best-performing systems and the most commonly adopted approaches. Furthermore, we discuss shared insights into open questions and challenges related to evaluation, misalignment, and methodological perspectives on model behaviour in low-resource languages and for under-represented cultures.

cs.CL↗

arXiv2026

Monte Carlo conformal prediction for quantifying uncertainty in radio galaxy classification under ambiguous ground truth

Dramatically increasing data volumes are forcing astronomers to adopt automated methods for the identification and classification of astronomical objects. Although deep-learning models are often well-suited to this task, obtaining a measure of uncertainty on their predictions is challenging. Here we consider the suitability of Monte Carlo conformal prediction (MCCP) set size and confidence as measures of model uncertainty for the astronomical classification of radio galaxies. We demonstrate this approach using model predictions from a pre-trained radio galaxy foundation model, fine-tuned on a smaller set of labelled radio galaxies. We calibrate the MCCP by obtaining annotator-derived soft label distributions, i.e. probability distributions over classes instead of single class assignments, for each of these labelled radio galaxies and compare the resulting set sizes and confidence scores to predictive entropy measures for each galaxy obtained using a supervised Bayesian deep-learning model trained using Hamiltonian Monte Carlo (HMC). The comparison reveals only a weak correlation between the measures.

astro-ph.IM↗

arXiv2026

FactCorrector: A Graph-Inspired Approach to Long-Form Factuality Correction of Large Language Models

Large language models (LLMs) are widely used in knowledge-intensive applications but often generate factually incorrect responses. A promising approach to rectify these flaws is correcting LLMs using feedback. Therefore, in this paper, we introduce FactCorrector, a new post-hoc correction method that adapts across domains without retraining and leverages structured feedback about the factuality of the original response to generate a correction. To support rigorous evaluations of factuality correction methods, we also develop the VELI5 benchmark, a novel dataset containing systematically injected factual errors and ground-truth corrections. Experiments on VELI5 and several popular long-form factuality datasets show that the FactCorrector approach significantly improves factual precision while preserving relevance, outperforming strong baselines. We release our code at https://ibm.biz/factcorrector.

cs.CL↗

arXiv2025

Comprehensiveness Metrics for Automatic Evaluation of Factual Recall in Text Generation

Despite demonstrating remarkable performance across a wide range of tasks, large language models (LLMs) have also been found to frequently produce outputs that are incomplete or selectively omit key information. In sensitive domains, such omissions can result in significant harm comparable to that posed by factual inaccuracies, including hallucinations. In this study, we address the challenge of evaluating the comprehensiveness of LLM-generated texts, focusing on the detection of missing information or underrepresented viewpoints. We investigate three automated evaluation metrics: (1) an NLI-based method that decomposes texts into atomic statements and uses natural language inference (NLI) to identify missing facts, (2) a Q&A-based metric that extracts question-answer pairs and compares responses across sources, and (3) an end-to-end approach that directly identifies missing content using LLMs. Our experiments demonstrate the surprising effectiveness of the simple end-to-end metric compared to more complex metrics, though at the cost of reduced robustness, interpretability and result granularity. We further assess the comprehensiveness of responses from several popular open-weight LLMs when answering user queries based on multiple sources.

cs.CL↗

arXiv2025

Zero-Shot Open-Schema Entity Structure Discovery

Entity structure extraction, which aims to extract entities and their associated attribute-value structures from text, is an essential task for text understanding and knowledge graph construction. Existing methods based on large language models (LLMs) typically rely heavily on predefined entity attribute schemas or annotated datasets, often leading to incomplete extraction results. To address these challenges, we introduce Zero-Shot Open-schema Entity Structure Discovery (ZOES), a novel approach to entity structure extraction that does not require any schema or annotated samples. ZOES operates via a principled mechanism of enrichment, refinement, and unification, based on the insight that an entity and its associated structure are mutually reinforcing. Experiments demonstrate that ZOES consistently enhances LLMs' ability to extract more complete entity structures across three different domains, showcasing both the effectiveness and generalizability of the method. These findings suggest that such an enrichment, refinement, and unification mechanism may serve as a principled approach to improving the quality of LLM-based entity structure discovery in various scenarios.

cs.CL↗

arXiv2025

QGen Studio: An Adaptive Question-Answer Generation, Training and Evaluation Platform

We present QGen Studio: an adaptive question-answer generation, training, and evaluation platform. QGen Studio enables users to leverage large language models (LLMs) to create custom question-answer datasets and fine-tune models on this synthetic data. It features a dataset viewer and model explorer to streamline this process. The dataset viewer provides key metrics and visualizes the context from which the QA pairs are generated, offering insights into data quality. The model explorer supports model comparison, allowing users to contrast the performance of their trained LLMs against other models, supporting performance benchmarking and refinement. QGen Studio delivers an interactive, end-to-end solution for generating QA datasets and training scalable, domain-adaptable models. The studio will be open-sourced soon, allowing users to deploy it locally.

cs.CL↗

arXiv2024

KnowledgeHub: An end-to-end Tool for Assisted Scientific Discovery

This paper describes the KnowledgeHub tool, a scientific literature Information Extraction (IE) and Question Answering (QA) pipeline. This is achieved by supporting the ingestion of PDF documents that are converted to text and structured representations. An ontology can then be constructed where a user defines the types of entities and relationships they want to capture. A browser-based annotation tool enables annotating the contents of the PDF documents according to the ontology. Named Entity Recognition (NER) and Relation Classification (RC) models can be trained on the resulting annotations and can be used to annotate the unannotated portion of the documents. A knowledge graph is constructed from these entity and relation triples which can be queried to obtain insights from the data. Furthermore, we integrate a suite of Large Language Models (LLMs) that can be used for QA and summarisation that is grounded in the included documents via a retrieval component. KnowledgeHub is a unique tool that supports annotation, IE and QA, which gives the user full insight into the knowledge discovery pipeline.

cs.IR↗

arXiv2022

gaBERT -- an Irish Language Model

The BERT family of neural language models have become highly popular due to their ability to provide sequences of text with rich context-sensitive token encodings which are able to generalise well to many NLP tasks. We introduce gaBERT, a monolingual BERT model for the Irish language. We compare our gaBERT model to multilingual BERT and the monolingual Irish WikiBERT, and we show that gaBERT provides better representations for a downstream parsing task. We also show how different filtering criteria, vocabulary size and the choice of subword tokenisation model affect downstream performance. We compare the results of fine-tuning a gaBERT model with an mBERT model for the task of identifying verbal multiword expressions, and show that the fine-tuned gaBERT model also performs better at this task. We release gaBERT and related code to the community.

cs.CL↗

arXiv2021

The DCU-EPFL Enhanced Dependency Parser at the IWPT 2021 Shared Task

We describe the DCU-EPFL submission to the IWPT 2021 Shared Task on Parsing into Enhanced Universal Dependencies. The task involves parsing Enhanced UD graphs, which are an extension of the basic dependency trees designed to be more facilitative towards representing semantic structure. Evaluation is carried out on 29 treebanks in 17 languages and participants are required to parse the data from each language starting from raw strings. Our approach uses the Stanza pipeline to preprocess the text files, XLMRoBERTa to obtain contextualized token representations, and an edge-scoring and labeling model to predict the enhanced graph. Finally, we run a post-processing script to ensure all of our outputs are valid Enhanced UD graphs. Our system places 6th out of 9 participants with a coarse Enhanced Labeled Attachment Score (ELAS) of 83.57. We carry out additional post-deadline experiments which include using Trankit for pre-processing, XLM-RoBERTa-LARGE, treebank concatenation, and multitask learning between a basic and an enhanced dependency parser. All of these modifications improve our initial score and our final system has a coarse ELAS of 88.04.

cs.CL↗

arXiv2020

The ADAPT Enhanced Dependency Parser at the IWPT 2020 Shared Task

We describe the ADAPT system for the 2020 IWPT Shared Task on parsing enhanced Universal Dependencies in 17 languages. We implement a pipeline approach using UDPipe and UDPipe-future to provide initial levels of annotation. The enhanced dependency graph is either produced by a graph-based semantic dependency parser or is built from the basic tree using a small set of heuristics. Our results show that, for the majority of languages, a semantic dependency parser can be successfully applied to the task of parsing enhanced dependencies. Unfortunately, we did not ensure a connected graph as part of our pipeline approach and our competition submission relied on a last-minute fix to pass the validation script which harmed our official evaluation scores significantly. Our submission ranked eighth in the official evaluation with a macro-averaged coarse ELAS F1 of 67.23 and a treebank average of 67.49. We later implemented our own graph-connecting fix which resulted in a score of 79.53 (language average) or 79.76 (treebank average), which would have placed fourth in the competition evaluation.

cs.CL↗

arXiv2020

Treebank Embedding Vectors for Out-of-domain Dependency Parsing

A recent advance in monolingual dependency parsing is the idea of a treebank embedding vector, which allows all treebanks for a particular language to be used as training data while at the same time allowing the model to prefer training data from one treebank over others and to select the preferred treebank at test time. We build on this idea by 1) introducing a method to predict a treebank vector for sentences that do not come from a treebank used in training, and 2) exploring what happens when we move away from predefined treebank embedding vectors during test time and instead devise tailored interpolations. We show that 1) there are interpolated vectors that are superior to the predefined ones, and 2) treebank vectors can be predicted with sufficient accuracy, for nine out of ten test languages, to match the performance of an oracle approach that knows the most suitable predefined treebank embedding for the test set.

cs.CL↗

arXiv2019

Cross-lingual Parsing with Polyglot Training and Multi-treebank Learning: A Faroese Case Study

Cross-lingual dependency parsing involves transferring syntactic knowledge from one language to another. It is a crucial component for inducing dependency parsers in low-resource scenarios where no training data for a language exists. Using Faroese as the target language, we compare two approaches using annotation projection: first, projecting from multiple monolingual source models; second, projecting from a single polyglot model which is trained on the combination of all source languages. Furthermore, we reproduce multi-source projection (Tyers et al., 2018), in which dependency trees of multiple sources are combined. Finally, we apply multi-treebank modelling to the projected treebanks, in addition to or alternatively to polyglot modelling on the source side. We find that polyglot training on the source languages produces an overall trend of better results on the target language but the single best result for the target language is obtained by projecting from monolingual source parsing models and then training multi-treebank POS tagging and parsing models on the target side.

cs.CL↗

arXiv2019

Designing a Symbolic Intermediate Representation for Neural Surface Realization

Generated output from neural NLG systems often contain errors such as hallucination, repetition or contradiction. This work focuses on designing a symbolic intermediate representation to be used in multi-stage neural generation with the intention of reducing the frequency of failed outputs. We show that surface realization from this intermediate representation is of high quality and when the full system is applied to the E2E dataset it outperforms the winner of the E2E challenge. Furthermore, by breaking out the surface realization step from typically end-to-end neural systems, we also provide a framework for non-neural content selection and planning systems to potentially take advantage of semi-supervised pretraining of neural surface realization models.

cs.CL↗

arXiv2014

Sterile neutrinos and right-handed currents in KATRIN

Kurie-plot experiments allow for neutrino-mass measurements based on kinematics in an almost model-independent manner. A future tritium-based KATRIN-like experiment can be sensitive to light sterile neutrinos with masses below 18 keV, which are among the prime candidates for warm dark matter. Here we consider such keV neutrinos in left--right symmetric extensions, i.e. coupled to right-handed currents, which allow for an enhanced contribution to beta decay even for small active--sterile mixing, without violating astrophysical X-ray constraints. The modified spectral shape is in principle distinguishable from the standard contribution---especially for sterile neutrino masses below 9 keV, which can lead to a distinct peak. We compare the sensitivity to constraints from the LHC and neutrinoless double beta decay.

hep-ph↗

arXiv2013

Lepton number and flavour violation in TeV-scale left-right symmetric theories with large left-right mixing

The various diagrams leading to neutrinoless double beta decay in the left-right symmetric model have different relative magnitudes, depending on the scale of new physics. Neutrinos acquire mass from both type I and/or type II seesaw terms, making an unambiguous analysis difficult. We study the half-life for double beta decay in the case of type II and type I dominance, in the former case including interference terms. If the heavy neutrinos of the type I seesaw model are at the TeV scale, certain processes can be enhanced. In particular, there are regions of parameter space in which the so-called lambda- and eta-diagrams can give sizable contributions to the half-life for the decay. We perform a detailed study of one such scenario, paying careful attention to constraints from lepton flavour violation.

hep-ph↗

arXiv2012

Linear Collider Test of a Neutrinoless Double Beta Decay Mechanism in left-right Symmetric Theories

There are various diagrams leading to neutrinoless double beta decay in left-right symmetric theories based on the gauge group SU(2)_L x SU(2)_R. All can in principle be tested at a linear collider running in electron-electron mode. We argue that the so-called lambda-diagram is the most promising one. Taking the current limit on this diagram from double beta decay experiments, we evaluate the relevant cross section e e to W_L W_R, where W_L is the Standard Model W-boson and W_R the one from SU(2)_R. It is observable if the life-time of double beta decay and the mass of the W_R are close to current limits. Beam polarization effects and the high-energy behaviour of the cross section are also analyzed.

hep-ph↗

arXiv2012

Sterile Neutrinos for Warm Dark Matter and the Reactor Anomaly in Flavor Symmetry Models

We construct a flavor symmetry model based on the tetrahedral group A_4 in which the right-handed neutrinos from the seesaw mechanism can be both keV warm dark matter particles and eV-scale sterile neutrinos. This is achieved by giving the right-handed neutrinos appropriate charges under the same Froggatt-Nielsen symmetry responsible for the hierarchy of the charged lepton masses. We discuss the effect of next-to-leading order corrections to deviate the zeroth order tri-bimaximal mixing. Those corrections have two sources: (i) higher order seesaw terms, which are important when the seesaw particles are eV-scale, and (ii) higher-dimensional effective operators suppressed by additional powers of the cut-off scale of the theory. Whereas the mixing angles of the active neutrinos typically receive corrections of the same order, the mixing of the sterile neutrinos with the active ones is rather stable as it is connected with a hierarchy of mass scales. We also modify an effective A_4 model to incorporate keV-scale sterile neutrinos.

hep-ph↗

arXiv2011

Light Sterile Neutrinos: Models and Phenomenology

Motivated by recent hints in particle physics and cosmology, we study the realization of eV-scale sterile neutrinos within both the seesaw mechanism and flavor symmetry theories. We show that light sterile neutrinos can rather easily be accommodated in the popular A_4 flavor symmetry models. The exact tri-bimaximal mixing pattern is perturbed due to active-sterile mixing, which we discuss in detail for one example. In addition, we find an interesting extension of the type I seesaw, which can provide a natural origin for eV-scale sterile neutrinos as well as visible admixtures between sterile and active neutrinos. We also show that the presence of sterile neutrinos would significantly change the observables in neutrino experiments, specifically the oscillation probabilities in short-baseline experiments and the effective mass in neutrino-less double beta decay. The latter can prove particularly helpful to strengthen the case for eV-scale sterile neutrinos.

hep-ph↗