arXiv subjects

Ian Harris

Publications and source records attributed to Ian Harris.

10 recordsLinked to original sources

arXiv2026

ConceptGuard: Benchmarking Context-Sensitive Unlearning in Large Language Models

Large Language Models (LLMs) increasingly require selective removal of harmful or sensitive knowledge, called unlearning, yet existing methods and benchmarks fail to evaluate this capability completely. Current approaches rely on disjoint forget and retain sets composed of independent facts, and measure success using simple and direct factual recall. This framing fails to capture a key requirement of unlearning, namely the ability to eliminate harmful behaviors while preserving benign and beneficial knowledge. We argue that effective unlearning must operate at the level of concepts, ensuring complete removal of unsafe applications while maintaining their correct and useful usage, thereby achieving conceptually meaningful and complete unlearning. To better evaluate unlearning techniques from such a practical viewpoint, we introduce the notion of dual-use concepts: concepts that can be used in both harmful and benign contexts. Building on these concepts, we construct a benchmark called ConceptGuard where forget and retain sets are explicitly complementary in concept usage. Our benchmark uniquely enables unlearning to be explored and gauged at the level of concepts, instead of sparse facts, and evaluation is intent-sensitive with the goal of maximizing contextual separation to promote safer behavior. We demonstrate that current unlearning techniques perform poorly under this setting, showing weak contextual separation alongside poor performance in ROUGE and concept-level metrics. Our results reveal strong forgetting-utility trade-offs, limited gains in contextual sensitivity, and poor consistency in concept-level control across methods, and provide ideas for unlearning approaches that better align with real-world safety requirements. Our dataset is publicly available.

cs.CL↗

arXiv2026

Memisis: Orchestrating and Evaluating Synthetic Data for Tabular Health Datasets

Synthetic data is widely used in healthcare to create datasets that preserve statistical properties of real data without exposing sensitive patient information. Generating and evaluating synthetic data across privacy, utility, and fairness dimensions is crucial for enabling high-quality data availability in downstream prediction tasks and clinical decision making. We present \textbf{Memisis}, a tool that orchestrates and evaluates synthetic data by leveraging existing synthesis libraries, large language models (LLMs), and state-of-the-art evaluation metrics. Our tool creates a unified workflow for data generation, validation, and evaluation. Users can control training size, training epochs, and the number of synthetic rows to sample. Beyond manual configuration, an interactive agent mode allows users to specify data generation goals in natural language, and the tool orchestrates the full pipeline by invoking existing synthesizers while performing the requisite evaluation. For the demo, we use an open-source schizophrenia dataset with protected attributes related to race and gender, evaluate six synthesizers spanning GANs, VAEs, diffusion models, and normalizing flows, and use a local LLM to orchestrate the workflow. The system affords users flexibility and control over the data generation and evaluation process.

cs.LG↗

arXiv2026

Fine-Tuning vs. RAG for Multi-Hop Question Answering with Novel Knowledge

Multi-hop question answering is widely used to evaluate the reasoning capabilities of large language models (LLMs), as it requires integrating multiple pieces of supporting knowledge to arrive at a correct answer. While prior work has explored different mechanisms for providing knowledge to LLMs, such as finetuning and retrieval-augmented generation (RAG), their relative effectiveness for multi-hop question answering remains insufficiently understood, particularly when the required knowledge is temporally novel. In this paper, we systematically compare parametric and non-parametric knowledge injection methods for open-domain multi-hop question answering. We evaluate unsupervised fine-tuning (continual pretraining), supervised fine-tuning, and retrieval-augmented generation across three 7B-parameter open-source LLMs. Experiments are conducted on two benchmarks: QASC, a standard multi-hop science question answering dataset, and a newly constructed dataset of over 10,000 multi-hop questions derived from Wikipedia events in 2024, designed to test knowledge beyond the models' pretraining cutoff. Our results show that unsupervised fine-tuning provides only limited gains over base models, suggesting that continual pretraining alone is insufficient for improving multi-hop reasoning accuracy. In contrast, retrieval-augmented generation yields substantial and consistent improvements, particularly when answering questions that rely on temporally novel information. Supervised fine-tuning achieves the highest overall accuracy across models and datasets. These findings highlight fundamental differences in how knowledge injection mechanisms support multi-hop question answering and underscore the importance of retrieval-based methods when external or compositional knowledge is required.

cs.CL↗

arXiv2025

Data Augmentation Supporting a Conversational Agent Designed for Smoking Cessation Support Groups

Online support groups for smoking cessation are economical and accessible, yet they often face challenges with low user engagement and stigma. The use of an automatic conversational agent would improve engagement by ensuring that all user comments receive a timely response.). We address the challenge of insufficient high-quality data by employing a two-level data augmentation strategy: synthetic data augmentation and real data augmentation. First, we fine-tuned an open source LLM to classify posts from our existing smoking cessation support groups and identify intents with low F1 (precision+recall) scores. Then, for these intents, we generate additional synthetic data using prompt engineering with the GPT model, with an average of 87\% of the generated synthetic posts deemed high quality by human annotators. Overall, the synthetic augmentation process resulted in 43\% of the original posts being selected for augmentation, followed by 140\% synthetic expansion of these posts. Additionally, we scraped more than 10,000 real posts from a related online support context, of which 73\% were validated as good quality by human annotators. Each synthetic or scraped post underwent rigorous validation involving human reviewers to ensure quality and relevance. The validated new data, combined with the original support group posts, formed an augmented dataset used to retrain the intent classifier. Performance evaluation of the retrained model demonstrated a 32\% improvement in F1, confirming the effectiveness of our data augmentation approach. Synthetic and real post augmentation led to similar performance improvements. This study provides a replicable framework for enhancing conversational agent performance in domains where data scarcity is a critical issue.

cs.CL↗

arXiv2025

Stringent Constraints on Gravitational Wave Signatures of Dark Electromagnetism in Neutron Star Binaries

Gravitational wave interferometers have studied compact object mergers and solidified our understanding of strong gravity. Their increasing precision raises the possibility of detecting new physics, especially in a neutron star binary system that may contain hidden-sector particles. In particular, a new vector force between binary constituents, giving rise to dark electromagnetic phenomena, could measurably alter the inspiral waveforms and thus be constrained by gravitational wave observations. In this work, we critically examine the mechanisms for neutron stars to acquire enough hidden-sector particles with requisite couplings to furnish a detectable signature from dark electromagnetism. We demonstrate that the repulsive nature of vector forces imposes stringent constraints on any putative particle physics model or astrophysical environment which could give rise to such gravitational signatures. We argue that absent an extreme fine-tuning of parameters, such signatures are well out of reach of any current or near-future gravitational wave observatory.

hep-ph↗

arXiv2025

CoopQ: Cooperative Game Inspired Layerwise Mixed Precision Quantization for LLMs

Large Language Models (LLMs) promise impressive capabilities, yet their multi-billion-parameter scale makes on-device or low-resource deployment prohibitive. Mixed-precision quantization offers a compelling solution, but existing methods struggle when the average precision drops below four bits, as they rely on isolated, layer-specific metrics that overlook critical inter-layer interactions affecting overall performance. To address these limitations, we first frame the mixed-precision quantization problem as a cooperative game among layers and introduce Shapley-based Progressive Quantization Estimation (SPQE) to efficiently obtain accurate Shapley estimates of layer sensitivities and inter-layer interactions. Leveraging the SPQE estimates, we propose Cooperative Game Inspired Mixed-Precision Quantization (CoopQ) which translates these Shapley estimates into a binary quadratic optimization formulation, assigning either 2 or 4-bit precision to layers under strict memory constraints. Comprehensive experiments conducted on Llama-3, Gemma-2, and Qwen-3 models across three independent PTQ backends (Quanto, HQQ, GPTQ) demonstrate CoopQ's scalability and consistently superior performance compared to methods relying solely on isolated metrics. Across average precisions spanning 4 bit down to 2 bit, CoopQ cuts Perplexity by 20 - 80 % relative to the best baseline, with the margin growing as the bit-width tightens.

cs.LG↗

arXiv2024

Building a Scalable, Effective, and Steerable Search and Ranking Platform

Modern e-commerce platforms offer vast product selections, making it difficult for customers to find items that they like and that are relevant to their current session intent. This is why it is key for e-commerce platforms to have near real-time scalable and adaptable personalized ranking and search systems. While numerous methods exist in the scientific literature for building such systems, many are unsuitable for large-scale industrial use due to complexity and performance limitations. Consequently, industrial ranking systems often resort to computationally efficient yet simplistic retrieval or candidate generation approaches, which overlook near real-time and heterogeneous customer signals, which results in a less personalized and relevant experience. Moreover, related customer experiences are served by completely different systems, which increases complexity, maintenance, and inconsistent experiences. In this paper, we present a personalized, adaptable near real-time ranking platform that is reusable across various use cases, such as browsing and search, and that is able to cater to millions of items and customers under heavy load (thousands of requests per second). We employ transformer-based models through different ranking layers which can learn complex behavior patterns directly from customer action sequences while being able to incorporate temporal (e.g. in-session) and contextual information. We validate our system through a series of comprehensive offline and online real-world experiments at a large online e-commerce platform, and we demonstrate its superiority when compared to existing systems, both in terms of customer experience as well as in net revenue. Finally, we share the lessons learned from building a comprehensive, modern ranking platform for use in a large-scale e-commerce environment.

cs.IR↗

arXiv2023

Constraining Dark Matter-Proton Scattering from Molecular Cloud Ionization

Optically dense clouds in the interstellar medium composed predominantly of molecular hydrogen, known as molecular clouds, are sensitive to energy injection in the form of photon absorption, cosmic-ray scattering, and dark matter (DM) scattering. The ionization rates in dense molecular clouds are heavily constrained by observations of abundances of various molecular tracers. Recent studies have set constraints on the DM-electron scattering cross section using measurements of ionization rates in dense molecular clouds. Here we calculate the analogous bounds on the DM-proton cross section using the molecular Migdal effect, recently adapted from the neutron scattering literature to the DM context. These bounds may be the strongest limits on a strongly-coupled DM subfraction, and represent the first application of the Migdal effect to astrophysical systems.

hep-ph↗

arXiv2022

The Molecular Migdal Effect

Nuclear scattering events with large momentum transfer in atomic, molecular, or solid-state systems may result in electronic excitations. In the context of atomic scattering by dark matter (DM), this is known as the Migdal effect, but the same effect has also been studied in molecules in the chemistry and neutron scattering literature. Here we present two distinct Migdal-like effects from DM scattering in molecules, which we collectively refer to as the molecular Migdal effect: a center-of-mass recoil, equivalent to the standard Migdal treatment, and a non-adiabatic coupling resulting from corrections to the Born-Oppenheimer approximation. The molecular bonds break spherical symmetry, leading to large daily modulation in the Migdal rate from anisotropies in the matrix elements. Our treatment reduces to the standard Migdal effect in atomic systems but does not rely on the impulse approximation or any semiclassical treatments of nuclear motion, and as such may be extended to models where DM scatters through a long-range force. We demonstrate all of these features in a few simple toy models of diatomic molecules, namely ${\rm H}_2^+$, N$_2$, and CO, and find total molecular Migdal rates competitive with those in semiconductors for the same target mass. We discuss how our results may be extended to more realistic targets comprised of larger molecules which could be deployed at the kilogram scale.

hep-ph↗

arXiv2022

A General Framework for Pairwise Unbiased Learning to Rank

Pairwise debiasing is one of the most effective strategies in reducing position bias in learning-to-rank (LTR) models. However, limiting the scope of this strategy, are the underlying assumptions required by many pairwise debiasing approaches. In this paper, we develop an approach based on a minimalistic set of assumptions that can be applied to a much broader range of user browsing patterns and arbitrary presentation layouts. We implement the approach as a simplified version of the Unbiased LambdaMART and demonstrate that it retains the underlying unbiasedness property in a wider variety of settings than the original algorithm. Finally, using simulations with "golden" relevance labels, we will show that the simplified version compares favourably with the original Unbiased LambdaMART when the examination of different positions in a ranked list is not assumed to be independent.

cs.IR↗