Searcharxiv⌕ Search

arXiv subjects

Chi-Kuang Yeh

Publications and source records attributed to Chi-Kuang Yeh.

12 recordsLinked to original sources

ragR: Retrieval-Augmented Generation and RAG Assessment in R

Retrieval-augmented generation (RAG) combines document retrieval with large language models (LLMs) to produce responses grounded in external evidence. While several R packages support core components of RAG workflows, integrated evaluation of RAG systems in R remains limited and is often conducted through Python-based tools, most notably the RAG assessment (RAGAS) framework. To address this gap, we introduce ragR, an R package that unifies document ingestion, embedding and vector storage, similarity-based retrieval, grounded generation, structured question-answer logging, and RAGAS-style evaluation within a single R-native workflow. Its evaluation module, ragR-RAGAS, uses an LLM as a judge to score four core RAGAS metrics: context precision, context recall, faithfulness, and answer relevance. To assess the credibility of these automated judgments, we compared ragR-RAGAS with two independent human annotators on three document collections. The human annotators demonstrated consistently high agreement, and ragR-RAGAS generally aligned with their judgments, although discrepancies were more pronounced for context precision and the identification of unsupported statements. By integrating RAG construction and evaluation within a reproducible workflow in R, ragR provides a practical framework for research, teaching, and moderate-scale experimentation on RAG systems entirely within the R ecosystem.

stat.CO↗

Adaptive Retrieval, Selective Answers: Evaluating Conformal Factuality in Agentic RAG

Retrieval-augmented generation (RAG) improves the grounding of language-model responses through external evidence, yet retrieved information alone does not ensure factual support for generated claims. In agentic RAG, evidence acquisition is further complicated by adaptive retrieval decisions, where an agent iteratively determines whether to retrieve additional information or terminate the search. We investigate how conformal factuality can be applied in this setting to improve the reliability of generated responses. We formalize the agent-driven retrieval process and characterize a query-aware relevance-scoring formulation that accounts for evidence collected across successive retrieval decisions. Building on this formulation, we evaluate post-generation conformal claim filtering using GPT-6 Luna and Llama 3.1 8B across HotpotQA, Natural Questions, and TriviaQA. Our experiments demonstrate that increasingly stringent conformal targets are consistently associated with higher observed response-level factual support, accompanied by reductions in claim retention and non-empty response coverage. The extent of these trade-offs varies across models and datasets, highlighting the importance of jointly evaluating factual reliability and answer availability. These findings establish an empirical foundation for studying conformal factuality in agent-driven retrieval systems and motivate further investigation into improving the reliability of their generated responses.

cs.CL↗

Severity-Controlled Prediction Sets for Medication Recommendation

Medication recommendation is a multi-label prediction task. However, existing prediction-set evaluation and calibration methods rely mainly on exact-overlap metrics and therefore treat all missed medications as equally severe, overlooking how closely the prediction set matches them within the medication hierarchy. To address this, we introduce Conformal Severity Control (CSC), a post-hoc framework for calibrating medication prediction sets under hierarchy-aware mismatch severity. CSC uses the Anatomical Therapeutic Chemical (ATC) hierarchy to quantify mismatch severity and controls patient-level severity without retraining the predictor, supporting both average severity and upper-quantile severity for the highest-severity mismatches. Experiments on MIMIC-IV show that CSC tracks user-specified severity tolerances and shifts prediction sets away from distant mismatches toward exact or close matches.

cs.LG↗

Patients-like-me: A Variational LM--GNN Framework for Explainable Clinical Prediction

Language models (LMs) offer strong textual representations for electronic health records (EHRs), but they encode patient sequences in isolation and provide limited explainability. Graph neural networks (GNNs) complement LMs by incorporating inter-patient relationships and enabling reference-patient attribution, yet they rely on high-quality patient representations. We propose Patients-like-me (PLM), a unified LM--GNN framework that integrates local patient semantics with global cohort structure. To train PLM efficiently, we introduce a Variational Expectation-Maximization algorithm that alternates LM and GNN updates under a supervised variational objective. Extensive experiments on MIMIC-III and MIMIC-IV show that PLM consistently outperforms state-of-the-art methods, with improvements generalizing across encoder-only and decoder-only LM backbones. These gains are achieved with only modest additional computational overhead. PLM also provides reference-patient explanations by retrieving influential similar patients, while edge-masking experiments confirm that the highest-ranked references have the greatest impact on model predictions.

cs.CL↗

Conformal Factuality Control for Multi-Hop Retrieval-Augmented Generation

Retrieval-augmented generation (RAG) can ground large language models in external evidence, but retrieved context does not guarantee that generated claims are factually supported. This problem is especially relevant in multi-hop RAG, where retrieval and reasoning proceed through multiple dependent stages. We study whether claim-level conformal factuality control, previously developed for RAG, remains effective in this setting. We apply split-conformal claim filtering to multi-hop RAG and evaluate it on HotpotQA, Natural Questions, and TriviaQA using Llama 3.1 8B and GPT-4o-mini, together with a single-hop reference experiment. Across all six multi-hop model-dataset configurations, increasingly stringent conformal targets consistently increase the fraction of responses whose retained claims are fully supported. At the 95% target, this rate ranges from 95.80% to 97.20%, compared with 55.60%-76.03% without filtering. However, the improvement is strongly selective: only 4.41%-31.09% of generated claims are retained and 9.70%-51.40% of responses remain non-empty at the 95% target. These results show that conformal factuality extends to multi-hop RAG, while demonstrating that nominal reliability must be interpreted jointly with claim retention and abstention.

cs.CL↗

AURA: Adaptive Uncertainty-aware Refinement for LLM-as-a-Judge Auditing

Large language models (LLMs) are increasingly used as judges for open-ended generation, as large-scale human evaluation is often expensive and difficult to scale, yet their preferences remain imperfect proxies for human judgment. Existing auditing pipelines often assume that a reliable subset of examples or clean supervision signals are available beforehand, for example from human annotation, heuristic filtering, or the outputs of strong judges. In LLM evaluation, this assumption is fragile: the initial split may inherit judge bias, while human verification is typically too scarce to define stable groups at scale. We propose AURA, an adaptive uncertainty--aware refinement framework for auditing pairwise LLM--as--a--judge decisions under selected human verification. AURA iteratively learns a human-consistency signal, propagates reliable evidence, and prioritizes uncertain comparisons for human review. The key idea is to treat trust in a judge as a latent quantity that is progressively refined as evidence accumulates. We provide a compact formulation, a stable refinement procedure, and a comprehensive evaluation on both synthetic and real pairwise LLM-answer data.

stat.ML↗

Quantifying and Auditing LLM Evaluation via Positive--Unlabeled Learning

Large Language Models (LLMs) are increasingly used as judges for scalable evaluation, yet such LLM--as--a--Judge systems exhibit systematic biases that are decoupled from semantic quality, most notably verbosity bias. Meanwhile, human supervision is costly and typically selective, yielding reliable positive judgments but leaving most outputs unlabelled and potentially mixed in quality. We formulate LLM evaluation under selective human supervision as a positive--unlabelled learning problem and propose a geometric auditing framework based on Partial Optimal Transport. By aligning a small set of human--verified positives with a reliable subset of unlabelled outputs in a fixed embedding space, our method identifies human--consistent preferences and corrects biased judges without retraining. Experiments demonstrate improved alignment with human preferences, increased robustness to presentation biases, and interpretable confidence estimates, offering a scalable and statistically grounded alternative to existing LLM--as--a--judge pipelines.

stat.ML↗

Single and multi-objective optimal designs for group testing experiments with a focus on screening for an infectious disease

Group testing techniques are widely used in resource-constrained settings, such as infectious-disease screening, blood safety, DNA library screening, and industrial inspection, where the efficient use of limited testing resources depends critically on how the initial study is designed. This paper discusses various ways that group testing experiments can be designed more efficiently and flexibly, under a user-specified optimality criterion and cost structure. We construct optimal designs to estimate model parameters beyond the \(D\)-optimality criterion to include the \(A\)-, \(c\)-, \(E\)-optimality, and extend the framework for finding optimal designs with multiple objectives. For large studies, we use a general theory and obtain various types of optimal approximate designs. When sample sizes are small, we propose two algorithms to construct highly efficient exact designs under realistic budget constraints. Additionally, we investigate properties of the proposed designs under various operational uncertainties and create a Shiny app to facilitate implementation of the proposed designs. To fix ideas, we focus on finding highly efficient group testing designs for a Chlamydia screening trial with imperfect assays under budget constraints and show the advantages of our optimal designs over current methods.

stat.ME↗

Positive and Unlabeled Data: Model, Estimation, Inference, and Classification

This study introduces a new approach to addressing positive and unlabeled (PU) data through the double exponential tilting model (DETM). Traditional methods often fall short because they only apply to selected completely at random (SCAR) PU data, where the labeled positive and unlabeled positive data are assumed to be from the same distribution. In contrast, our DETM's dual structure effectively accommodates the more complex and underexplored selected at random PU data, where the labeled and unlabeled positive data can be from different distributions. We rigorously establish the theoretical foundations of DETM, including identifiability, parameter estimation, and asymptotic properties. Additionally, we move forward to statistical inference by developing a goodness-of-fit test for the SCAR condition and constructing confidence intervals for the proportion of positive instances in the target domain. We leverage an approximated Bayes classifier for classification tasks, demonstrating DETM's robust performance in prediction. Through theoretical insights and practical applications, this study highlights DETM as a comprehensive framework for addressing the challenges of PU data.

stat.ME↗

CVXSADes: a stochastic algorithm for constructing optimal exact regression designs with single or multiple objectives

We propose an algorithm to construct optimal exact designs (EDs). Most of the work in the optimal regression design literature focuses on the approximate design (AD) paradigm due to its desired properties, including the optimality verification conditions derived by Kiefer (1959, 1974). ADs may have unbalanced weights, and practitioners may have difficulty implementing them with a designated run size $n$. Some EDs are constructed using rounding methods to get an integer number of runs at each support point of an AD, but this approach may not yield optimal results. To construct EDs, one may need to perform new combinatorial constructions for each $n$, and there is no unified approach to construct them. Therefore, we develop a systematic way to construct EDs for any given $n$. Our method can transform ADs into EDs while retaining high statistical efficiency in two steps. The first step involves constructing an AD by utilizing the convex nature of many design criteria. The second step employs a simulated annealing algorithm to search for the ED stochastically. Through several applications, we demonstrate the utility of our method for various design problems. Additionally, we show that the design efficiency approaches unity as the number of design points increases.

stat.ME↗

Functional Spherical Autocorrelation: A Robust Estimate of the Autocorrelation of a Functional Time Series

We propose a new autocorrelation measure for functional time series that we term spherical autocorrelation. It is based on measuring the average angle between lagged pairs of series after having been projected onto the unit sphere. This new measure enjoys several complimentary advantages compared to existing autocorrelation measures for functional data, since it both 1) describes a notion of sign or direction of serial dependence in the series, and 2) is more robust to outliers. The asymptotic properties of estimators of the spherical autocorrelation are established, and are used to construct confidence intervals and portmanteau white noise tests. These confidence intervals and tests are shown to be effective in simulation experiments, and demonstrated in applications to model selection for daily electricity price curves, and measuring the volatility in densely observed asset price data.

stat.ME↗

Evaluating real-time probabilistic forecasts with application to National Basketball Association outcome prediction

Motivated by the goal of evaluating real-time forecasts of home team win probabilities in the National Basketball Association, we develop new tools for measuring the quality of continuously updated probabilistic forecasts. This includes introducing calibration surface plots, and simple graphical summaries of them, to evaluate at a glance whether a given continuously updated probability forecasting method is well-calibrated, as well as developing statistical tests and graphical tools to evaluate the skill, or relative performance, of two competing continuously updated forecasting methods. These tools are studied by means of a Monte Carlo simulation study of simulated basketball games, and demonstrated in an application to evaluate the continuously updated forecasts published by the United States-based multinational sports network ESPN on its principle webpage {\tt espn.com}. This application lends statistical evidence that the forecasts published there are well-calibrated, and exhibit improved skill over several naïve models, but do not demonstrate significantly improved skill over simple logistic regression models based solely on a measurement of each teams' relative strength, and the evolving score difference throughout the game.

stat.ME↗