Searcharxiv⌕ Search

arXiv subjects

Muhammad Aimal Rehman

Publications and source records attributed to Muhammad Aimal Rehman.

3 recordsLinked to original sources

ragR: Retrieval-Augmented Generation and RAG Assessment in R

Retrieval-augmented generation (RAG) combines document retrieval with large language models (LLMs) to produce responses grounded in external evidence. While several R packages support core components of RAG workflows, integrated evaluation of RAG systems in R remains limited and is often conducted through Python-based tools, most notably the RAG assessment (RAGAS) framework. To address this gap, we introduce ragR, an R package that unifies document ingestion, embedding and vector storage, similarity-based retrieval, grounded generation, structured question-answer logging, and RAGAS-style evaluation within a single R-native workflow. Its evaluation module, ragR-RAGAS, uses an LLM as a judge to score four core RAGAS metrics: context precision, context recall, faithfulness, and answer relevance. To assess the credibility of these automated judgments, we compared ragR-RAGAS with two independent human annotators on three document collections. The human annotators demonstrated consistently high agreement, and ragR-RAGAS generally aligned with their judgments, although discrepancies were more pronounced for context precision and the identification of unsupported statements. By integrating RAG construction and evaluation within a reproducible workflow in R, ragR provides a practical framework for research, teaching, and moderate-scale experimentation on RAG systems entirely within the R ecosystem.

stat.CO↗

Adaptive Retrieval, Selective Answers: Evaluating Conformal Factuality in Agentic RAG

Retrieval-augmented generation (RAG) improves the grounding of language-model responses through external evidence, yet retrieved information alone does not ensure factual support for generated claims. In agentic RAG, evidence acquisition is further complicated by adaptive retrieval decisions, where an agent iteratively determines whether to retrieve additional information or terminate the search. We investigate how conformal factuality can be applied in this setting to improve the reliability of generated responses. We formalize the agent-driven retrieval process and characterize a query-aware relevance-scoring formulation that accounts for evidence collected across successive retrieval decisions. Building on this formulation, we evaluate post-generation conformal claim filtering using GPT-6 Luna and Llama 3.1 8B across HotpotQA, Natural Questions, and TriviaQA. Our experiments demonstrate that increasingly stringent conformal targets are consistently associated with higher observed response-level factual support, accompanied by reductions in claim retention and non-empty response coverage. The extent of these trade-offs varies across models and datasets, highlighting the importance of jointly evaluating factual reliability and answer availability. These findings establish an empirical foundation for studying conformal factuality in agent-driven retrieval systems and motivate further investigation into improving the reliability of their generated responses.

cs.CL↗

Conformal Factuality Control for Multi-Hop Retrieval-Augmented Generation

Retrieval-augmented generation (RAG) can ground large language models in external evidence, but retrieved context does not guarantee that generated claims are factually supported. This problem is especially relevant in multi-hop RAG, where retrieval and reasoning proceed through multiple dependent stages. We study whether claim-level conformal factuality control, previously developed for RAG, remains effective in this setting. We apply split-conformal claim filtering to multi-hop RAG and evaluate it on HotpotQA, Natural Questions, and TriviaQA using Llama 3.1 8B and GPT-4o-mini, together with a single-hop reference experiment. Across all six multi-hop model-dataset configurations, increasingly stringent conformal targets consistently increase the fraction of responses whose retained claims are fully supported. At the 95% target, this rate ranges from 95.80% to 97.20%, compared with 55.60%-76.03% without filtering. However, the improvement is strongly selective: only 4.41%-31.09% of generated claims are retained and 9.70%-51.40% of responses remain non-empty at the 95% target. These results show that conformal factuality extends to multi-hop RAG, while demonstrating that nominal reliability must be interpreted jointly with claim retention and abstention.

cs.CL↗