arXiv subjects

Huanshuo Liu

Publications and source records attributed to Huanshuo Liu.

3 recordsLinked to original sources

arXiv2024

CtrlA: Adaptive Retrieval-Augmented Generation via Inherent Control

Retrieval-augmented generation (RAG) has emerged as a promising solution for mitigating hallucinations of large language models (LLMs) with retrieved external knowledge. Adaptive RAG enhances this approach by enabling dynamic retrieval during generation, activating retrieval only when the query exceeds LLM's internal knowledge. Existing methods primarily focus on detecting LLM's confidence via statistical uncertainty. Instead, we present the first attempts to solve adaptive RAG from a representation perspective and develop an inherent control-based framework, termed \name. Specifically, we extract the features that represent the honesty and confidence directions of LLM and adopt them to control LLM behavior and guide retrieval timing decisions. We also design a simple yet effective query formulation strategy to support adaptive retrieval. Experiments show that \name is superior to existing adaptive RAG methods on a diverse set of tasks, the honesty steering can effectively make LLMs more honest and confidence monitoring is a promising indicator of retrieval trigger.Our code is available at \url{https://github.com/HSLiu-Initial/CtrlA}.

cs.CL↗

arXiv2024

Retrieval-Oriented Knowledge for Click-Through Rate Prediction

Click-through rate (CTR) prediction is crucial for personalized online services. Sample-level retrieval-based models, such as RIM, have demonstrated remarkable performance. However, they face challenges including inference inefficiency and high resource consumption due to the retrieval process, which hinder their practical application in industrial settings. To address this, we propose a universal plug-and-play \underline{r}etrieval-\underline{o}riented \underline{k}nowledge (\textbf{\name}) framework that bypasses the real retrieval process. The framework features a knowledge base that preserves and imitates the retrieved \& aggregated representations using a decomposition-reconstruction paradigm. Knowledge distillation and contrastive learning optimize the knowledge base, enabling the integration of retrieval-enhanced representations with various CTR models. Experiments on three large-scale datasets demonstrate \name's exceptional compatibility and performance, with the neural knowledge base serving as an effective surrogate for the retrieval pool. \name surpasses the teacher model while maintaining superior inference efficiency and demonstrates the feasibility of distilling knowledge from non-parametric methods using a parametric approach. These results highlight \name's strong potential for real-world applications and its ability to transform retrieval-based methods into practical solutions. Our implementation code is available to support reproducibility in \url{https://github.com/HSLiu-Initial/ROK.git}.

cs.IR↗

arXiv2024

Evaluating the External and Parametric Knowledge Fusion of Large Language Models

Integrating external knowledge into large language models (LLMs) presents a promising solution to overcome the limitations imposed by their antiquated and static parametric memory. Prior studies, however, have tended to over-reliance on external knowledge, underestimating the valuable contributions of an LLMs' intrinsic parametric knowledge. The efficacy of LLMs in blending external and parametric knowledge remains largely unexplored, especially in cases where external knowledge is incomplete and necessitates supplementation by their parametric knowledge. We propose to deconstruct knowledge fusion into four distinct scenarios, offering the first thorough investigation of LLM behavior across each. We develop a systematic pipeline for data construction and knowledge infusion to simulate these fusion scenarios, facilitating a series of controlled experiments. Our investigation reveals that enhancing parametric knowledge within LLMs can significantly bolster their capability for knowledge integration. Nonetheless, we identify persistent challenges in memorizing and eliciting parametric knowledge, and determining parametric knowledge boundaries. Our findings aim to steer future explorations on harmonizing external and parametric knowledge within LLMs.

cs.CL↗