Improving Contextual Retrieval for Long Documents using Semantic Chunking and Hybrid Retrieval Strategies

Abstract

The vast amount of information available in lengthy, distributed documents makes finding relevant answers difficult, especially when the answer requires connecting evidence from multiple sources. Standard Retrieval-Augmented Generation (RAG) systems often struggle with this type of "multi-hop" reasoning and frequently face issues with context fragmentation. In this paper, we propose a hybrid RAG framework that integrates the structured reasoning of knowledge graphs with the broad semantic coverage of dense vector retrieval. Our approach employs a multi-stage pipeline using transformer-based summarization and semantic chunking to improve context retention. We construct a dynamic knowledge graph to capture explicit relationships between entities while simultaneously maintaining a dense vector index. During inference, the system uses a hierarchical mechanism that prioritizes graph traversal for precision and falls back to vector retrieval when necessary. We evaluated our framework on the MultiHop-RAG dataset. The results show that this hybrid approach outperforms standalone vector and graph-based baselines, achieving higher Exact Match scores and improved retrieval accuracy for complex queries. This confirms that merging structured reasoning with standard vector retrieval creates a more robust system for handling complex knowledge-intensive questions.