Text2GraphQuery-Bench: A Text to Graph Query Benchmark

Abstract

Graph models are fundamental to data analysis in domains rich with complex relationships. Unlike SQL, which benefits from a rel- atively unified standard and widespread familiarity, graph query languages are diverse (e.g., Cypher, GQL, SQL/PGQ) and far less fa- miliar to most users, making them significantly harder to learn and use. Text-to-Graph-Query systems address this barrier by trans- lating natural language into executable graph queries, enabling LLMs to serve as interfaces for Graph Database Management Systems (GDBMS). Existing benchmarks are limited in language coverage, rely on rigid synthesis, and lack comprehensive evaluation. We present Text2GraphQuery-Bench, the first benchmark covering all mainstream declarative property graph query languages (Cypher, GQL, and SQL/PGQ). It contains 267,276 (Question, Graph Query) pairs across 34 databases and 13 domains. Its construction supports adaptation from heterogeneous resources and domain-aware synthesis, while its Graph-IR-based design enables rapid extension to new languages. The evaluation protocol reports Grammar, GLEU, Similarity, and EX under graph-native difficulty, question abstraction, and schema aliasing. Experiments on 8 LLMs reveal: (i) a significant language gap exists - zero-shot GQL and SQL/PGQ Grammar is far below Cypher, yet few-shot prompting largely recovers it; (ii) fine-tuning an 8B model reaches or exceeds zero-shot large models, indicating unfamiliarity - rather than model capacity - is the primary barrier; (iii) as supervision increases, syntax errors recede, shifting bottlenecks to aggregation logic in GQL and schema linking in SQL/PGQ; (iv) higher question abstraction degrades EX due to intent-to-schema grounding issues, while schema aliasing has minimal impact; (v) EX consistently degrades from Easy to Extra Hard, with Extra Hard remaining a persistent bottleneck. *(Due to arXiv constraints, this abstract is shortened. See PDF for the full version.)*