Effective Bug Detection in Graph Database Engines: An LLM-based Approach

Graph database engines play a pivotal role in efficiently storing and managing graph data across various domains, including bioinformatics, knowledge graphs, and recommender systems. Graph databases must be accurate because errors lead to faulty analysis. Current bug-detection approaches are confined to specific graph query languages, limiting their applicabilities when handling graph database engines that use various graph query languages across various domains. Moreover, they require extensive prior knowledge to generate queries for detecting bugs. To address these challenges, we introduce DGDB, a novel paradigm harnessing large language models (LLM), such as ChatGPT, for comprehensive bug detection in graph database engines. DGDB leverages ChatGPT to generate high-quality queries for different graph query languages. It subsequently employs differential testing to identify bugs in graph database engines. We applied this paradigm to graph database engines based on Cypher, Gremlin, and SPARQL, and detected a total of 23 previously unknown wrong-result bugs. DGDB achieves at least 20.41% improvement in the non-empty-result query ratio and detects more than three times as many bugs as existing state-of-the-art methods on Cypher-based graph database engines, with further significant gains when employing more advanced LLM.