A Preliminary Roadmap for LLMs as Assistants in Exploring, Analyzing, and Visualizing Knowledge Graphs

Ranked by how close their title and abstract are to this paper’s. Select some to read them together.

Closest papers

1. Very close Cites this paper Not relevant

   The Role of Visualization in LLM-Assisted Knowledge Graph Systems: Effects on User Trust, Exploration, and Workflows

   Harry Li, Gabriel Appleby, Kenneth Alperin et al. · 2025 · arXiv.org · preprint, not peer reviewed · cited 3 times · Not summarized yet

   Abstract

   Knowledge graphs (KGs) are powerful data structures, but exploring them effectively remains difficult for even expert users. Large language models (LLMs) are increasingly used to address this gap, yet little is known empirically about how their usage with KGs shapes user trust, exploration strategies, or downstream decision-making - raising key design challenges for LLM-based KG visual analysis systems. To study these effects, we developed LinkQ, a KG exploration system that converts natural language questions into structured queries with an LLM. We collaborated with KG experts to design five visual mechanisms that help users assess the accuracy of both KG queries and LLM responses: an LLM-K…

2. Very close Not relevant

   LLMs for Knowledge Graph Construction and Reasoning: Recent Capabilities and Future Opportunities

   Yuqi Zhu, Xiaohan Wang, Jing Chen et al. · 2023 · Computation and Language · cited 337 times · Not summarized yet

   Abstract

   This paper presents an exhaustive quantitative and qualitative evaluation of Large Language Models (LLMs) for Knowledge Graph (KG) construction and reasoning. We employ eight distinct datasets that encompass aspects including entity, relation and event extraction, link prediction, and question answering. Empirically, our findings suggest that GPT-4 outperforms ChatGPT in the majority of tasks and even surpasses fine-tuned models in certain reasoning and question-answering datasets. Moreover, our investigation extends to the potential generalization ability of LLMs for information extraction, which culminates in the presentation of the Virtual Knowledge Extraction task and the development of …

3. Very close Not relevant

   Designing for Collaboration: Visualization to Enable Human–LLM Analytical Partnership

   Mai Elshehaly, Radu Jianu, A. Slingsby et al. · 2025 · IEEE Computer Graphics and Applications · cited 3 times · Not summarized yet

   Abstract

   Visualization artifacts have long served as anchors for collaboration and knowledge transfer in data analysis. While effective for human–human collaboration, little is known about their role in capturing and externalizing knowledge when working with large language models (LLMs). Despite the growing role of LLMs in analytics, their linear text-based workflows limit the ability to structure artifacts into useful and traceable representations of the analytical process. We argue that dynamic visual representations of evolving analysis—organizing artifacts and provenance into semantic structures, such as idea development and shifts in inquiry—are critical for effective human–LLM workflows. We dem…

4. Very close Not relevant

   LinkQ: An LLM-Assisted Visual Interface for Knowledge Graph Question-Answering

   Harry Li, Gabriel Appleby, Ashley Suh · 2024 · Visual .. · cited 15 times · Not summarized yet

   Abstract

   We present LinkQ, a system that leverages a large language model (LLM) to facilitate knowledge graph (KG) query construction through natural language question-answering. Traditional approaches often require detailed knowledge of a graph querying language, limiting the ability for users – even experts – to acquire valuable insights from KGs. LinkQ simplifies this process by implementing a multistep protocol in which the LLM interprets a user’s question, then systematically converts it into a well-formed query. LinkQ helps users iteratively refine any open-ended questions into precise ones, supporting both targeted and exploratory analysis. Further, LinkQ guards against the LLM hallucinating o…

5. Very close Cited by this paper Not relevant

   Unifying Large Language Models and Knowledge Graphs: A Roadmap

   Shirui Pan, Linhao Luo, Yufei Wang et al. · 2023 · Computation and Language · cited 1,930 times · AI summary available

   Abstract

   Large language models (LLMs), such as ChatGPT and GPT4, are making new waves in the field of natural language processing and artificial intelligence, due to their emergent ability and generalizability. However, LLMs are black-box models, which often fall short of capturing and accessing factual knowledge. In contrast, Knowledge Graphs (KGs), Wikipedia and Huapu for example, are structured knowledge models that explicitly store rich factual knowledge. KGs can enhance LLMs by providing external knowledge for inference and interpretability. Meanwhile, KGs are difficult to construct and evolving by nature, which challenges the existing methods in KGs to generate new facts and represent unseen kn…

6. Very close Not relevant

   LLMMaps -- A Visual Metaphor for Stratified Evaluation of Large Language Models

   Patrik Puchert, Poonam Poonam, Christian van Onzenoodt et al. · 2023 · Computation and Language · preprint, not peer reviewed · cited 12 times · AI summary available

   Abstract

   Large Language Models (LLMs) have revolutionized natural language processing and demonstrated impressive capabilities in various tasks. Unfortunately, they are prone to hallucinations, where the model exposes incorrect or false information in its responses, which renders diligent evaluation approaches mandatory. While LLM performance in specific knowledge fields is often evaluated based on question and answer (Q&A) datasets, such evaluations usually report only a single accuracy number for the entire field, a procedure which is problematic with respect to transparency and model improvement. A stratified evaluation could instead reveal subfields, where hallucinations are more likely to occur …

7. Very close Cited by this paper Not relevant

   MindMap: Knowledge Graph Prompting Sparks Graph of Thoughts in Large Language Models

   Yi-Lin Wen, Zifeng Wang, Jimeng Sun · 2023 · Annual Meeting of the Association for Computational Linguistics · cited 191 times · Not summarized yet

   Abstract

   Large language models (LLMs) have achieved remarkable performance in natural language understanding and generation tasks. However, they often suffer from limitations such as difficulty in incorporating new knowledge, generating hallucinations, and explaining their reasoning process. To address these challenges, we propose a novel prompting pipeline, named \method, that leverages knowledge graphs (KGs) to enhance LLMs' inference and transparency. Our method enables LLMs to comprehend KG inputs and infer with a combination of implicit and external knowledge. Moreover, our method elicits the mind map of LLMs, which reveals their reasoning pathways based on the ontology of knowledge. We evaluate…

8. Very close Not relevant

   AGENTiGraph: An Interactive Knowledge Graph Platform for LLM-based Chatbots Utilizing Private Data

   Xinjie Zhao, Moritz Blum, Rui Yang et al. · 2024 · arXiv.org · preprint, not peer reviewed · cited 16 times · Not summarized yet

   Abstract

   Large Language Models~(LLMs) have demonstrated capabilities across various applications but face challenges such as hallucination, limited reasoning abilities, and factual inconsistencies, especially when tackling complex, domain-specific tasks like question answering~(QA). While Knowledge Graphs~(KGs) have been shown to help mitigate these issues, research on the integration of LLMs with background KGs remains limited. In particular, user accessibility and the flexibility of the underlying KG have not been thoroughly explored. We introduce AGENTiGraph (Adaptive Generative ENgine for Task-based Interaction and Graphical Representation), a platform for knowledge management through natural lan…

Prior work it builds on

1. Close Used in depth by this paper A method it uses Not relevant

   Knowledge Graphs in Practice: Characterizing their Users, Challenges, and Visualization Opportunities

   Harry Li, Gabriel Appleby, C. Brumar et al. · 2023 · IEEE Transactions on Visualization and Computer Graphics · cited 53 times · Not summarized yet

   Abstract

   This study presents insights from interviews with nineteen Knowledge Graph (KG) practitioners who work in both enterprise and academic settings on a wide variety of use cases. Through this study, we identify critical challenges experienced by KG practitioners when creating, exploring, and analyzing KGs that could be alleviated through visualization design. Our findings reveal three major personas among KG practitioners – KG Builders, Analysts, and Consumers – each of whom have their own distinct expertise and needs. We discover that KG Builders would benefit from schema enforcers, while KG Analysts need customizable query builders that provide interim query results. For KG Consumers, we iden…

2. Very close A method it uses Not relevant

   Can Knowledge Graphs Reduce Hallucinations in LLMs? : A Survey

   Garima Agrawal, Tharindu Kumarage, Zeyad Alghamdi et al. · 2023 · North American Chapter of the Association for Computational Linguistics · cited 239 times · Not summarized yet

   Abstract

   The contemporary LLMs are prone to producing hallucinations, stemming mainly from the knowledge gaps within the models. To address this critical limitation, researchers employ diverse strategies to augment the LLMs by incorporating external knowledge, aiming to reduce hallucinations and enhance reasoning accuracy. Among these strategies, leveraging knowledge graphs as a source of external information has demonstrated promising results. In this survey, we comprehensively review these knowledge-graph-based augmentation techniques in LLMs, focusing on their efficacy in mitigating hallucinations. We systematically categorize these methods into three overarching groups, offering methodological co…

3. Close Used in depth by this paper Not relevant

   Enhancing LLM Factual Accuracy with RAG to Counter Hallucinations: A Case Study on Domain-Specific Queries in Private Knowledge-Bases

   Jia-Rui Li, Ye Yuan, Zehua Zhang · 2024 · arXiv.org · preprint, not peer reviewed · cited 148 times · Not summarized yet

   Abstract

   We proposed an end-to-end system design towards utilizing Retrieval Augmented Generation (RAG) to improve the factual accuracy of Large Language Models (LLMs) for domain-specific and time-sensitive queries related to private knowledge-bases. Our system integrates RAG pipeline with upstream datasets processing and downstream performance evaluation. Addressing the challenge of LLM hallucinations, we finetune models with a curated dataset which originates from CMU's extensive resources and annotated with the teacher model. Our experiments demonstrate the system's effectiveness in generating more accurate answers to domain-specific and time-sensitive inquiries. The results also revealed the limi…

4. Close A method it uses Not relevant

   Using an LLM to Help with Code Understanding

   Daye Nam, A. Macvean, Vincent J. Hellendoorn et al. · 2023 · International Conference on Software Engineering · cited 723 times · Not summarized yet

   Abstract

   Understanding code is challenging, especially when working in new and complex development environments. Code comments and documentation can help, but are typically scarce or hard to navigate. Large language models (LLMs) are revolutionizing the process of writing code. Can they do the same for helping understand it? In this study, we provide a first investigation of an LLM-based conversational UI built directly in the IDE that is geared towards code understanding. Our IDE plugin queries OpenAI's GPT-3.5-turbo model with four high-level requests without the user having to write explicit prompts: to explain a highlighted section of code, provide details of API calls used in the code, explain k…

5. Close Not relevant

   Reasoning on Graphs: Faithful and Interpretable Large Language Model Reasoning

   Lin-Hao Luo, Yuan-Fang Li, G. Haffari et al. · 2023 · International Conference on Learning Representations · cited 592 times · Not summarized yet

   Abstract

   Large language models (LLMs) have demonstrated impressive reasoning abilities in complex tasks. However, they lack up-to-date knowledge and experience hallucinations during reasoning, which can lead to incorrect reasoning processes and diminish their performance and trustworthiness. Knowledge graphs (KGs), which capture vast amounts of facts in a structured format, offer a reliable source of knowledge for reasoning. Nevertheless, existing KG-based LLM reasoning methods only treat KGs as factual knowledge bases and overlook the importance of their structural information for reasoning. In this paper, we propose a novel method called reasoning on graphs (RoG) that synergizes LLMs with KGs to en…

Papers that cite it

1. Very close Not relevant

   LLM-Powered Knowledge Graphs for Enterprise Intelligence and Analytics

   Rajeev Kumar, Kumar Ishan, H. Kumar et al. · 2025 · International Conference on Recent Advances in Information Technology · cited 13 times · Not summarized yet

   Abstract

   Disconnected data silos within enterprises obstruct the extraction of actionable insights, diminishing efficiency in areas such as product development, client engagement, meeting preparation, and analytics-driven decision-making. This paper introduces a framework that uses large language models (LLMs) to unify various data sources into a comprehensive, activity-centric knowledge graph. The framework automates tasks such as entity extraction, relationship inference, and semantic enrichment, enabling advanced querying, reasoning, and analytics across data types like emails, calendars, chats, documents, and logs. Designed for enterprise flexibility, it supports applications such as contextual s…

2. Close Not relevant

   Mitigating LLM Hallucinations with Knowledge Graphs: A Case Study

   Harry Li, Gabriel Appleby, Kenneth Alperin et al. · 2025 · arXiv.org · preprint, not peer reviewed · cited 5 times · Not summarized yet

   Abstract

   High-stakes domains like cyber operations need responsible and trustworthy AI methods. While large language models (LLMs) are becoming increasingly popular in these domains, they still suffer from hallucinations. This research paper provides learning outcomes from a case study with LinkQ, an open-source natural language interface that was developed to combat hallucinations by forcing an LLM to query a knowledge graph (KG) for ground-truth data during question-answering (QA). We conduct a quantitative evaluation of LinkQ using a well-known KGQA dataset, showing that the system outperforms GPT-4 but still struggles with certain question categories - suggesting that alternative query constructi…

3. Close Not relevant

   GraphSeek: Next-Generation Graph Analytics with LLMs

   Maciej Besta, Lukasz Jarmocik, O. Hrycyna et al. · 2026 · arXiv.org · preprint, not peer reviewed · Not summarized yet

   Abstract

   Graphs are foundational across domains but remain hard to use without deep expertise. LLMs promise accessible natural language (NL) graph analytics, yet they fail to process industry-scale property graphs effectively and efficiently: such datasets are large, highly heterogeneous, structurally complex, and evolve dynamically. To address this, we devise a novel abstraction for complex multi-query analytics over such graphs. Its key idea is to replace brittle generation of graph queries directly from NL with planning over a Semantic Catalog that describes both the graph schema and the graph operations. Concretely, this induces a clean separation between a Semantic Plane for LLM planning and bro…

4. Close Not relevant

   Leveraging Knowledge Graphs and LLMs for Context-Aware Messaging

   Rajeev Kumar, H. Kumar, K. Shalini · 2025 · 2025 IEEE International Conference on Emerging Technologies and Applications (MPSec ICETA) · cited 3 times · Not summarized yet

   Abstract

   Personalized messaging plays an essential role in improving communication in areas such as healthcare, education, and professional engagement. This paper introduces a framework that uses the Knowledge Graph (KG) to dynamically rephrase written communications by integrating individual and contextspecific data. The knowledge graph represents individuals, locations, and events as critical nodes, linking entities mentioned in messages to their corresponding graph nodes. The extraction of relevant information, such as preferences, professional roles, and cultural norms, is then combined with the original message and processed through a large language model (LLM) to generate personalized responses…

5. Close Not relevant

   Transforming Natural Language into Knowledge Graph Queries with LinkQ: An Agentic Visual Interface

   Harry Li, Gabriel Appleby, Kenneth Alperin et al. · 2026 · International Working Conference on Advanced Visual Interfaces · Not summarized yet

   Abstract

   Exploring Knowledge Graphs (KGs) remains an open challenge, as doing so requires expertise in unconventional querying languages and reasoning over complex, evolving schemas. We present LinkQ, an agentic KG-exploration system that helps users refine natural language questions into structured queries through guided large language model (LLM) collaboration. LinkQ’s design was informed by a formative study with KG experts, resulting in a human-in-the-loop interface that prioritizes transparency through visual mechanisms for verifying the agent’s behavior and generated outputs. LinkQ has five primary components: an agentic state diagram that contextualizes each stage of the agent-querying pipelin…