In their study, Li, Appleby, and Suh explore the potential of large language models (LLMs) in assisting users with the visual exploration and analysis of knowledge graphs (KGs). Through a mixed-methods approach involving surveys and interviews with 20 professionals from various sectors, including industry, government laboratories, and academia, the researchers identified key preferences and concerns regarding the integration of LLMs in KG exploration. The findings reveal a strong desire among participants for LLMs to aid in data retrieval from KGs through collaborative query construction. Additionally, participants expressed interest in using LLMs to uncover interesting relationships within KGs through multi-turn conversations and to generate on-demand visualizations that enhance trust in the model's outputs. To facilitate interactions with LLMs, participants favored a chat-based 'widget' integrated into their existing analysis workflows, allowing them to guide the model through interactions with visualizations. When it comes to viewing LLM outputs, participants preferred a combination of annotated visuals (such as subgraphs or tables extracted from KGs) alongside summarizing text. However, concerns were raised about the model's ability to maintain semantic intent when translating natural language questions into KG queries. Based on their analysis of interviews, the researchers propose a preliminary roadmap for designing LLM-driven systems for exploring knowledge graphs. This roadmap outlines future opportunities in this emerging design space and provides valuable insights for further research and development in leveraging LLMs as assistants in KG exploration and visualization.
SummaryResearchers studied how big language models can help people explore and understand knowledge graphs. They talked to 20 professionals from different fields to learn what they want from these models. The professionals want the models to help them find information in the graphs by working together on questions. They are also interested in using the models to discover connections in the graphs and create visualizations when needed. They prefer a chat-based tool that fits into their current work routines for interacting with the models.
Definitions
- Language Models (LLMs): Computer programs that can understand and generate human language.
- Knowledge Graphs (KGs): Visual representations of interconnected data or information.
- Collaborative: Working together with others towards a common goal.
- Visualizations: Images or diagrams that represent data or information visually.
- Semantic Intent: The meaning or purpose behind words or phrases used in communication.
Introduction
The use of knowledge graphs (KGs) has become increasingly popular in various industries, including healthcare, finance, and e-commerce. KGs are powerful tools for organizing and representing complex data relationships, making them valuable resources for data analysis and decision-making. However, as the size and complexity of KGs continue to grow, so does the need for efficient ways to explore and analyze them.
In recent years, large language models (LLMs) have emerged as a promising technology for natural language processing tasks. These models are trained on vast amounts of text data and can generate human-like responses to queries or prompts. As such, they have the potential to assist users in exploring KGs by translating natural language questions into database queries or generating visualizations based on user input.
Through a mixed-method approach involving surveys and interviews with 20 professionals from various sectors, including industry, government laboratories, and academia, the researchers identify key preferences and concerns regarding the integration of LLMs in KG exploration.
Key Findings
The study reveals that there is a strong desire among participants for LLMs to aid in data retrieval from KGs through collaborative query construction. This means that users want to work together with the model to refine their queries iteratively until they get desired results. This approach allows users to leverage both their domain expertise and the model's capabilities effectively.
Additionally, participants expressed interest in using LLMs to uncover interesting relationships within KGs through multi-turn conversations. By engaging in back-and-forth dialogues with the model about specific entities or concepts within a graph, users can gain deeper insights into complex relationships between different nodes.
Another important finding is that participants want LLM-driven systems to generate on-demand visualizations that enhance trust in the model's outputs. This means that users want to see visual representations of the data and relationships within KGs, generated by the model based on their queries or prompts. These visualizations can help users better understand and validate the results provided by LLMs.
Preferred Interaction and Output Formats
To facilitate interactions with LLMs, participants favored a chat-based 'widget' integrated into their existing analysis workflows. This widget would allow them to guide the model through interactions with visualizations, such as pointing to specific nodes or relationships within a graph. This approach makes it easier for users to provide context and refine their queries in real-time.
When it comes to viewing LLM outputs, participants preferred a combination of annotated visuals (such as subgraphs or tables extracted from KGs) alongside summarizing text. The use of both formats allows for a more comprehensive understanding of the results and helps users identify patterns or insights that may not be apparent from just one format alone.
Concerns about Semantic Intent
One concern raised by participants was about the model's ability to maintain semantic intent when translating natural language questions into KG queries. In other words, there is a risk that the model may misinterpret user input and generate incorrect results. To address this concern, researchers suggest incorporating user feedback mechanisms into LLM-driven systems so that users can correct any errors made by the model.
Proposed Roadmap
This roadmap outlines future opportunities in this emerging design space and provides valuable insights for further research and development in leveraging LLMs as assistants in KG exploration and visualization.
The proposed roadmap includes steps such as identifying appropriate datasets for training LLM models specific to different domains, developing interactive interfaces that enable collaborative query construction between users and models, integrating user feedback mechanisms into LLM-driven systems, and exploring different visualization techniques for presenting LLM outputs.
Conclusion
The findings reveal key preferences and concerns among professionals from various sectors regarding the integration of LLMs in KG exploration. By incorporating these insights into future research and development, we can harness the power of LLMs to enhance our understanding and utilization of knowledge graphs in different industries.
We present a mixed-methods study to explore how large language models (LLMs) can assist users in the visual exploration and analysis of knowledge graphs (KGs). We surveyed and interviewed 20 professionals from industry, government laboratories, and academia who regularly work with KGs and LLMs, either collaboratively or concurrently. Our findings show that participants overwhelmingly want an LLM to facilitate data retrieval from KGs through joint query construction, to identify interesting relationships in the KG through multi-turn conversation, and to create on-demand visualizations from the KG that enhance their trust in the LLM's outputs. To interact with an LLM, participants strongly prefer a chat-based 'widget,' built on top of their regular analysis workflows, with the ability to guide the LLM using their interactions with a visualization. When viewing an LLM's outputs, participants similarly prefer a combination of annotated visuals (e.g., subgraphs or tables extracted from the KG) alongside summarizing text. However, participants also expressed concerns about an LLM's ability to maintain semantic intent when translating natural language questions into KG queries, the risk of an LLM 'hallucinating' false data from the KG, and the difficulties of engineering a 'perfect prompt.' From the analysis of our interviews, we contribute a preliminary roadmap for the design of LLM-driven knowledge graph exploration systems and outline future opportunities in this emergent design space.
