# Boosting Language Models Reasoning with Chain-of-Knowledge Prompting

**Authors:** Jianing Wang, Qiushi Sun, Xiang Li, Ming Gao  
**arXiv:** [2306.06427](https://arxiv.org/abs/2306.06427)  
**Date:** 2023-06-10  
**Summary page:** https://www.summarizepaper.com/en/paper/2306.06427v3/

## Key points

- Researchers propose Chain-of-Knowledge (CoK) prompting to enhance reasoning capabilities of Large Language Models (LLMs)
- CoK aims to elicit LLMs to generate explicit pieces of knowledge evidence in the form of structured triples
- Introduces F^2-Verification method to evaluate factuality and faithfulness of generated evidence triples
- Outperforms other prompt methods across various reasoning tasks
- Plan to enhance performance of larger scale LLMs, integrate external knowledge bases for real-time verification, and conduct interpretability analysis on reasoning processes
- Limitations include finite coverage of evidence triples in knowledge bases and potentially increased API calls compared to traditional methods
- Utilizing publicly available data sources for knowledge bases ensures factual information is incorporated without introducing additional bias or harmful answers
- Acknowledges support from various funding sources and valuable feedback received during discussions

## Summary

Researchers propose a new approach, called Chain-of-Knowledge (CoK) prompting, to enhance the reasoning capabilities of Large Language Models (LLMs). Unlike traditional Chain-of-Thought (CoT) prompting, which can lead to unfactual and unfaithful reasoning chains, CoK aims to elicit LLMs to generate explicit pieces of knowledge evidence in the form of structured triples. This approach is inspired by human behavior, where individuals create mind maps or knowledge maps before answering complex questions. To further improve reliability, the researchers introduce a F^2-Verification method that evaluates the factuality and faithfulness of generated evidence triples. If an unreliable response is detected, incorrect evidence can be identified to prompt the LLM to reconsider its reasoning process. Extensive experiments demonstrate that this method outperforms other prompt methods across various reasoning tasks including commonsense, factual, symbolic, and arithmetic reasoning. Moving forward, the researchers plan to enhance the performance of larger scale LLMs, integrate external knowledge bases such as search engines for real-time verification, and conduct interpretability analysis on LLMs' reasoning processes. Despite its success in improving reasoning capabilities, CoK has limitations such as finite coverage of evidence triples in knowledge bases and potentially increased API calls compared to traditional CoT methods. From a social impact and ethics perspective, utilizing publicly available data sources for knowledge bases ensures that factual information is incorporated into LLMs' reasoning processes without introducing additional bias. This approach also helps prevent models from providing irresponsible or harmful answers. The study acknowledges support from various funding sources and expresses gratitude for valuable feedback received during discussions.