# Implicit Chain of Thought Reasoning via Knowledge Distillation

**Authors:** Yuntian Deng, Kiran Prasad, Roland Fernandez, Paul Smolensky, Vishrav Chaudhary, Stuart Shieber  
**arXiv:** [2311.01460](https://arxiv.org/abs/2311.01460)  
**Date:** 2023-11-02  
**Summary page:** https://www.summarizepaper.com/en/paper/2311.01460v1/

## Key points

- Authors propose an alternative approach to reasoning in language models (LMs)
- LMs can reason effectively with intermediate computation not expressed in natural language
- Internal hidden states of the language model used for implicit reasoning instead of explicit chain-of-thought reasoning steps
- Implicit reasoning steps distilled from a teacher model trained on explicit chain-of-thought reasoning
- Vertical reasoning among hidden states in different layers instead of horizontal reasoning by generating intermediate words
- Experiments conducted on multi-digit multiplication task and grade school math problem dataset
- Results show that this alternative reasoning method allows solving previously unsolvable tasks without explicit chain-of-thought reasoning
- Comparable speed achieved to no chain-of-thought reasoning
- Enhances language models' ability to reason by leveraging implicit reasoning through hidden states
- Opens up new possibilities for solving complex tasks effectively and efficiently.

## Summary

In the paper "Implicit Chain of Thought Reasoning via Knowledge Distillation," authors Yuntian Deng, Kiran Prasad, Roland Fernandez, Paul Smolensky, Vishrav Chaudhary, and Stuart Shieber explore an alternative approach to reasoning in language models (LMs). Rather than prompting or finetuning LMs to generate a chain of thought reasoning steps before producing the final answer as is typically done by researchers, this study suggests that LMs could reason more effectively with intermediate computation that is not expressed in natural language. The authors propose using the internal hidden states of the language model to perform implicit reasoning instead of explicitly producing the chain of thought reasoning steps. They distill these implicit reasoning steps from a teacher model trained on explicit chain-of-thought reasoning. This approach enables vertical reasoning among the hidden states in different layers rather than conducting horizontal reasoning by generating intermediate words one-by-one. To evaluate their approach, the researchers conduct experiments on a multi-digit multiplication task and a grade school math problem dataset. The results demonstrate that this alternative reasoning method allows for solving tasks that were previously unsolvable without explicit chain-of-thought reasoning while achieving comparable speed to no chain-of-thought reasoning. Overall, this study presents a novel way to enhance language models' ability to reason by leveraging implicit reasoning through hidden states. By distilling knowledge from an explicit chain-of-thought teacher model, the proposed approach opens up new possibilities for solving complex tasks effectively and efficiently.