# An automatically discovered chain-of-thought prompt generalizes to novel models and datasets

**Authors:** Konstantin Hebenstreit, Robert Praas, Louis P Kiesewetter, Matthias Samwald  
**arXiv:** [2305.02897](https://arxiv.org/abs/2305.02897)  
**Date:** 2023-05-04  
**Summary page:** https://www.summarizepaper.com/en/paper/2305.02897v1/

## Key points

- Large language models (LLMs) have shown remarkable performance in natural language processing tasks.
- However, their lack of explainability and interpretability has raised concerns about their reliability and trustworthiness.
- Emergent chain-of-thought (CoT) reasoning capabilities promise to address these issues by improving the performance and explainability of LLMs.
- A small-scale study was conducted to compare the performance of a range of zero-shot prompts for inducing CoT reasoning across six recently released LLMs: davinci-002, davinci-003, GPT-3.5-turbo, GPT-4, Flan-T5-xxl, and Cohere command xlarge on a mixture of six question answering datasets from various domains such as science and medicine.
- The CoT prompt discovered through automated prompt discovery demonstrated robust performance across all experimental conditions and produced the best results when applied to GPT 4.
- The study also included descriptions of various datasets used in the experiments such as StrategyQA, WorldTree v2, OpenBookQA, MedQA, MedMCQA which require implicit reasoning and multi step answer strategies based on prior knowledge or domain specific knowledge. Additionally critiques strategy requiring initial answers followed by critique then revised response were also included in the experiment.
- The study concludes that further research is needed to evaluate the performance of CoT prompts on different models and datasets. However this study provides evidence that automated prompt discovery can be a useful tool for developing CoT prompts that generalize well across novel models and datasets.

## Summary

In recent years, large language models (LLMs) have shown remarkable performance in natural language processing tasks. However, their lack of explainability and interpretability has raised concerns about their reliability and trustworthiness. Emergent chain-of-thought (CoT) reasoning capabilities promise to address these issues by improving the performance and explainability of LLMs. To evaluate whether a CoT prompt that was previously discovered through automated prompt discovery could show robust performance across experimental conditions and produce the best results when applied to state-of-the-art models, a small-scale study was conducted to compare the performance of a range of zero-shot prompts for inducing CoT reasoning across six recently released LLMs: davinci-002, davinci-003, GPT-3.5-turbo, GPT-4, Flan-T5-xxl, and Cohere command xlarge on a mixture of six question answering datasets from various domains such as science and medicine. The LangChain framework was used to access several APIs for the experiments. The results showed that the CoT prompt discovered through automated prompt discovery demonstrated robust performance across all experimental conditions and produced the best results when applied to GPT 4. This finding suggests that this CoT prompt can generalize well across novel models and datasets. The study also included descriptions of various datasets used in the experiments such as StrategyQA, WorldTree v2, OpenBookQA, MedQA, MedMCQA which require implicit reasoning and multi step answer strategies based on prior knowledge or domain specific knowledge. Additionally critiques strategy requiring initial answers followed by critique then revised response were also included in the experiment. The study concludes that further research is needed to evaluate the performance of CoT prompts on different models and datasets. However this study provides evidence that automated prompt discovery can be a useful tool for developing CoT prompts that generalize well across novel models and datasets.