# Few shot chain-of-thought driven reasoning to prompt LLMs for open ended medical question answering

**Authors:** Ojas Gramopadhye, Saeel Sandeep Nachane, Prateek Chanda, Ganesh Ramakrishnan, Kshitij Sharad Jadhav, Yatin Nandwani, Dinesh Raghu, Sachindra Joshi  
**arXiv:** [2403.04890](https://arxiv.org/abs/2403.04890)  
**Date:** 2024-03-07  
**Summary page:** https://www.summarizepaper.com/en/paper/2403.04890v1/

## Key points

- Large Language models (LLMs) revolutionizing healthcare sector by automating tasks like clinical documentation, information retrieval, and decision support
- LLM-driven tools interpreting patient queries, providing information on symptoms, diseases, treatments, and healthcare guidelines to enhance patient education and engagement
- Growing emphasis on accuracy and verifiability in medical scenarios with advancements in prompt engineering techniques for Large Language Models
- Fact verification LLMs crucial for automated fact-checking processes involving claim detection, evidence retrieval, and claim verification
- Chain-of-Thought Prompting instrumental in scaling up language models for reasoning-intensive tasks by prompting LLMs to generate step-by-step solutions through CoT reasoning
- Exploration of LLMs in generating accurate and reasoning-based responses to medical questions signifies significant advancement in the field
- Models like PubMedGPT and Codex setting benchmarks on datasets like MedQA by incorporating innovative approaches such as Classification head, Chain-of-Thought reasoning, and Knowledge Grounding
- Mimicking real-life clinical scenarios with subjective responses using Chain of Thought (CoT) reasoning based on subjective response generation for the dataset
- Reward training mechanism utilized to ensure response verification by providing appropriate verified responses from the language model
- Better learning strategies developed through modifications of existing prompts like 5-shot-codex-CoT-prompt for the subjective MedQA dataset and introduction of an incremental-reasoning prompt
- Evaluations demonstrating that greedy decoding with incremental reasoning method shows superior performance compared to other decoding strategies

## Summary

Large Language models (LLMs) have shown immense potential in revolutionizing the healthcare sector by automating tasks such as clinical documentation, information retrieval, and decision support. These LLM-driven tools can interpret patient queries, provide information on symptoms, diseases, treatments, and healthcare guidelines. This enhances patient education and engagement by making it more accessible and user-friendly. With advancements in prompt engineering techniques for Large Language Models, there is a growing emphasis on accuracy and verifiability in medical scenarios. Fact verification LLMs have emerged as a crucial tool for automated fact-checking processes. They involve claim detection, evidence retrieval, and claim verification to ensure the accuracy of responses generated by LLMs. Chain-of-Thought Prompting has been instrumental in scaling up language models for reasoning-intensive tasks. By prompting LLMs to generate step-by-step solutions through CoT reasoning, significant improvements have been observed in various challenging tasks. This approach allows LLMs to bridge the gap with human-level performances for complex tasks and datasets like MedQA. The exploration of LLMs in generating accurate and reasoning-based responses to medical questions signifies a significant advancement in the field. Models like PubMedGPT and Codex have set benchmarks on datasets like MedQA by incorporating innovative approaches such as Classification head, Chain-of-Thought reasoning, and Knowledge Grounding. These approaches highlight not only what is answered but also how the answer is derived. In this paper  is proposed to mimic real-life clinical scenarios with subjective responses. The Chain of Thought (CoT) reasoning based on subjective response generation is explored for this dataset using appropriate LM-driven forward reasoning for correct responses to medical questions. A reward training mechanism is utilized to ensure response verification by providing appropriate verified responses from the language model. Furthermore, better learning strategies are developed through modifications of existing prompts like 5-shot-codex-CoT-prompt for the subjective MedQA dataset and the introduction of an incremental-reasoning prompt. Evaluations demonstrate that the incremental reasoning prompt outperforms other strategies such as prompt chaining and eliminative reasoning in certain scenarios. Greedy decoding with incremental reasoning method shows superior performance compared to other decoding strategies. Overall, these advancements showcase the potential of Large Language Models in transforming healthcare delivery by providing personalized information, improving decision-making processes, and ultimately leading to better health outcomes for patients.