# DERA: Enhancing Large Language Model Completions with Dialog-Enabled Resolving Agents

**Authors:** Varun Nair, Elliot Schumacher, Geoffrey Tso, Anitha Kannan  
**arXiv:** [2303.17071](https://arxiv.org/abs/2303.17071)  
**Date:** 2023-03-30  
**Summary page:** https://www.summarizepaper.com/en/paper/2303.17071v1/

## Key points

- Large language models (LLMs) are valuable tools for natural language understanding tasks in safety-critical applications such as healthcare.
- The accuracy and completeness of LLMs' outputs are crucial for their utility.
- Dialog-enabled resolving agents (DERA) leverage the conversational abilities of LLMs to provide a forum for models to communicate feedback and improve output.
- DERA can be used for medical conversation summarization, which involves encapsulating patient-doctor conversations into structured summaries that accurately capture important information.
- DERA setup involves two agent types - a Researcher and a Decider - who both have access to the full medical conversation between the patient and physician.
- Human evaluation studies showed that physicians preferred DERA-generated summaries over initial GPT-4 generated summaries by 90% to 10% and captured far more clinical information than initial GPT-4 generated summaries.
- The study demonstrates the potential of DERA as a valuable tool for improving the accuracy and completeness of medical conversation summarization.

## Summary

Large language models (LLMs) have become increasingly valuable tools for natural language understanding tasks, including those in safety-critical applications such as healthcare. However, the utility of these models is dependent on their ability to generate outputs that are factually accurate and complete. To address this challenge, researchers have developed dialog-enabled resolving agents (DERA), which leverage the conversational abilities of LLMs like GPT-4 to provide a simple and interpretable forum for models to communicate feedback and iteratively improve output. One specific application of DERA is medical conversation summarization, which involves encapsulating patient-doctor conversations into structured summaries that accurately capture important information. The goal is to provide doctors with useful summaries for downstream tasks such as clinical decision-making. In this study, the researchers focused on summarizing patient-doctor chats into six independent sections: Demographics and Social Determinants of Health, Medical Intent, Pertinent Positives, Pertinent Negatives, Pertinent Unknowns, and Medical History. The DERA setup for medical conversation summarization involves two agent types - a Researcher and a Decider - who both have access to the full medical conversation between the patient and physician. The Decider generates an initial summary of the medical conversation and shares it with the Researcher. The Researcher's role is to identify any discrepancies in the summary and point them out to the Decider. The Decider then either accepts or rejects these suggestions before writing accepted suggestions to a shared scratchpad that it uses at the end of the conversation to generate the final summary. To evaluate DERA's effectiveness in generating better summaries than base GPT-4 performance, human evaluation studies were conducted with four licensed physicians on a random subset of 50 encounters from a dataset containing 500 medical encounters from a chat-based telehealth platform. Results showed that physicians preferred DERA-generated summaries over initial GPT-4 generated summaries by 90% to 10%. Additionally, DERA summaries captured far more clinical information than initial GPT-4 generated summaries. The amount of summaries containing "harmful" information dropped from 2% in the initial summary to 0% in the final DERA summary. Overall, this study demonstrates the potential of DERA as a valuable tool for improving the accuracy and completeness of medical conversation summarization. The researchers also released an open-ended MEDQA dataset at https://github.com/curai/curai-research/tree/main/DERA for further research and development. However, it is important to note that these findings are limited in number and drawn from a patient population specific to the telehealth platform so caution should be exercised when generalizing these results to other settings.