# Quality of Answers of Generative Large Language Models vs Peer Patients for Interpreting Lab Test Results for Lay Patients: Evaluation Study

**Authors:** Zhe He, Balu Bhasuran, Qiao Jin, Shubo Tian, Karim Hanna, Cindy Shavor, Lisbeth Garcia Arguello, Patrick Murray, Zhiyong Lu  
**arXiv:** [2402.01693](https://arxiv.org/abs/2402.01693)  
**Date:** 2024-01-23  
**Summary page:** https://www.summarizepaper.com/en/arxiv-id/2402.01693v1/

## Key points

- Lab results can be confusing for patients, highlighting a need for accessible and accurate information.
- Large language models (LLMs) like ChatGPT are being used to provide relevant responses to lab test-related questions.
- A recent study evaluated the effectiveness of LLMs in generating responses by analyzing 53 QA pairs from Yahoo! Answers.
- Four different LLMs, including GPT-4, were assessed using standard QA evaluation metrics.
- GPT-4's responses outperformed other LLMs and human responses in terms of accuracy, helpfulness, relevance, and safety.
- However, some LLM responses lacked interpretation within a medical context or contained incorrect statements.
- The study identified areas for improvement in enhancing the quality of LLM-generated responses.

## Summary

Lab results can be confusing and difficult for patients to understand, leading to a need for accessible and accurate information. Large language models (LLMs) like ChatGPT have emerged as a promising tool to address this issue by providing relevant and helpful responses to lab test-related questions. In a recent study, researchers aimed to evaluate the effectiveness of using LLMs in generating responses to such queries from patients. The study involved collecting question and answer data related to lab test results from Yahoo! Answers and selecting 53 QA pairs for analysis. Using the LangChain framework and ChatGPT web portal, responses were generated by four different LLMs including GPT-4, Meta LLaMA 2, MedAlpaca, and ORCA_mini. The similarity of their answers was assessed using standard QA evaluation metrics such as ROUGE, BLEU, METEOR, and BERTScore. An LLM-based evaluator was also employed to determine the quality of responses in terms of relevance, correctness, helpfulness, and safety compared to baseline models. Subsequently, a manual evaluation involving medical experts was conducted on seven selected questions across all four aspects. The results indicated that GPT-4's responses outperformed other LLMs and human responses in terms of accuracy, helpfulness, relevance, and safety. However,it was noted that LLM responses occasionally lacked interpretation within a medical context or contained incorrect statements or lacked references.Despite the overall superiority of GPT-4's responses in comparison to other models and human answers from Q&A websites,the study identified several areas for improvement in enhancing the quality of LLM-generated responses. The authors of this evaluation study include Zhe He, Balu Bhasuran, Qiao Jin, Shubo Tian,Karim Hanna,Cindy Shavor,Lisbeth Garcia Arguello ,Patrick Murray,and Zhiyong Lu. The full paper can be accessed via the provided link for further details on their methodology and findings in evaluating the quality of LLM responses in interpreting lab test results for lay patients.