# Automatic Evaluation of Healthcare LLMs Beyond Question-Answering

**Authors:** Anna Arias-Duart, Pablo Agustin Martin-Torres, Daniel Hinjos, Pablo Bernabeu-Perez, Lucia Urcelay Ganzabal, Marta Gonzalez Mallo, Ashwin Kumar Gururajan, Enrique Lopez-Cuena, Sergio Alvarez-Napagao, Dario Garcia-Gasulla  
**arXiv:** [2502.06666](https://arxiv.org/abs/2502.06666)  
**Date:** 2025-02-10  
**Summary page:** https://www.summarizepaper.com/en/paper/2502.06666v1/

## Key points

- Paper title: "Automatic Evaluation of Healthcare LLMs Beyond Question-Answering"
- Authors: Anna Arias-Duart, Pablo Agustin Martin-Torres, Daniel Hinjos, Pablo Bernabeu-Perez, Lucia Urcelay Ganzabal, Marta Gonzalez Mallo, Ashwin Kumar Gururajan, Enrique Lopez-Cuena, Sergio Alvarez-Napagao, and Dario Garcia-Gasulla
- Focus on evaluating Large Language Models (LLMs) in the healthcare domain
- Importance of factuality and discourse in healthcare applications
- Introduction of a multi-axis suite for evaluating healthcare LLMs
- Exploration of correlations between open and close benchmarks and metrics to identify blind spots and overlaps in current evaluation methodologies
- Introduction of a new medical benchmark called CareQA for holistic assessment of healthcare LLM performance
- Proposal of a novel metric called Relaxed Perplexity for evaluating open-ended responses
- Aim to enhance understanding of evaluating healthcare LLMs beyond traditional question-answering tasks
- Findings offer valuable insights for improving evaluation process and advancing development of language models tailored for healthcare applications

## Summary

The paper titled "Automatic Evaluation of Healthcare LLMs Beyond Question-Answering" by Anna Arias-Duart, Pablo Agustin Martin-Torres, Daniel Hinjos, Pablo Bernabeu-Perez, Lucia Urcelay Ganzabal, Marta Gonzalez Mallo, Ashwin Kumar Gururajan, Enrique Lopez-Cuena, Sergio Alvarez-Napagao, and Dario Garcia-Gasulla delves into the evaluation of Large Language Models (LLMs) in the healthcare domain. The authors highlight the importance of considering both factuality and discourse in healthcare applications and introduce a comprehensive multi-axis suite for evaluating healthcare LLMs. By exploring correlations between open and close benchmarks and metrics, they identify blind spots and overlaps in current evaluation methodologies. As part of their study, they introduce a new medical benchmark called CareQA to provide a more holistic assessment of healthcare LLM performance. Additionally, they propose a novel metric called Relaxed Perplexity for evaluating open-ended responses. Through their research, the authors aim to enhance our understanding of how best to evaluate healthcare LLMs beyond traditional question-answering tasks. The findings presented offer valuable insights for improving the evaluation process and advancing the development of language models tailored specifically for healthcare applications.