# Evaluating Correctness and Faithfulness of Instruction-Following Models for Question Answering

**Authors:** Vaibhav Adlakha, Parishad BehnamGhader, Xing Han Lu, Nicholas Meade, Siva Reddy  
**arXiv:** [2307.16877](https://arxiv.org/abs/2307.16877)  
**Date:** 2023-07-31  
**Summary page:** https://www.summarizepaper.com/en/paper/2307.16877v1/

## Key points

- Authors evaluate performance of instruction-following models in retrieval-augmented settings for QA tasks
- Models include Llama-2, GPT-3.5, Flan-T5, and Alpaca
- Investigation done across three diverse QA tasks
- Automatic and human evaluation used to assess correctness and faithfulness of models
- Traditional QA metrics like exact match (EM) and F1 found inadequate due to verbosity introduced by retrieved documents
- Proposed simple token-overlap based and model-based metrics to address this issue
- Instruction-following models shown to be competitive and sometimes outperform fine-tuned models in terms of correctness
- However, these models struggle with adherence to provided knowledge and generate responses with hallucinations
- Authors encourage holistic evaluation of instruction-following models for QA tasks

## Summary

, , , , 

In this work, the authors evaluate the performance of instruction-following models in retrieval-augmented settings for question answering (QA) tasks. These models, such as Llama-2, GPT-3.5, Flan-T5, and Alpaca, are attractive alternatives to fine-tuned approaches as they can be adapted to various information domains without additional fine-tuning. The authors investigate these models across three diverse QA tasks and use both automatic and human evaluation to assess their correctness (how well they satisfy the user's information need) and faithfulness (whether they produce a response based on the provided knowledge). They highlight the shortcomings of traditional QA metrics like exact match (EM) and F1 for accurately quantifying model performance due to the additional verbosity introduced by retrieved documents. To address this issue, the authors propose simple token-overlap based and model-based metrics that reflect the true performance of these models. Their analysis reveals that instruction-following models are competitive and sometimes even outperform fine-tuned models in terms of correctness. However, these models struggle to adhere to the provided knowledge and often generate responses that contain hallucinations. The authors hope that their work encourages a more holistic evaluation of instruction-following models for QA tasks.