, , , ,
In this work, the authors evaluate the performance of instruction-following models in retrieval-augmented settings for question answering (QA) tasks. These models, such as Llama-2, GPT-3.5, Flan-T5, and Alpaca, are attractive alternatives to fine-tuned approaches as they can be adapted to various information domains without additional fine-tuning. The authors investigate these models across three diverse QA tasks and use both automatic and human evaluation to assess their correctness (how well they satisfy the user's information need) and faithfulness (whether they produce a response based on the provided knowledge). They highlight the shortcomings of traditional QA metrics like exact match (EM) and F1 for accurately quantifying model performance due to the additional verbosity introduced by retrieved documents. To address this issue, the authors propose simple token-overlap based and model-based metrics that reflect the true performance of these models. Their analysis reveals that instruction-following models are competitive and sometimes even outperform fine-tuned models in terms of correctness. However, these models struggle to adhere to the provided knowledge and often generate responses that contain hallucinations. The authors hope that their work encourages a more holistic evaluation of instruction-following models for QA tasks.
- Authors evaluate performance of instruction-following models in retrieval-augmented settings for QA tasks
- Models include Llama-2, GPT-3.5, Flan-T5, and Alpaca
- Investigation done across three diverse QA tasks
- Automatic and human evaluation used to assess correctness and faithfulness of models
- Traditional QA metrics like exact match (EM) and F1 found inadequate due to verbosity introduced by retrieved documents
- Proposed simple token-overlap based and model-based metrics to address this issue
- Instruction-following models shown to be competitive and sometimes outperform fine-tuned models in terms of correctness
- However, these models struggle with adherence to provided knowledge and generate responses with hallucinations
- Authors encourage holistic evaluation of instruction-following models for QA tasks
In this study, the authors looked at how well different models can follow instructions to answer questions. They tested four models called Llama-2, GPT-3.5, Flan-T5, and Alpaca. They did this by giving the models three different types of questions to answer. They used both automatic and human evaluation to see if the models were correct and faithful in their answers. The usual ways of measuring correctness didn't work well because the retrieved documents made the answers too long. So they came up with new ways to measure how well the models did. The instruction-following models were shown to be good at getting the right answers, but sometimes they made things up instead of using what they knew. The authors think that we should look at these models in a more complete way when evaluating them for answering questions."
Definitions
- Authors: People who wrote a book or article
- Models: Different ways of doing something
- Evaluation: Checking how good or bad something is
- Correctness: Being right or accurate
- Faithfulness: Staying true to something
Introduction
Question answering (QA) is a fundamental task in natural language processing (NLP) that aims to automatically answer questions posed in natural language. With the increasing availability of large-scale pre-trained models, such as BERT and GPT-3, there has been significant progress in QA systems. However, these models often require fine-tuning on specific datasets for optimal performance, making them less adaptable to new domains or languages.
Instruction-following models have emerged as an alternative approach to traditional fine-tuned models. These models can be adapted to various information domains without additional fine-tuning and have shown promising results in tasks like machine translation and text summarization. In this research paper, "Evaluating Instruction-Following Models for Retrieval-Augmented Question Answering," the authors evaluate the performance of instruction-following models in retrieval-augmented settings for QA tasks.
Methodology
The authors use four instruction-following models: Llama-2, GPT-3.5, Flan-T5, and Alpaca. They compare these models against two baseline approaches - a simple TF-IDF retrieval model and a fine-tuned BERT model - across three diverse QA tasks: open-domain factoid QA (OpenQA), closed-book commonsense reasoning (CommonsenseQA), and multi-hop science question answering (ScienceQA).
To assess the correctness of these models - how well they satisfy the user's information need - the authors use traditional metrics like exact match (EM) and F1 scores. However, they highlight that these metrics are not suitable for evaluating instruction-following models due to their reliance on strict string matching between generated answers and ground truth answers.
To address this issue, the authors propose two novel metrics: token-overlap based metric (TOM) and model-based metric (MBM). TOM measures how many tokens from retrieved documents are present in the generated answer, while MBM uses a pre-trained language model to score the relevance of retrieved documents to the question. These metrics provide a more accurate reflection of model performance by considering the additional verbosity introduced by retrieved documents.
Results
The authors' analysis reveals that instruction-following models perform competitively and sometimes even outperform fine-tuned models in terms of correctness. For OpenQA and CommonsenseQA tasks, TOM scores show that these models generate answers with high token overlap with retrieved documents. However, for ScienceQA, TOM scores are lower due to the multi-hop nature of the task.
On the other hand, MBM scores highlight a major weakness of instruction-following models - their lack of faithfulness. These models often produce responses that contain hallucinations - information not present in any retrieved document but still deemed relevant by the model. This issue is particularly evident in ScienceQA, where Alpaca generates significantly more hallucinations than other models.
Conclusion
In conclusion, this research paper provides a comprehensive evaluation of instruction-following models for retrieval-augmented QA tasks. The authors' proposed metrics - TOM and MBM - offer a more accurate representation of model performance compared to traditional metrics like EM and F1. Their analysis shows that these models are competitive with fine-tuned approaches in terms of correctness but struggle with faithfulness.
This work highlights the need for a more holistic evaluation approach when assessing instruction-following models for QA tasks. Future research could focus on improving these models' faithfulness through techniques like knowledge distillation or incorporating external knowledge sources into their training process.
Overall, this research contributes valuable insights into understanding and evaluating instruction-following models for QA tasks and paves the way for further advancements in this field.
