# Little Giants: Exploring the Potential of Small LLMs as Evaluation Metrics in Summarization in the Eval4NLP 2023 Shared Task

**Authors:** Neema Kotonya, Saran Krishnasamy, Joel Tetreault, Alejandro Jaimes  
**arXiv:** [2311.00686](https://arxiv.org/abs/2311.00686)  
**Date:** 2023-11-01  
**Summary page:** https://www.summarizepaper.com/en/paper/2311.00686v1/

## Key points

- Participation in the 2023 Eval4NLP shared task on evaluating prompt-based techniques for enhancing Large Language Models (LLMs) in quality estimation tasks
- Systematic experiments conducted using various prompting techniques, including standard prompting, prompts informed by annotator instructions, and innovative chain-of-thought prompting
- Integration of prompting approaches with zero-shot and one-shot learning methods to optimize evaluation procedures
- Development of a set of instructions to evaluate the quality of news article summaries based on coherence, consistency, fluency, and relevance
- Refinement of prompts through manual prompt rewriting, instruction enhancement via LLMs, and prompt refinement through paraphrasing
- Experimentation with prompts instructing LLMs to output scores and explanations, but poor results obtained
- Utilization of Codalab as the platform for submitting system entries for evaluation purposes
- Use of Kendall rank coefficient as an evaluation metric due to its suitability for situations with specific assumptions or small sample sizes
- Employment of Core Prompts, prompt refinement, and further prompt refinement strategies to enhance prompt effectiveness and interpretability
- Combination of approaches using a "small" open-source model (orca_mini_v3_7B) resulting in competitive results

## Summary

This paper describes our participation in the 2023 Eval4NLP shared task, which focuses on evaluating prompt-based techniques to enhance the performance of Large Language Models (LLMs) in quality estimation tasks, specifically for machine translations and summaries. We conducted systematic experiments using various prompting techniques, including standard prompting, prompts informed by annotator instructions, and innovative chain-of-thought prompting. Additionally, we integrated these approaches with zero-shot and one-shot learning methods to optimize our evaluation procedures. To evaluate the quality of news article summaries based on coherence, consistency, fluency, and relevance for each sentence in the summary with respect to the article, we developed a set of instructions. Each aspect was scored on a scale from 1 (worst) to 5 (best). We refined these prompts using three key strategies: manual prompt rewriting; instruction enhancement via LLMs; and prompt refinement through paraphrasing. In manual prompt rewriting, we meticulously rewrote the instructions to elicit fine-grained answers and employed templates specifying the desired answer format. We also experimented with prompts that instructed the LLM to output both scores and explanations; however, we found that prompting for explanations alongside quality estimation yielded poor results. In instruction enhancement via LLMs strategy, we provided a seed prompt as context and prompted a separate LLM to enhance the existing instructions. Various phrases such as "Improve the following instructions" or "Rewrite the following instructions to yield better responses" were used. For evaluation purposes, we utilized Codalab as the platform for submitting our system entries. The organizers of the shared task provided direct assessment baselines for LLMs as reference points for evaluating system performance. The Kendall rank coefficient was used as an evaluation metric due to its suitability for situations where data does not meet specific assumptions or when dealing with small sample sizes. We employed three main classes of strategies to enhance prompt effectiveness and interpretability: Core Prompts; prompt refinement; and further prompt refinement. Core Prompts involved one-step methods for generating prompts while prompt refinement focused on manual and automatic methods to refine them. Finally, further prompt refinement outlined two simple approaches to improve generated prompts. Overall our experiments showed that combining these approaches using a "small" open-source model (orca_mini_v3_7B) resulted in competitive results. We believe that our findings contribute to the field of quality estimation and demonstrate the effectiveness of prompt-based techniques in empowering LLMs for handling such tasks.