# GPTScore: Evaluate as You Desire

**Authors:** Jinlan Fu, See-Kiong Ng, Zhengbao Jiang, Pengfei Liu  
**arXiv:** [2302.04166](https://arxiv.org/abs/2302.04166)  
**Date:** 2023-02-08  
**Summary page:** https://www.summarizepaper.com/en/paper/2302.04166v1/

## Key points

- Generative AI has advanced the development of sophisticated models for generating high-quality text, images, and other outputs.
- Evaluating the quality of generated outputs is a challenging task that has not received enough attention.
- The paper introduces a novel evaluation framework called GPTScore.
- GPTScore leverages generative pre-trained models to score generated texts based on natural language instructions.
- This approach allows customized and multi-faceted evaluation without annotated samples.
- Experiments were conducted on four text generation tasks, evaluating 22 different aspects across 37 datasets.
- Results demonstrate that GPTScore effectively enables users to achieve desired evaluations for texts.
- The framework provides a flexible and intuitive way to assess the quality of generated texts using natural language instructions.
- The code for GPTScore is publicly available on GitHub at https://github.com/jinlanfu/GPTScore.
- This framework opens up new possibilities for evaluating generative AI outputs and contributes to advancing text evaluation.

## Summary

Generative Artificial Intelligence (AI) has advanced the development of sophisticated models that can generate high-quality text, images, and other outputs using large pre-trained models. Evaluating the quality of these generated outputs is a more challenging task than the generation process itself, and it has not received enough attention in recent times. To address this issue, this paper introduces a novel evaluation framework called GPTScore. GPTScore leverages the emerging capabilities of generative pre-trained models, such as zero-shot instruction, to score generated texts. It allows users to evaluate texts based on their desired criteria simply by providing natural language instructions. This approach overcomes long-standing challenges in text evaluation by enabling customized and multi-faceted evaluation without the need for annotated samples. The authors conducted experiments on four text generation tasks, evaluating 22 different aspects across 37 datasets. The results demonstrate that GPTScore effectively enables users to achieve their desired evaluations for texts. By utilizing natural language instructions, this framework provides a flexible and intuitive way to assess the quality of generated texts. The code for GPTScore is publicly available on GitHub at https://github.com/jinlanfu/GPTScore. The authors emphasize that this framework opens up new possibilities for evaluating generative AI outputs and contributes to advancing the field of text evaluation.