# ANLS* -- A Universal Document Processing Metric for Generative Large Language Models

**Authors:** David Peer, Philemon Schöpf, Volckmar Nebendahl, Alexander Rietzler, Sebastian Stabinger  
**arXiv:** [2402.03848](https://arxiv.org/abs/2402.03848)  
**Date:** 2024-02-06  
**Summary page:** https://www.summarizepaper.com/en/paper/2402.03848v1/

## Key points

- Discriminative models like LayoutLMv3 have limitations in tasks requiring text synthesis, translation, or enhancement as they lack token generation ability.
- Generative large language models (GLLMs) offer enhanced zero-shot capabilities and revolutionize document processing tasks.
- The ANLS* metric has been introduced for evaluating generative models across various tasks such as information extraction and classification.
- SFT, a novel approach for generating prompts in documents, outperformed other techniques like LATIN in an extensive comparison across 21 cases.
- Evolving evaluation metrics like ANLS* alongside innovative prompting strategies can significantly enhance document processing capabilities.

## Summary

, , , , 

In the realm of document processing tasks, discriminative models like LayoutLMv3 have made significant strides in advancing the state-of-the-art. However, these models still face limitations when it comes to tasks requiring text synthesis, translation, or enhancement as they lack the ability to generate tokens and typically only label them. For instance, tasks that involve extracting date-time information and converting it into a specific format pose a challenge for discriminative models. This is where generative large language models (GLLMs) come into play. The emergence of GLLMs has revolutionized the field by offering enhanced zero-shot capabilities that eliminate the need for downstream datasets and costly fine-tuning processes. While traditional discriminative models excel at making predictions within predefined classes for binary true or false evaluations, evaluating GLLMs presents a unique challenge due to their generative nature. To address this issue, a new metric called ANLS* has been introduced specifically tailored for evaluating generative models across various tasks such as information extraction and classification. The ANLS* metric builds upon existing ANLS metrics as a drop-in replacement while remaining compatible with previously reported scores. An evaluation involving 7 diverse datasets and 3 different GLLMs using the ANLS* metric showcases its significance in assessing model performance accurately. Furthermore, a novel approach named SFT for generating prompts in documents has been benchmarked against other techniques like LATIN. In an extensive comparison across 21 cases, SFT outperformed other methods in 15 instances, even surpassing the state-of-the-art by up to 15 percentage points. This demonstrates the efficacy of leveraging GLLMs alongside innovative prompting strategies to enhance document processing tasks significantly. Overall, this research spearheaded by David Peer et al., with contributions from Philemon Schöpf, Volckmar Nebendahl, Alexander Rietzler, and Sebastian Stabinger underlines the importance of evolving evaluation metrics like ANLS* in tandem with cutting-edge techniques like SFT to push the boundaries of document processing capabilities further.