# How do decoding algorithms distribute information in dialogue responses?

**Authors:** Saranya Venkatraman, He He, David Reitter  
**arXiv:** [2303.17006](https://arxiv.org/abs/2303.17006)  
**Date:** 2023-03-29  
**Summary page:** https://www.summarizepaper.com/en/arxiv-id/2303.17006v1/

## Key points

- The Uniform Information Density (UID) principle is a linguistic phenomenon where humans tend to distribute information evenly in their utterances.
- The authors investigate whether decoding algorithms implicitly follow the UID principle and whether adherence to UID is desirable for dialogue generation.
- Model-generated responses follow the UID principle to a greater extent than human responses.
- Decoding algorithms that promote UID do not generate higher-quality responses.
- Non-uniformity of information density correlates with the quality of responses with very low/high surprisal, suggesting that encouraging non-uniform responses could be a potential solution to the "likelihood trap" problem.
- Instead of optimizing for uniform text, decoding algorithms should be tuned to follow the information density patterns of human-generated non-uniform data when generating responses outside of the "safe" likelihood range as a means to generate higher quality responses across the entire likelihood space.
- The study has some limitations as all machine responses are generated using the same transformers based model architecture and does not explore individual differences between different model architectures.
- Due to limited resources, large-scale human annotations across multiple corpora were not collected.
- Human annotations on dialogue response quality were collected using MTurk with no restrictions on minimum or maximum number of examples annotators had to rate.
- The payment amount was set at $0.5 per HIT for an hourly rate of about $12 per hour.
- This study provides insights into how decoding algorithms distribute information in dialogue responses and highlights potential solutions for improving response quality in natural language generation tasks.

## Summary

The Uniform Information Density (UID) principle is a linguistic phenomenon where humans tend to distribute information evenly in their utterances. In this study, the authors investigate whether decoding algorithms implicitly follow the UID principle and whether adherence to UID is desirable for dialogue generation. The authors generate responses using different decoding algorithms with GPT-2 on the Persona-Chat dataset and collect human judgments on their quality using Amazon Mechanical Turk. Surprisingly, they find that model-generated responses follow the UID principle to a greater extent than human responses. However, they also find that decoding algorithms that promote UID do not generate higher-quality responses. Instead, the authors observe that non-uniformity of information density correlates with the quality of responses with very low/high surprisal. This suggests that encouraging non-uniform responses could be a potential solution to the "likelihood trap" problem where models generate lower quality text when sampling from the extremities of their likelihood space. Therefore, instead of optimizing for uniform text, decoding algorithms should be tuned to follow the information density patterns of human-generated non-uniform data when generating responses outside of the "safe" likelihood range as a means to generate higher quality responses across the entire likelihood space. The study has some limitations as all machine responses are generated using the same transformers based model architecture and does not explore individual differences between different model architectures. Additionally, due to limited resources, large-scale human annotations across multiple corpora were not collected. In terms of ethical considerations, human annotations on dialogue response quality were collected using MTurk with no restrictions on minimum or maximum number of examples annotators had to rate. The payment amount was set at $0.5 per HIT for an hourly rate of about $12 per hour. Overall, this study provides insights into how decoding algorithms distribute information in dialogue responses and highlights potential solutions for improving response quality in natural language generation tasks.