# Synthesizing Human Gaze Feedback for Improved NLP Performance

**Authors:** Varun Khurana, Yaman Kumar Singla, Nora Hollenstein, Rajesh Kumar, Balaji Krishnamurthy  
**arXiv:** [2302.05721](https://arxiv.org/abs/2302.05721)  
**Date:** 2023-02-11  
**Summary page:** https://www.summarizepaper.com/en/arxiv-id/2302.05721v1/

## Key points

- Human feedback improves the performance of natural language processing (NLP) models
- Feedback can be explicit or implicit
- Human gaze patterns aid in understanding and performance of NLP models
- ScanTextGAN is a model for generating human scanpaths over text to address challenges of collecting real eyetracking data for NLP tasks
- Generated scanpaths can approximate meaningful cognitive signals in human gaze patterns
- Synthetically generated scanpaths improve the performance of all downstream NLP tasks
- This study presents a promising solution to the challenge of collecting real eyetracking data for NLP tasks
- Integrating synthetic feedback can enhance the performance of machine learning models
- Potential applications beyond NLP tasks include improving user experience design and website optimization

## Summary

The integration of human feedback in natural language processing (NLP) models has been shown to improve their performance. This feedback can be explicit, such as ranking used in training language models, or implicit, such as using human cognitive signals in the form of eyetracking. Previous research has revealed that cognitive processes gleaned from human gaze patterns, such as human scanpaths, aid in the understanding and performance of NLP models. To address the challenge of collecting real eyetracking data for NLP tasks due to expensive and precise equipment coupled with privacy invasion issues, a team of researchers led by Varun Khurana proposed ScanTextGAN - a novel model for generating human scanpaths over text. The researchers showed that ScanTextGAN-generated scanpaths can approximate meaningful cognitive signals in human gaze patterns and included synthetically generated scanpaths in four popular NLP tasks spanning six different datasets as proof of concept. They demonstrated that the models augmented with generated scanpaths improve the performance of all downstream NLP tasks. The authors note that while previous studies have explored using synthetic data to augment machine learning models, they are not aware of any prior work on generating synthetic eye-tracking data for NLP tasks. They also highlight potential applications beyond NLP tasks such as improving user experience design and website optimization. Overall, this study presents a promising solution to the challenge of collecting real eyetracking data for NLP tasks and demonstrates how integrating synthetic feedback can enhance the performance of machine learning models.