Results of the summarizing process for the arXiv paper: 2112.08608v1

Introduce a new multiple-choice QA dataset called QuALITY to enable building and testing models on long-document comprehension. The context passages in English have an average length of about 5,000 tokens, which is much longer than typical current models can process. Unlike prior work with passages, the questions are written and validated by contributors who have read the entire passage instead of relying on summaries or excerpts. The authors found that only half of the questions are answerable by annotators working under tight time constraints, indicating that skimming and simple search are not enough to consistently perform well. Current models perform poorly on this task (55.4%) and significantly lag behind human performance (93.5%). The paper's authors include Richard Yuanzhe Pang, Alicia Parrish, Nitish Joshi, Nikita Nangia, Jason Phang, Angelica Chen, Vishakh Padmakumar, Johnny Ma, Jana Thompson, He He and Samuel R. Bowman.

• Introduction of a new multiple-choice QA dataset called QuALITY
• Enables building and testing models on long-document comprehension
• Context passages in English have an average length of about 5,000 tokens
• Questions are written and validated by contributors who have read the entire passage
• Only half of the questions are answerable by annotators working under tight time constraints
• Current models perform poorly on this task (55.4%)
• Significantly lag behind human performance (93.5%)
• Authors include Richard Yuanzhe Pang, Alicia Parrish, Nitish Joshi, Nikita Nangia, Jason Phang, Angelica Chen, Vishakh Padmakumar, Johnny Ma, Jana Thompson, He He and Samuel R. Bowman

There is a new test called QuALITY for computers to read and answer questions about long passages. The passages are about 5,000 words long in English. People write the questions and make sure they are correct. Some of the questions are hard to answer quickly. Right now, computers do not do as well as people on this test. The people who made this test are Richard Yuanzhe Pang, Alicia Parrish, Nitish Joshi, Nikita Nangia, Jason Phang, Angelica Chen, Vishakh Padmakumar, Johnny Ma, Jana Thompson, He He and Samuel R. Bowman. Definitions- Multiple-choice: A type of question where you choose an answer from a list of options. - Dataset: A collection of information or data. - Comprehension: Understanding something that you have read or heard. - Passages: A piece of writing or text. - Annotators: People who add notes or comments to something they have read or seen. - Performance: How well someone or something does at a task or activity.

Introducing QuALITY: A New Multiple-Choice QA Dataset for Long-Document Comprehension

Researchers have recently introduced a new multiple-choice QA dataset called QuALITY to enable building and testing models on long-document comprehension. The context passages in English have an average length of about 5,000 tokens, which is much longer than typical current models can process. This paper was written by Richard Yuanzhe Pang, Alicia Parrish, Nitish Joshi, Nikita Nangia, Jason Phang, Angelica Chen, Vishakh Padmakumar, Johnny Ma, Jana Thompson, He He and Samuel R. Bowman.

What Makes QuALITY Different?

QuALITY differs from prior work with passages in that the questions are written and validated by contributors who have read the entire passage instead of relying on summaries or excerpts. The authors found that only half of the questions are answerable by annotators working under tight time constraints; this indicates that skimming and simple search are not enough to consistently perform well on this task.

How Well Do Current Models Perform?

Current models perform poorly on this task (55.4%) and significantly lag behind human performance (93.5%). This suggests there is still room for improvement when it comes to developing more accurate models for long document comprehension tasks such as those used in the QuALITY dataset.

Conclusion

The introduction of the QuALITY dataset provides researchers with a valuable tool for testing their models’ ability to comprehend longer documents accurately and efficiently. With its unique approach to question generation based on full reading of passages rather than summaries or excerpts alone, it offers an opportunity to push current model performance beyond what has been achieved so far in terms of accuracy and speed when dealing with longer documents