Results of the summarizing process for the arXiv paper: 2301.13848v1

Large language models (LLMs) have shown great potential in automatic summarization, but the reasons behind their success are not well understood. To address this issue, a human evaluation was conducted on ten LLMs across different pretraining methods, prompts, and model scales. The study made two important observations: first, instruction tuning is the key to the LLM's zero-shot summarization capability rather than model size; second, existing studies have been limited by low-quality references leading to underestimates of human performance and lower few-shot and fine-tuning performance. To better evaluate LLMs, high-quality summaries were collected from freelance writers for human evaluation. To ensure consistency in summary length between the freelance writer summaries and those generated by Instruct Davinci model, a new prompt was introduced that elicited summaries around 50 words long. The quality of the freelance writer summaries was evaluated using Mechanical Turkers and found to be much higher than the original reference summaries in CNN/DM and XSUM. Additionally, there was little difference between the quality of the freelance writer summaries and those generated by Instruct Davinci. Despite similar performance in quality control studies, LLM-generated summaries and freelance writer-generated summaries had distinctive styles with regard to paraphrasing and copying from source articles. Extractiveness measures were used to compare coverage and density between these two types of summaries. Annotators were recruited to compare Instruct Davinci-generated summaries with those written by freelance writers. On aggregate, Instruct Davinci was rated as comparable to freelance writers; however, individual annotators showed varying preferences for either Instruct Davinci or freelance writers. Overall, this study identified instruction tuning as crucial for LLMs' summarization capability rather than model scale. It also highlighted issues with low-quality references used in previous studies and proposed collecting better quality summaries from freelance writers as a solution. These findings contribute towards improving LLM evaluation techniques for future research in automatic summarization.

• - Large language models (LLMs) have potential in automatic summarization
• - Human evaluation conducted on 10 LLMs across different pretraining methods, prompts, and model scales
• - Instruction tuning is key to LLM's zero-shot summarization capability rather than model size
• - Existing studies limited by low-quality references leading to underestimates of human performance and lower few-shot and fine-tuning performance
• - High-quality summaries collected from freelance writers for human evaluation
• - Quality of freelance writer summaries evaluated using Mechanical Turkers found to be much higher than original reference summaries in CNN/DM and XSUM
• - Little difference between quality of freelance writer summaries and those generated by Instruct Davinci model
• - LLM-generated summaries and freelance writer-generated summaries had distinctive styles with regard to paraphrasing and copying from source articles
• - Annotators recruited to compare Instruct Davinci-generated summaries with those written by freelance writers; overall, Instruct Davinci rated as comparable to freelance writers but individual annotators showed varying preferences
• - Instruction tuning crucial for LLMs' summarization capability rather than model scale
• - Issues with low-quality references used in previous studies highlighted
• - Proposed collecting better quality summaries from freelance writers as a solution
• - Findings contribute towards improving LLM evaluation techniques for future research in automatic summarization.

Summary: Large language models (LLMs) can help summarize text. People tested 10 different LLMs to see how well they worked. They found that tuning the instructions is more important than making the model bigger. Some previous studies had bad examples to compare with, so they weren't accurate. Freelance writers made better summaries than the bad examples. The Instruct Davinci model also made good summaries, but it was a little different from what people wrote. Definitions: - Large language models (LLMs): computer programs that can understand and generate human language - Automatic summarization: using a computer program to create a short summary of a longer piece of text - Pretraining methods: ways of teaching an LLM how to understand language before it starts summarizing - Zero-shot summarization: when an LLM creates a summary without being specifically trained on that topic - Few-shot and fine-tuning performance: how well an LLM can make summaries after being trained on just a few examples or after being adjusted slightly

Exploring the Success of Large Language Models in Automatic Summarization

The development of large language models (LLMs) has revolutionized natural language processing and enabled machines to generate human-like summaries. However, the reasons behind their success are not well understood. To address this issue, a recent study conducted a human evaluation on ten LLMs across different pretraining methods, prompts, and model scales. The results of this evaluation revealed two important observations about LLMs: instruction tuning is the key to the LLM's zero-shot summarization capability rather than model size; and existing studies have been limited by low-quality references leading to underestimates of human performance and lower few-shot and fine-tuning performance.

Collecting High Quality Summaries from Freelance Writers

To better evaluate LLMs, high-quality summaries were collected from freelance writers for human evaluation. To ensure consistency in summary length between the freelance writer summaries and those generated by Instruct Davinci model, a new prompt was introduced that elicited summaries around 50 words long. The quality of the freelance writer summaries was evaluated using Mechanical Turkers and found to be much higher than the original reference summaries in CNN/DM and XSUM datasets. Additionally, there was little difference between the quality of the freelance writer summaries and those generated by Instruct Davinci.

Distinctive Styles Between Freelance Writer Summaries & LLM Generated Summaries

Despite similar performance in quality control studies, it was observed that there were distinctive styles with regard to paraphrasing and copying from source articles between LLM-generated summaries and freelance writer-generated ones. Extractiveness measures were used to compare coverage density between these two types of summaries as well as recruit annotators who compared Instruct Davinci-generated summaries with those written by freelance writers. On aggregate, Instruct Davinci was rated as comparable to freelance writers; however individual annotators showed varying preferences for either Instruct Davinci or freelance writers when comparing them side by side..

Conclusion

Overall, this study identified instruction tuning as crucial for LLMs' summarization capability rather than model scale. It also highlighted issues with low-quality references used in previous studies which led researchers into underestimating human performance levels when evaluating automatic summarization systems based on such references - proposing collecting better quality reference material from freelancers instead as an effective solution towards improving future research efforts within this field . These findings contribute towards improving LLM evaluation techniques for future research in automatic summarization while providing valuable insights into how best utilize large language models for generating accurate yet concise machine generated text outputs that can rival even professional level writing standards at times!