Podcast Summary Assessment: A Resource for Evaluating Summary Assessment Methods
AI-generated Key Points
- Automatic summary assessment is valuable for evaluating machine-generated and human-produced summaries.
- It helps in developing summary generation systems and detecting inappropriate summaries.
- Summary assessment can be done by ranking summary generation systems, ranking specific document summaries, or estimating the quality of a document-summary pair on an absolute scale.
- Existing datasets for summary assessment are typically based on news summarization datasets like CNN/DailyMail or XSum.
- The podcast summary assessment corpus is a new dataset that contains long-input speech-based documents and was evaluated by human experts at TREC2020.
- This dataset is unique as it allows identification of inappropriate reference summaries within the podcast corpus.
- The authors explore existing assessment methods including model-free and model-based approaches and present benchmark results using this dataset.
- Summary assessment is also applied for data selection to filter reference summary-document pairings for training purposes.
- The experimental results provide insights into both the summary assessment and generation tasks.
- The podcast summary assessment data is publicly available.
Authors: Potsawee Manakul, Mark J. F. Gales
Abstract: Automatic summary assessment is useful for both machine-generated and human-produced summaries. Automatically evaluating the summary text given the document enables, for example, summary generation system development and detection of inappropriate summaries. Summary assessment can be run in a number of modes: ranking summary generation systems; ranking summaries of a particular document; and estimating the quality of a document-summary pair on an absolute scale. Existing datasets with annotation for summary assessment are usually based on news summarization datasets such as CNN/DailyMail or XSum. In this work, we describe a new dataset, the podcast summary assessment corpus, a collection of podcast summaries that were evaluated by human experts at TREC2020. Compared to existing summary assessment data, this dataset has two unique aspects: (i) long-input, speech podcast based, documents; and (ii) an opportunity to detect inappropriate reference summaries in podcast corpus. First, we examine existing assessment methods, including model-free and model-based methods, and provide benchmark results for this long-input summary assessment dataset. Second, with the aim of filtering reference summary-document pairings for training, we apply summary assessment for data selection. The experimental results on these two aspects provide interesting insights on the summary assessment and generation tasks. The podcast summary assessment data is available.
Ask questions about this paper to our AI assistant
You can also chat with multiple papers at once here.
Assess the quality of the AI-generated content by voting
Score: 0
Why do we need votes?
Votes are used to determine whether we need to re-run our summarizing tools. If the count reaches -10, our tools can be restarted.
The previous summary was created more than a year ago and can be re-run (if necessary) by clicking on the Run button below.
Similar papers summarized with our AI tools
Navigate through even more similar papers through a
tree representationLook for similar papers (in beta version)
By clicking on the button above, our algorithm will scan all papers in our database to find the closest based on the contents of the full papers and not just on metadata. Please note that it only works for papers that we have generated summaries for and you can rerun it from time to time to get a more accurate result while our database grows.
Disclaimer: The AI-based summarization tool and virtual assistant provided on this website may not always provide accurate and complete summaries or responses. We encourage you to carefully review and evaluate the generated content to ensure its quality and relevance to your needs.