In this supplementary report on the responsible use of technology-assisted research assessment, authors Kayvan Kousha and Mike Thelwall from the Statistical Cybermetrics and Research Evaluation Group at the University of Wolverhampton in the UK delve into various aspects of predicting journal article citation counts or quality from both article text and metadata. The report covers a range of indicators that are associated with higher impact or quality research, including factors such as article titles, abstracts, lengths, cited references, readability, number of authors, international collaborations, journal impact factors, and authors' h-index. The report highlights studies that have utilized machine learning techniques to predict citation counts or quality scores for journal articles and conference papers. It also explores evidence regarding the strength of association between bibliometric indicators and quality score rankings from previous UK Research Assessment Exercises (RAEs) and REFs in different subjects and years, as well as similar evidence from other countries like Australia and Italy. Furthermore, the report surveys studies that have used public datasets of citations, social media indicators, or open review texts from platforms such as Dimensions, OpenCitations, Altmetric.com,
and Publons to help predict the scholarly impact of articles. These findings were used to inform experiments utilizing machine learning to predict REF journal article quality scores. Additionally,the report delves into technology aimed at automating editorial processes for providing quality control for papers and reviewers' suggestions. It also discusses tools designed to match reviewers with articles efficiently and automatically categorize journal articles into specific fields. The report addresses issues related to bias and transparency in technology-assisted assessment processes. Overall,this detailed summary provides a comprehensive overview of how technology is being leveraged to support research assessment practices while emphasizing responsible usage considerations in evaluating scholarly work.
- Authors Kayvan Kousha and Mike Thelwall from the University of Wolverhampton delve into predicting journal article citation counts or quality using article text and metadata
- Indicators associated with higher impact research include article titles, abstracts, cited references, readability, number of authors, international collaborations, journal impact factors, and authors' h-index
- Machine learning techniques are used to predict citation counts or quality scores for journal articles and conference papers
- Evidence shows the association between bibliometric indicators and quality score rankings in UK Research Assessment Exercises (RAEs) and REFs as well as in other countries like Australia and Italy
- Public datasets from platforms like Dimensions, OpenCitations, Altmetric.com, and Publons are used to predict scholarly impact of articles
- Technology is utilized to automate editorial processes for quality control and reviewer suggestions
- Tools are designed to efficiently match reviewers with articles and categorize journal articles into specific fields
- The report addresses issues related to bias and transparency in technology-assisted assessment processes.
- Authors from a university study how to predict how popular or good a research article is using its text and information.
- Important signs of successful research include the title, summary, references, readability, authors, collaborations, journal importance, and authors' reputation.
- Computers are used to guess how many times an article will be mentioned or how good it is.
- Proof shows that certain signs can tell us if an article is good in different countries.
- Data from websites help us guess how important articles are.
Definitions
- Authors: People who write books or articles.
- Predict: To guess what will happen in the future.
- Citation counts: How many times an article is mentioned by others.
- Quality: How good something is compared to others.
- Metadata: Information about something's details.
- Machine learning techniques: Using computers to learn and make predictions without being programmed directly.
- Scholarly impact: The influence or importance of academic work on others in the field.
- Bias: Unfair preferences towards one thing over another.
- Transparency: Being clear and open about how things work.
Introduction
In the world of academia, research assessment is a crucial aspect that helps determine the quality and impact of scholarly work. Traditionally, this process has been carried out manually by experts in the field. However, with the advancement of technology, there has been an increasing trend towards using technology-assisted methods for research assessment. In this supplementary report, authors Kayvan Kousha and Mike Thelwall from the University of Wolverhampton's Statistical Cybermetrics and Research Evaluation Group delve into various aspects of predicting journal article citation counts or quality using technology-assisted methods.
Overview of Technology-Assisted Research Assessment
The report covers a range of indicators that are associated with higher impact or quality research. These include factors such as article titles, abstracts, lengths, cited references, readability, number of authors, international collaborations,journal impact factors,and authors' h-index. The report highlights studies that have utilized machine learning techniques to predict citation counts or quality scores for journal articles and conference papers.
Furthermore,the report explores evidence regarding the strength of association between bibliometric indicators and quality score rankings from previous UK Research Assessment Exercises (RAEs)and REFs in different subjects and years. It also examines similar evidence from other countries like Australia and Italy.
Use of Public Datasets
One interesting aspect covered in this report is the use of public datasets to help predict scholarly impact. This includes data from citations,social media indicators,and open review texts from platforms such as Dimensions,OpenCitations,Altmetric.com,and Publons. These datasets provide valuable insights into how technology can be leveraged to support research assessment practices.
Experiments Utilizing Machine Learning
Based on these findings,the authors conducted experiments utilizing machine learning to predict REF journal article quality scores.This demonstrates how advanced technologies can be used to automate certain aspects of research assessment, making the process more efficient and accurate.
Technology for Editorial Processes
The report also delves into technology aimed at automating editorial processes for providing quality control for papers and reviewers' suggestions. This includes tools designed to match reviewers with articles efficiently and automatically categorize journal articles into specific fields. These technologies not only save time but also help reduce bias in the review process.
Responsible Usage Considerations
While technology-assisted research assessment has many benefits, it is essential to consider responsible usage practices. The report addresses issues related to bias and transparency in these processes, emphasizing the need for ethical considerations when utilizing technology in research assessment.
Conclusion
In conclusion, this supplementary report provides a comprehensive overview of how technology is being leveraged to support research assessment practices. It covers various indicators associated with higher impact or quality research, experiments utilizing machine learning techniques,and the use of public datasets to predict scholarly impact. Additionally,it discusses technologies aimed at automating editorial processes and highlights responsible usage considerations in technology-assisted assessment methods. Overall,this report sheds light on the potential of technology in improving research assessment while emphasizing ethical considerations that must be taken into account when utilizing these methods.
