# GPT vs Human for Scientific Reviews: A Dual Source Review on Applications of ChatGPT in Science

**Authors:** Chenxi Wu, Alan John Varghese, Vivek Oommen, George Em Karniadakis  
**arXiv:** [2312.03769](https://arxiv.org/abs/2312.03769)  
**Date:** 2023-12-05  
**Summary page:** https://www.summarizepaper.com/en/arxiv-id/2312.03769v1/

## Key points

- Large Language Models (LLMs) have the potential to speed up scientific reviews by utilizing unbiased quantitative metrics, fostering cross-disciplinary connections, and pinpointing emerging trends and research gaps.
- Current LLMs lack a deep understanding of complex methodologies, struggle with evaluating innovative claims, and are unable to address ethical concerns and conflicts of interest.
- The study focused on 13 GPT-related papers from various scientific domains reviewed by both a human reviewer and SciSpace, a large language model.
- Findings revealed that 50% of SciSpace's responses to objective questions aligned with those of a human reviewer.
- GPT-4 often rated the human reviewer higher in accuracy while favoring SciSpace for structure, clarity, and completeness.
- Uninformed evaluators like GPT-3.5 and the crowd panel displayed varying preferences between SciSpace and human responses for subjective questions. The crowd panel showed a preference for human responses in these cases.
- GPT-4 rated both SciSpace and human responses equally in terms of accuracy and structure but leaned towards SciSpace for completeness.
- The study highlights strengths such as structuring information comprehensively but also points out limitations in areas requiring deep understanding of methodologies and ethical considerations.
- Further research is needed to enhance the capabilities of large language models for more effective scientific review processes across disciplines.

## Summary

In their paper titled "GPT vs Human for Scientific Reviews: A Dual Source Review on Applications of ChatGPT in Science," authors Chenxi Wu, Alan John Varghese, Vivek Oommen, and George Em Karniadakis explore the potential of Large Language Models (LLMs) in speeding up scientific reviews. These LLMs have the capability to utilize unbiased quantitative metrics, foster cross-disciplinary connections, and pinpoint emerging trends and research gaps by analyzing vast amounts of data. However, the authors note that current LLMs lack a deep understanding of complex methodologies and struggle with evaluating innovative claims. They are also unable to address ethical concerns and conflicts of interest. The study focuses on 13 GPT-related papers from various scientific domains that were reviewed by both a human reviewer and SciSpace, a large language model. The reviews were then evaluated by three different types of evaluators: GPT-3.5 (uninformed evaluator), a crowd panel, and GPT-4 (informed evaluator). The findings reveal that 50% of SciSpace's responses to objective questions align with those of a human reviewer. Interestingly, GPT-4 often rated the human reviewer higher in accuracy while favoring SciSpace for structure, clarity, and completeness. When it comes to subjective questions, uninformed evaluators like GPT-3.5 and the crowd panel displayed varying preferences between SciSpace and human responses. The crowd panel specifically showed a preference for human responses in these cases. However,GPT-4 rated both SciSpace and human responses equally in terms of accuracy and structure but leaned towards SciSpace for completeness. This study sheds light on the strengths and limitations of using large language models like SciSpace in scientific reviews. While they show promise in certain aspects such as structuring information comprehensively, there is still room for improvement in areas requiring deep understanding of methodologies and ethical considerations. Further research may help enhance the capabilities of these models for more effective scientific review processes across disciplines.