# ChatGPT-Crawler: Find out if ChatGPT really knows what it's talking about

**Authors:** Aman Rangapur, Haoran Wang  
**arXiv:** [2304.03325](https://arxiv.org/abs/2304.03325)  
**Date:** 2023-04-06  
**Summary page:** https://www.summarizepaper.com/en/paper/2304.03325v1/

## Key points

- Large language models like ChatGPT developed by OpenAI have impressive performance on various tasks
- Early adopters regard it as a disruptive technology in fields such as customer service, education, healthcare, and finance
- Previous studies found that ChatGPT performs well on most jobs but struggles on low-resource activities and fine-grained downstream tasks like sequence tagging
- Ethical considerations are being explored regarding human-computer interaction (HCI), education, medical applications, and writing
- This research specifically examines the responses generated by ChatGPT from different Conversational QA corpora which mimic human conversation with elements such as small talk, humor, and emotion
- The study employed BERT similarity scores to compare these responses with correct answers and obtain Natural Language Inference (NLI) labels for evaluation
- Findings suggest that ChatGPT has strengths in understanding context and handling natural language while being flexible enough to handle a wide variety of topics and questions
- However, its lack of specific knowledge on certain topics can lead to inaccurate responses along with its difficulty in understanding ambiguous or unclear questions or statements resulting in inaccurate or nonsensical responses.
- The research also conducted a case study comparing GPT-3 & GPT-4's performance using different evaluation metrics to measure various aspects of text generation.
- The study found that GPT-4 was significantly enhanced compared to GPT-3 when given a context.

## Summary

Large language models, such as ChatGPT developed by OpenAI, have gained significant interest for their impressive performance on various tasks. Early adopters regard it as a disruptive technology in many fields like customer service, education, healthcare, and finance. It is essential to comprehend the opinions of these initial users to provide valuable insights into the potential strengths, weaknesses, and success or failure of the technology in different areas. Previous studies have assessed ChatGPT's performance on various tasks and found that while it performs well on most jobs, it struggles on low-resource activities and fine-grained downstream tasks like sequence tagging. Additionally, ethical considerations are being explored regarding human-computer interaction (HCI), education, medical applications and writing. This research specifically examines the responses generated by ChatGPT from different Conversational QA corpora. Conversational QA corpora aim to mimic human conversation with elements such as small talk, humor and emotion. This makes it more challenging for chatbots to reply since they need to understand not only the literal meaning of words but also context tone and intent behind them. The study employed BERT similarity scores to compare these responses with correct answers and obtain Natural Language Inference (NLI) labels. Evaluation scores were computed and compared to determine the overall performance of GPT-3 & GPT-4. The findings suggest that ChatGPT has strengths in understanding context and handling natural language while being flexible enough to handle a wide variety of topics and questions. However its lack of specific knowledge on certain topics can lead to inaccurate responses along with its difficulty in understanding ambiguous or unclear questions or statements resulting in inaccurate or nonsensical responses. In addition to identifying areas where ChatGPT may be prone to error when answering questions from Conversational QA corpora through BERT similarity scores analysis; this research also conducted a case study comparing GPT-3 & GPT-4's performance using different evaluation metrics to measure various aspects of text generation. The study found that GPT-4 was significantly enhanced compared to GPT-3 when given a context.