The potential of Large Language Models (LLMs), particularly ChatGPT, to exhibit human-like psychology is still largely unexplored. While existing works have studied the virtual personalities of LLMs, they rarely explore the possibility of analyzing human personalities via LLMs. In this paper, authors Haocong Rao, Cyril Leung, and Chunyan Miao present a generic evaluation framework for LLMs to assess human personalities based on Myers Briggs Type Indicator (MBTI) tests. The proposed framework involves devising unbiased prompts by randomly permuting options in MBTI questions and adopting the average testing result to encourage more impartial answer generation. The authors also propose replacing the subject in question statements to enable flexible queries and assessments on different subjects from LLMs. Finally, they re-formulate the question instructions in a manner of correctness evaluation to facilitate LLMs to generate clearer responses. This framework enables LLMs to flexibly assess personalities of different groups of people. The authors further propose three evaluation metrics to measure the consistency, robustness, and fairness of assessment results from state-of-the-art LLMs including ChatGPT and InstructGPT. Their experiments reveal ChatGPT's ability to assess human personalities with consistent and fairer assessments despite lower robustness against prompt biases compared with InstructGPT. Overall, this paper presents an innovative approach that allows for more nuanced analysis of human personality traits using advanced language models like ChatGPT. By providing a comprehensive evaluation framework that measures consistency, robustness, and fairness in assessing different groups' personalities through MBTI tests, this study opens up new avenues for exploring how AI can better understand human psychology.

• - Large Language Models (LLMs) like ChatGPT have potential to exhibit human-like psychology
• - Existing works have studied virtual personalities of LLMs but not analyzed human personalities via LLMs
• - Authors present a generic evaluation framework for LLMs to assess human personalities based on Myers Briggs Type Indicator (MBTI) tests
• - Framework involves devising unbiased prompts by randomly permuting options in MBTI questions and adopting the average testing result to encourage more impartial answer generation
• - The authors propose replacing the subject in question statements to enable flexible queries and assessments on different subjects from LLMs
• - They re-formulate the question instructions in a manner of correctness evaluation to facilitate LLMs to generate clearer responses.
• - This framework enables LLMs to flexibly assess personalities of different groups of people.
• - Three evaluation metrics proposed: consistency, robustness, and fairness of assessment results from state-of-the-art LLMs including ChatGPT and InstructGPT.
• - Experiments reveal ChatGPT's ability to assess human personalities with consistent and fairer assessments despite lower robustness against prompt biases compared with InstructGPT.
• - This study opens up new avenues for exploring how AI can better understand human psychology.

Large Language Models (LLMs) like ChatGPT are computer programs that can talk and write like humans. They can understand and respond to questions just like people do. The authors of a study created a way to test if LLMs can understand human personalities based on a personality test called Myers Briggs Type Indicator (MBTI). They made sure the questions were fair by randomly changing the order of the answers. The authors also made it possible for LLMs to answer questions about different types of people. They tested two different LLMs, ChatGPT and InstructGPT, and found that ChatGPT was better at being consistent and fair but not as good at handling tricky questions compared to InstructGPT. This study helps us learn more about how computers can understand people better.

Exploring the Potential of Large Language Models to Assess Human Personality Traits

In recent years, large language models (LLMs) such as ChatGPT have become increasingly popular for their ability to generate human-like conversations. However, the potential of these LLMs to exhibit human-like psychology is still largely unexplored. While existing works have studied the virtual personalities of LLMs, they rarely explore the possibility of analyzing human personalities via LLMs.

In a new paper published in AAAI 2021, authors Haocong Rao, Cyril Leung and Chunyan Miao present a generic evaluation framework for LLMs that allows them to assess human personalities based on Myers Briggs Type Indicator (MBTI) tests. The proposed framework involves devising unbiased prompts by randomly permuting options in MBTI questions and adopting the average testing result to encourage more impartial answer generation. The authors also propose replacing the subject in question statements to enable flexible queries and assessments on different subjects from LLMs. Finally, they re-formulate the question instructions in a manner of correctness evaluation to facilitate LLMs to generate clearer responses.

This framework enables LLMs to flexibly assess personalities of different groups of people. To measure this capability further, the authors propose three evaluation metrics: consistency which measures how well an assessment result matches with its ground truth; robustness which evaluates how well an assessment result resists prompt biases; and fairness which measures whether an assessment result is biased towards certain groups or not.

The authors conducted experiments using state-of-the-art LLM models including ChatGPT and InstructGPT on two datasets: one containing MBTI test results from real people and another containing simulated results generated by humans following MBTI guidelines. Their experiments revealed ChatGPT's ability to assess human personalities with consistent and fairer assessments despite lower robustness against prompt biases compared with InstructGPT.

Conclusion

Overall, this paper presents an innovative approach that allows for more nuanced analysis of human personality traits using advanced language models like ChatGPT.