# Wisdom of the Silicon Crowd: LLM Ensemble Prediction Capabilities Match Human Crowd Accuracy

**Authors:** Philipp Schoenegger, Indre Tuminauskaite, Peter S. Park, Philip E. Tetlock  
**arXiv:** [2402.19379](https://arxiv.org/abs/2402.19379)  
**Date:** 2024-02-29  
**Summary page:** https://www.summarizepaper.com/en/arxiv-id/2402.19379v1/

## Key points

- Study titled "Wisdom of the Silicon Crowd" compares forecasting accuracy in human and machine predictions
- Research builds on 'wisdom of the crowd' effect for more accurate forecasts from diverse groups
- LLM ensemble approach with twelve models used to predict outcomes for 31 binary questions
- Aggregated LLM predictions performed similarly to human crowd forecasts, outperforming a no-information benchmark
- Acquiescence effect observed with mean model predictions skewed above 50%
- Experiment showed that exposing specific LLM models (GPT-4 and Claude 2) to median human predictions improved accuracy by 17% to 28%
- Method did not surpass averaging human and machine forecasts in accuracy
- Findings suggest LLMs can achieve comparable forecasting accuracy to human crowds through effective aggregation methods, enhancing decision-making processes.

## Summary

In the study titled "Wisdom of the Silicon Crowd: LLM Ensemble Prediction Capabilities Match Human Crowd Accuracy," conducted by Philipp Schoenegger, Indre Tuminauskaite, Peter S. Park, and Philip E. Tetlock, the researchers delve into the realm of forecasting accuracy in human and machine predictions. The research builds upon the concept of the 'wisdom of the crowd' effect where aggregating predictions from a diverse group of individuals leads to more accurate forecasts about future events. The study focuses on large language models (LLMs) and their forecasting abilities compared to human crowds. Previous findings indicated that individual frontier LLMs fall short when pitted against aggregated human forecasts from crowd tournaments. To address this gap, the researchers employed an LLM ensemble approach comprising twelve LLMs to predict outcomes for 31 binary questions. These predictions were then compared to those made by a crowd of 925 human forecasters over a three-month forecasting tournament period. The main analysis revealed that the aggregated LLM predictions outperformed a simple no-information benchmark and were statistically equivalent to those generated by the human crowd. Interestingly, there was an observed acquiescence effect in which mean model predictions skewed above 50%, despite an almost even distribution of positive and negative resolutions in the questions. In a subsequent experiment (Study 2), the researchers explored whether exposing two specific LLM models (GPT-4 and Claude 2) to median human predictions could enhance their forecasting accuracy. The results showed that both models benefited from incorporating human cognitive output, leading to improvements ranging between 17% and 28% in accuracy. However, it was noted that this method did not surpass simply averaging human and machine forecasts in terms of accuracy. Overall, these findings suggest that LLMs have the potential to achieve forecasting accuracy comparable to that of human crowd forecasting tournaments through effective aggregation methods. This replication of the 'wisdom of the crowd' effect for LLMs opens up possibilities for their utilization across various applications within society, highlighting their promising role in enhancing predictive capabilities in decision-making processes.