Large language models can predict social science experiments but can't replace human insight

Large language models can predict the results of social science experiments

Luo, X. et al. Large language models surpass human experts in predicting neuroscience results. Nat. Hum. Behav. 9, 305-315 (2025). Crockett, M. & Messeri, L. Should large language models replace human participants? Preprint at PsyArXiv https://doi.org/10.31234/osf.io/4zdx9 (2024). Atari, M., Xue, M. J., Park, P. S., Blasi, D. & Henrich, J. Which humans? Preprint at PsyArXiv https://doi.org/10.31234/osf.io/5b26t (2023). Filippas, A., Horton, J. J. & Manning, B. S. Large language models as simulated economic agents: what can we learn from homo silicus? In Proc. 25th ACM Conference on Economics and Computation 614-615 (ACM, 2024). Meta Fundamental AI Research Diplomacy Team (FAIR) et al. Human-level play in the game of diplomacy by combining language models with strategic reasoning. Science 378, 1067-1074 (2022). Park, J. S. et al. Generative agents: interactive simulacra of human behavior. In Proc. 36th Annual ACM Symposium on User Interface Software and Technology (eds Follmer, S. et al.) 1-22 (ACM, 2023). Time-sharing Experiments for the Social Sciences. TESS www.tessexperiments.org (2026). Chang, S., Kennedy, A., Leonard, A. & List, J. A. 12 Best Practices for Leveraging Generative AI in Experimental Research (National Bureau of Economic Research, 2024). Harding, J., D'Alessandro, W., Laskowski, N. G. & Long, R. AI language models cannot replace human research participants. AI Soc. 39, 2603-2605 (2024). Binz, M. et al. A foundation model to predict and capture human cognition. Nature 644, 1002-1009 (2025). Petrov, N. B., Serapio-García, G. & Rentfrow, J. Limited ability of LLMs to simulate human psychological behaviours: a psychometric analysis. Preprint at https://arxiv.org/abs/2405.07248 (2024). Chen, Y., Hu, Y. & Lu, Y. Predicting field experiments with large language models. Preprint at https://arxiv.org/abs/2504.01167 (2025). Wang, X. et al. Self-consistency improves chain of thought reasoning in language models. In Proc. 11th International Conference on Learning Representations (poster) (ICLR, 2023). Arora, S. et al. Ask me anything: a simple strategy for prompting language models. In Proc. 11th International Conference on Learning Representations (ICLR, 2023). Voronov, A., Wolf, L. & Ryabinin, M. Mind your format: towards consistent evaluation of in-context learning improvements. In Findings of the Association for Computational Linguistics: ACL 2024 (eds Ku, L.-W. et al.) 6287-6310 (ACL, 2024). Hou, B., O'Connor, J., Andreas, J., Chang, S. & Zhang, Y. Promptboosting: black-box text classification with ten forward passes. In Proc. International Conference on Machine Learning (eds Krause, A. et al.) 13309-13324 (PMLR, 2023). DellaVigna, S. & Vivalt, E. Forecasting social science: evidence from 100 projects. National Bureau of Economic Research https://doi.org/10.3386/w34493 (2025). Voelkel, J. G. et al. A registered report megastudy on the persuasiveness of the most-cited climate messages. Nat. Clim. Chang. 16, 214-225 (2026). Mason, K. et al. A megastudy of behavioral interventions to increase voter registration ahead of the 2024 US presidential election. Preprint at PsyArXiv https://doi.org/10.31234/osf.io/p6b2e_v3 (2025). Druckman, J. N. Experimental Thinking (Cambridge Univ. Press, 2022). Wang, A., Morgenstern, J. & Dickerson, J. P. Large language models that replace human participants can harmfully misportray and flatten identity groups. Nat. Mach. Intell. 7, 400-411 (2025). Park, P. S., Schoenegger, P. & Zhu, C. Diminished diversity-of-thought in a standard large language model. Behav. Res. Methods 56, 5754-5770 (2024). Response retrieved from ChatGPT-4 on 22 May. ChatGPT https://chatgpt.com/share/2a2b66d0-e9fd-421f-bedd-bb6f9354742a (2024). Zhu, J.-Q., Xie, H., Arumugam, D., Wilson, R. C. & Griffiths, T. L. Using reinforcement learning to train large language models to explain human decisions. Preprint at https://arxiv.org/abs/2505.11614 (2025). Tyner, A. H. et al. Investigating the replicability of the social and behavioural sciences. Nature 652, 143-150 (2026). Cremer, D. D. & Kasparov, G. AI should augment human intelligence, not replace it. Harvard Business Review https://hbr.org/2021/03/ai-should-augment-human-intelligence-not-replace-it (2021). Cummins, J. The threat of analytic flexibility in using large language models to simulate human data: a call to attention. Preprint at https://arxiv.org/abs/2509.13397 (2025). Achiam, J. et al. GPT-4 technical report. Preprint at https://arxiv.org/abs/2303.08774 (2023). Viechtbauer, W. metafor: Meta-Analysis Package for R v.4.8-0 (2015). Rios, K., Roth, Z. C. & Coleman, T. J. III. The importance of scientists' intellectual humility for communicating effectively across ideological and identity-based divides. Proc. Natl Acad. Sci. USA 122, e2400930121 (2025).

AI can predict how you'll respond to a survey. But that's not the same as understanding you

What makes people change their minds, or their behaviour? Social scientists spend a lot of time thinking about this question, and experiments are one of the most powerful ways to answer it. Experiments - testing ideas on real people - take considerable amounts of time and money. Enter large language models (LLMs): artificial intelligence (AI) systems trained to mimic certain kinds of text-based human behaviour based on vast amounts of human-produced text. A new study led by Harvard psychology researcher Ashwini Ashokkumar, published today in Nature, suggests LLMs such as GPT-4 can predict the outcomes of many social science experiments surprisingly well. But the results come with a warning: a system that predicts human responses is not necessarily a system that understands human behaviour, and "synthetic respondents" or "silicon samples" are not a direct substitute for real people. A striking result Ashokkumar and her colleagues assembled 70 real experiments already conducted in the United States involving almost 120,000 participants. They then gave GPT-4 descriptions of hypothetical respondents alongside experimental messages and survey questions, and asked it to estimate how those people would respond under different conditions. Next they compared GPT-4's predictions with the real results, and found a strong correlation. The model could often distinguish between interventions that were more or less effective. This is a striking result. It suggests LLMs may capture meaningful patterns in the social world - at least in the kinds of text-based US survey experiments examined in the study. But it's not evidence that AI has discovered a reliable shortcut around human research. Useful forecasts are not the same as understanding US scholars Lisa Messeri and Molly J. Crockett have warned that AI systems can create "illusions of understanding": outputs that look insightful and useful, while encouraging users to overestimate what has actually been understood. An LLM can generate a plausible explanation or a persuasive forecast. But this may reflect sophisticated pattern-matching rather than genuine insight into the mechanisms behind the observed behaviour. For example, the new study found GPT-4 was often good at ranking the likely effects of different treatments. But it systematically estimated the effects were around twice as big as the real results. That distinction is crucial. A tool may tell researchers that message X will probably work better than message Y, yet still be unreliable about whether the difference will be tiny, modest or transformative. A powerful new tool for pilot studies Used carefully, this information could still be highly valuable. Researchers often run small pilot studies before launching expensive experiments. These pilots help refine interventions and estimate whether a proposed effect is large enough to justify a larger study. LLM-generated forecasts may supplement these pilots. For example, researchers might simulate how different demographic profiles respond to several versions of a vaccination message, workplace intervention or policy framing. However, the new study found that combining LLM predictions with human forecasts was more accurate than either source alone. The most useful future may not be AI replacing human researchers or research subjects, but AI helping researchers decide where to direct scarce human resources. The temptation of 'silicon sampling' The idea of "synthetic respondents" or "silicon sampling" is not just of scientific interest. It is increasingly being discussed in polling, market research and public consultation, where supporters see opportunities for faster and cheaper testing. However, critics warn it could undermine trust if simulations are presented as real public opinion. For a politician wondering what the public would think of a new tax policy, or a company curious about a new advertising campaign, LLMs can supply a quick and plausible answer. But it is not quite the same as measuring public opinion. A conventional survey collects responses from people living in a particular society at a particular moment. A synthetic sample draws instead on patterns encoded in the model's training data, prompt design and guardrails. It may reproduce elements of human judgement, but it lacks lived experience, local knowledge and a real stake in the issue being studied. This gap may be especially important for emerging issues, marginalised communities, fast-moving events, and populations that are poorly represented in online data. Ashokkumar and colleagues found the model performed broadly well across demographic groups, but also identified some differences in accuracy favouring white and Republican samples in the US context. Without careful calibration, synthetic respondents may reproduce dominant patterns in available data while smoothing over disagreement or minority perspectives. Similar applications - and risks - are emerging in AI-assisted expert panels, forecasting and Delphi-style consensus tools. They may make expert forecasting and deliberation processes faster and more accessible, but model-generated agreement can also mask genuine and useful disagreement, rather than resolve it. Silicon sampling may be useful for generating hypotheses and stress-testing assumptions, but it is no substitute for listening to real populations. The risk is not using synthetic agents. The risk is mistaking a model-generated proxy for the population itself. The risk of optimising harmful persuasion The same predictive capability can also be misused. The authors tested whether GPT-4 could identify social media content likely to reduce COVID vaccination intentions. Although the model may refuse to generate anti-vaccination messages directly, it could still help identify which harmful messages from an existing set of options were likely to be most effective. This highlights the need for safeguards that go beyond blocking obviously harmful prompts. Systems may also need protections against using AI to rank, optimise or target harmful persuasion. Research using proprietary LLMs such as GPT-4 is also vulnerable to changes made by model providers. The models can change or be retired without notice, which makes it hard or impossible for other researchers to verify or repeat AI-based findings. The central lesson is not that AI prediction is useless, nor that it is magic. LLMs may become valuable instruments for social science, helping researchers pilot ideas, prioritise interventions and explore scenarios at low cost. At the same time, there's a risk they may get enough things right to make us think we understand more than we do.