There was a study that claimed that OpenAI’s ChatGPT can have a positive impact on student learning withdrew about a year after publication. The journal’s publisher, Springer Nature, cited “inconsistencies” in the analysis and lack of confidence in the results — but not before the paper garnered hundreds of citations and made the rounds on social media.
“The authors of the paper made very compelling claims about the benefits of ChatGPT on learning outcomes,” he said. Ben WilliamsonSenior Lecturer at the Center for Digital Education Research and the Edinburgh Future Institute at the University of Edinburgh, Scotland, Ars. “It was hailed by many on social media as one of the first hard, gold-standard proofs that ChatGPT, and more broadly generative AI, benefits learners.”
The retracted paper analyzed the results of 51 previous research studies and tried to measure “the effects of ChatGPT on students’ learning performance, learning perception, and higher-order thinking.” His meta-analysis calculated the effect size between experimental groups of different studies that used ChatGPT in education and control groups that did not use an AI chatbot.
According to the researchers who authored the paper, this analysis allegedly showed that “ChatGPT had a large positive effect on improving learning performance” and “a moderate positive effect on increasing learning perception” and “encouraging higher-order thinking.” Now the retracted results have emerged for the first time magazine Humanities & Social Sciences Communications, published May 6, 2025 by Springer Nature.
“In some cases, the results appear to be confounded by syntheses of very poor quality studies or studies that cannot be accurately compared because of very different methods, populations, and samples,” Williamson told Ars. “It really seemed like a paper that shouldn’t have been published in the first place.”
Williamson also questioned the timing of the paper’s publication, just two and a half years after OpenAI was released ChatGPT In November 2022. “It’s impossible to conduct, review and publish dozens of high-quality studies on ChatGPT and learning performance in that time,” Williamson said.
A legacy that will outlast the retreat
Since its publication, the study has been cited 262 times in other articles published by peer-reviewed journals of Springer Nature, and a total of 504 quotes from both peer-reviewed and non-peer-reviewed sources. It also attracted nearly half a million readers and received quite a lot focus online ranking in the 99th percentile for journal articles in terms of attention score.
