In sum, our reanalysis suggests that publication bias is the main driver of Wang and Fan’s1 conclusion that ChatGPT has a positive impact on student learning. Once publication bias is corrected for, the effects of ChatGPT on learning decrease by a factor of two to three, and the evidence in favor of those effects disappears. The conclusions from our reanalysis hold across different publication bias adjustments. Importantly, our findings do not imply that there is no positive effect of ChatGPT, or other large language models on learning; in fact, our analysis reveals that there is “absence of evidence” rather than “evidence of absence”. The present literature appears to be contaminated by publication bias; high-quality registered reports15 are needed to properly evaluate the effect of large language models in educational settings.

Students increasingly use large language models such as ChatGPT to help them with their study tasks. Consequently, there is an acute interest in ascertaining and quantifying the effects of large language models in educational settings. In a recent article, Wang and Fan conducted a comprehensive meta-analysis on the impact of ChatGPT on students’ learning performance, learning perception, and higher-order thinking featuring 51 studies. Wang and Fan1 conclude that “ChatGPT has a large positive impact on improving learning performance (g = 0.867) and a moderately positive impact on enhancing learning perception (g = 0.456) and fostering higher-order thinking (g = 0.457).” Here we show that these effects greatly diminish once publication bias is accounted for, and the evidence in favor of the benefits disappears. In order to properly evaluate the benefit of large language models in educational settings, we believe that it is essential to design high-quality, pre-registered experiments.