Five months after the launch of ChatGPT, Sal Kahn, the founder of the online learning giant Kahn Academy, predicted that, “We’re at the cusp of using AI for probably the biggest positive transformation that education has ever seen.” As of now, this appears to have been an aspirational proclamation made with little evidence.
In the history of education, such declarations are commonplace, and it comes as little surprise that a tech CEO sees promise in an innovative technology. For centuries, optimists—and those with vested financial interests in technological innovations—have touted the potential world-changing benefits of blackboards, erasable pencils, projectors, radio, television, cell phones, social media, computers, and plenty more. Without a doubt, technological advancements from air conditioning to the internet have transformed schools, which have in turn brought changes both positive (classrooms free from sweltering heat) and negative (screen addictions).
As an increasing number of corporations are investing in the advancement of AI, the evidence of this fluid technology on student learning has proven difficult to pin down. Will it revolutionize education as every child gets a personalized AI tutor, consequently boosting academic achievement? Or will it become a glorified machine widely used for cheating and one that minimizes thinking and thereby leads to cognitive atrophy?
A new behemoth report from the Organization for Economic Co-operation and Development (OECD) analyzing the emergence of AI in education thus far seems to suggest that it might be a bit of both. According to the OECD analysis, generative AI (i.e., the machine-learning models in widespread use, like ChatGPT, Gemini, and Claude) can support knowledge acquisition for students by facilitating learning and improving creativity. Key word: “can.” Emerging data highlighted in the report also point to several snags.
While available research suggests that access to generative AI can improve student performance—on practice problems, final exams, and essays, for example—this does not prove sustained improvements in student learning. For example, in one randomized experimental study comparing student writing with and without AI support, the ChatGPT-supported group showed greater improvements in scores on their writing, yet the return did not yield greater knowledge acquisition or transfer. Additionally, the AI-users exhibited a distinct reliance on the technology and, compared to the non-AI supported groups, “were less likely to engage in metacognitive activities,” such as self-reflection.
In other words, improved performance on AI-assisted assignments does not necessarily translate to improved understanding. Using AI for writing may have created a better final product, but it didn’t make students any better writers than if they’d completed their work without AI assistance and may have only taught them how to rely on the technology.
In another RCT, access to generative AI improved math performance during practice by 48 percent, but once those supports were removed and students took exams in a closed-book environment, students performed 17 percent worse than students who studied with just a textbook.
Outsourcing thinking to a chatbot may produce better scores on essays or exams, but only so long as students have access to it. Remove the tool and students can no longer perform at that level without the assist.
A similar dynamic plays out with simpler technologies. Little Johnny may perform better on a test with access to a calculator, but if those supports are removed and he doesn’t know his times tables, then he has not actually learned the material. In the physical domain, an athlete could lift more if assisted by robotics, but that’s no measure of their physical prowess. Worse, their muscular strength may even decrease over time, making it much more difficult to bench 200 when the robot isn’t around.
When it comes to affective capacities such as creativity, the OECD report cites a study measuring how the use of a general-purpose LLM can enhance creativity and quality of writing. The group using generative AI outperformed those going it alone—with only their brains to rely on—in terms of both creativity and writing quality. (Importantly though, the ideas of AI users were not as “funny.”)
Does this really mean AI is enhancing our creativity? Or are we just outsourcing it and receiving high marks? There’s an important distinction.
The report details an interview with Ronald Beghetto, an Arizona State University professor who specializes in using AI to support creative processes. Beghetto explains that, to use generative AI effectively, one needs “a clear goal, prior content knowledge, and a sense of what you want to build.” If AI is ineffective in transferring domain-specific knowledge, and students using it lack the content knowledge and understanding to employ it well, that renders the tool essentially useless. An intuitive first step seems to be to teach students a curriculum rich in foundational knowledge and essential skills.
The report acknowledges that “beneficial effects of a support tool depend on whether and to what extent the tool can elicit conducive interaction patterns.” Namely, researchers and educators should concern themselves with how learners use AI—the measurable effects on learning are simply not clear-cut.
The topline conclusion of the OECD authors is, “GenAI can support learning when guided by clear teaching principles,” but the details of specific studies give cause for skepticism. In short, the current research is helpful, but frankly, the effects on student learning remain a grab-bag, with results contingent on a multitude of conditions.
Despite the mixed results, there has been a rapid uptick in teacher and student usage of AI. Many districts and schools have embraced the technology. Yet, the return on this investment when it comes to genuine learning and understanding is not uniform or conclusive. Moreover, there are real tradeoffs to consider. Will discussions with AI distort children’s social skills? Will AI cultivate an overreliance on technology, extending beyond the classroom, and foster addiction-like behavioral symptoms as social media algorithms have?
Assuming students use it well (asking for feedback instead of writing a draft) still the question remains: Why not ask another knowledgeable human for feedback? Dare we suggest, a teacher? A peer? A parent? That approach has worked for a long time. At the very least, the report’s indefinite conclusions related to student learning warrant a tap on the brakes of the AI hype-train.
