For many students and teachers, the pivot from in-person to remote learning in March 2020 was a sudden lurch from the known to the unknown. Writ large, research shows the academic impact of that move was devastating. But details matter—and so do exceptions. Little attention has been paid to students who were already familiar with working in remote modes prior to Covid-mitigation school closures. Is it possible that they fared better than their peers, experiencing what amounted to a simple extension of their status quo? New research out of Germany tries to answer that question.
The researchers followed the longitudinal performance of 2,700 students who studied mathematics using an intelligent tutoring system (ITS) called Bettermarks. It is a robust, adaptive online system that can be used in class or at home and covers typical math content taught in grades four through twelve, from basic addition and subtraction to probability and statistics. The “teaching” portion of Bettermarks is mostly overview rather than direct instruction and is primarily used to give students practice problem sets to complete. That said, the system is also interactive—programmed to give feedback when common errors are made (“don’t add the numerators and the denominators,” “find the lowest common denominator,” etc.), provide a finite number of brief hints when requested, and supply students with more practice problems of varying difficulty in areas where they display specific weaknesses. This ITS can be programmed to provide unlimited problem sets upon student demand or to limit available sets to those assigned by a teacher. For this study, students were only working on assigned sets and had to have completed a minimum of five sets in each of the time periods under review.
The analysis covered ITS usage between January 2017 and May 2021 and, significantly, only included students who used the system before, during, and after pandemic-related school closures in Germany. The goal was to get the fullest picture of usage and outcome changes from open schools to closed schools and back again. The analysts particularly focused on potential differential effects for lower- versus higher-performing students, as previous research showed inconclusive and potentially conflicting results for those students. Based on pre-closure performance on ITS problem sets between January 1, 2017, and March 15, 2020, analysts assigned students to either a low- or high-performing group for comparison.
During the first round of school closures in Germany (March 16 through May 31, 2020), all students using the ITS showed an increase in relative accuracy rates compared to their pre-closure performance. Higher-performing students exhibited the highest absolute accuracy in terms of raw scores, but initially-lower-performing students showed larger growth in accuracy in the period relative to the full sample, with their absolute accuracy rising quicker nearer the end of the period.
German schools reopened for in-person learning in the fall of 2020 but closed again on January 1, 2021, for two full months. Looking at ITS performance during this initial return-to-business-as-usual period, the researchers saw a similar pattern to the first closure period, with an overall increase in relative accuracy rates driven by an acceleration of absolute accuracy by initially-lower-performing students. Results during the second closure period (through February 28, 2021) showed a similar pattern. Finally, in-person learning returned for good as of March 1, 2021, and the researchers looked at ITS performance from this point through the end of the school year on May 31, 2021. Perhaps unsurprisingly at this point: The results were the same again.
What to make of this steady pattern of improvement? First and foremost is the evidence that these students were not negatively impacted by the switch from in-person to remote schooling, even through the repeated open-and-close cycles they experienced. In fact, the trajectory for all students in this study was entirely upward, with low-achievers gaining the most. Whatever was going on within their daily class instruction, these findings indicate that students remained engaged and productive. This is not ironclad proof that students became more adept at learning from online resources due to increased use during school closures, although that could certainly be part of the explanation for the results observed. Additional suggestions from the report’s authors include a decrease of “math anxiety” (especially in lower-performing students) due to the lack of face-to-face competition or judgment from peers, and the possibility that use of—and success in—the ITS was being “incentivized” by teachers who made scores part of students’ class grade.
In the end, these findings are part of the complex puzzle of what happened to students as a result of Covid-mitigation school closures. It’s important to know these things, even if we’re just trying to put the pieces together years later—especially the more positive parts of the picture, as small and localized as they may have been.
SOURCE: Markus Wolfgang Hermann Spitzer and Korbinian Moeller, “Performance increases in mathematics within an intelligent tutoring system during COVID-19 related school closures: a large-scale longitudinal evaluation,” Computers and Education Open Journal (February 2024).
