Throughout the pandemic, we encountered much speculation about the impact that remote learning would have on student performance. The expected learning loss was a concern not just of American parents and educators, but of citizens all around the world. Research is now being conducted in other countries and some states, but to date we have no comprehensive evaluation of how virtual learning impacted students across America.
To fill this gap, Rebecca Jack, Clare Halloran, James Okun, and Emily Oster investigated how children’s schooling modes during the pandemic affected their test scores. The research team used state assessment data in math and English language arts for grades three through eight across eleven states to compare pass rates prior to and during the pandemic. They started by comparing these pass rates, and they then looked deeper into how the effects of the pandemic on pass rates vary by state, school mode, and demographics to get a better understanding of the various ways students were impacted by virtual learning.
The researchers used data from the COVID-19 School Data Hub (which was created by their team using state level data) to identify district-level schooling mode data from 2020–21. Schooling modes were sorted into three categories: in-person, virtual, and hybrid. Schools were grouped as in-person if most or all students had access to in-person instruction, and schools were grouped as virtual if most or all students received instruction virtually. Any combination of instruction modes qualified as hybrid instruction.
The researchers used district level state standardized test scores from spring 2016–19 and 2021, and first used these data to identify states that could be included in their analysis. States were selected that had at least two years of pre-pandemic test scores available, as well as scores from spring 2021. The researchers included states only if no significant changes were made to the state’s standardized test content. This process resulted in a sample of eleven states.
The research team also used district demographic data from NCES and county data for controls. This told them the share of enrolled students by race and ethnicity, English-learner status, and information on the share of students enrolled in subsidized lunches.
With this data, the researchers used a regression analysis to compare district schooling mode and the district’s average pass rate for third through eighth graders on state standardized ELA and math exams, controlling for regional differences, changes in school enrollment, and test participation. By doing so, the team did their best to account for any changes that would be due to changes outside of schooling mode that occurred in schools during the pandemic.
Not surprisingly, the study found declines in the pass rates in spring 2021 across all districts and groups of students. The average pass rate, between the years 2019 and 2021, declined by 12.8 percentage points in math and by 6.8 points in ELA. Yet virtual learning seems to have impacted math scores more than ELA; this might be due to the differences in skills that are taught and applied in these different subjects, and how teaching hard skills in subjects like math might have been more difficult to translate into virtual instruction.
The researchers also found greater pass rate declines in districts that offered less in-person schooling. And in schools with higher populations of historically underserved students, there was a smaller chance the schools would offer access to in-person learning. Larger declines were also found in districts with a larger proportion of Black students, which may be tied to the finding that underserved communities had less access to in-person learning.
The magnitudes of these effects are significant. Moving from completely virtual to full access to in-person learning would have reduced the declines in pass rates by 13 or 14 percentage points in math and by about 8 percentage points in ELA. This again shows a significant difference in the effects on learning in the two subjects. Additionally, the researchers found that moving from fully virtual to hybrid instruction would have reduced the declines by 7 percentage points in math and 5 or 6 points in ELA.
The study has some caveats. It is limited in its selection of states, for example, especially since the selection was based on the availability of test scores. In states where there were no documented test scores for the 2020–21 school year, there may be different or more severe effects.
But overall, the study provides important, albeit predictable, implications about how there are specific populations that may need more targeted attention during the Covid recovery process. Underserved communities were disproportionately affected by pandemic instruction, and thus have greater ground to makeup. Recovery funds might, then, be focused on areas that struggled the most to get their students back in person.
SOURCE: Rebecca Jack, Clare Halloran, James Okun, and Emily Oster, “Pandemic Schooling Mode and Student Test Scores: Evidence from U.S. School Districts” (April 2022).
