While some evidence indicates a return to pre-pandemic levels of academic achievement for students across the country whose learning was interrupted by school closures and remote education, not every student is there yet, and the gaps between subgroups of students are either the same or worse than they were in 2019. Additionally, evidence suggests that the negative impacts have been larger in math than in English language arts. Out-of-school time, and summer programs specifically, have emerged as promising means by which to boost achievement. A recent meta-analysis—by a team of researchers led by Kathleen Lynch from the University of Connecticut—looked to determine whether the data support summer math programming as an effective way to increase student outcomes.
The two most recent meta-analyses on the impacts of out-of-school-time programs were released in 2000 and 2006; many of the studies included at the time are now decades old and “use samples and methodologies that are now dated.” Broadly adhering to the What Works Clearinghouse recommendation of a twenty-year-review time limit, Lynch and her team gathered thirty-seven studies published between 1998 and 2020 that analyzed a summer mathematics intervention, included treatment and control groups, presented outcomes comparing those groups, and presented sufficient information to calculate effect sizes. Study types included randomized experiment and regression discontinuity designs (30 percent), as well as quasi-experimental designs that met all the previously-listed criteria (70 percent). Two of the studies had been previously included in the 2006 meta-analysis; all the others were new. Thirty-five of the studies took place in the United States, one in Canada, and one in England. The studies looked at both large national programs like Upward Bound and small local programs conducted at the school and district levels; most involved a majority of low-income, non-White students. The interventions were nearly evenly split between elementary and middle/high school students. Interestingly, 11 percent of the summer programs studied took place fully online. Most programs were rather intense, with a mean duration of 158 hours.
The outcomes were not particularly surprising. Compared to control group individuals, students who participated in summer math programs experienced significantly greater improvements in math learning, however it was measured in the individual studies. The average weighted estimate was a boost of 0.10 standard deviations overall, as well as for the subset of studies whose outcomes were measured by standardized tests. This would equate to a 4 percentile-point improvement. Longer-term outcomes—such as future math course-taking and grades—were also positive but less precisely measurable with the data available. There was no difference in outcomes for children based on income level; all students benefitted equally by participating in summer math programs.
However, there was variation in the positive treatment effects, and a few studies actually showed mild negative impacts for treatment group students. In looking at features that might play a role in these variations, Lynch and her team compared aspects such as teacher professional development and the coursework provided to the students. The most positive outcomes occurred in programs focused specifically on math, as opposed those that included reading or science as part of the curriculum. At the same time, weak or negative results were more likely in programs that used math textbooks (i.e., summer programs that look too much like school), perhaps an indicator of a program that used summer school as “punishment” for students who were faring poorly during the school year.
Lynch and her team conclude that summer programs can be valuable options for schools to help their students recover, relearn, and even move ahead in their math learning. However, a data-backed message that such programs must be properly tailored, of high quality, outcomes-focused, and available to all who might need them is both welcome and timely.
SOURCE: Kathleen Lynch, Lilly An, Zid Mancenido, “The Impact of Summer Programs on Student Mathematics Achievement: A Meta-Analysis,” Review of Educational Research (July 2022).
