States embraced school turnaround efforts in the wake of the No Child Left Behind (NCLB) Act in the early 2000s. These took various forms at first, as each state pursued their own turnaround strategies per NCLB’s requirements. School Improvement Grants were added to the mix in the Obama years—that is, funding to assist improvement efforts specifically in the lowest-performing 5 percent of schools—kicking that transformative work into overdrive. Most pundits have been bearish on the outcomes writ large, especially in terms of scale and cost. But what have researchers found? A recent meta-analysis of impact evaluation studies published by the Annenberg Institute provides a detailed look from across the country.
Researchers from the University of Virginia, Harvard University, and the Education Trust initially identified an astonishing 11,120 published studies related to school turnaround efforts between 2000 and 2018. Refinement for content and rigor brought the total to 207. The final criteria focused on specific types of interventions studied (such as charter conversion, state takeover, technical assistance, “transformation,” or “turnaround”), academic outcomes examined (seven outcomes were identified, including test scores, suspension rates, attendance, graduation, and grade progression), and research methodology utilized (difference in differences, regression discontinuity, randomized control, instrumental variables, and matching). Sixty-seven studies met all the criteria, which between them included 141 total academic outcome estimates upon which the meta-analysis was conducted. Both the intervention types and the academic outcomes were coded and then compared.
Researchers found that the average intervention effort demonstrated modest but positive results on math test scores, and positive but not significant effects on English language arts (ELA) scores. Results suggested positive effects on school attendance, reduced disciplinary infractions, and increased graduation rates. Importantly, there was no evidence of negative effects on low-stakes testing or tests in other subjects like science or the humanities, which suggests observed improvements were a result of real learning gains rather than data manipulation or other systemic gaming. It is noteworthy that math test score improvement was most pronounced among Hispanic students.
Researchers found that there weren’t really any outstanding differences in effects by intervention type. Still, charter conversion was observed to be more successful at improving math outcomes than other strategies, and districtwide interventions yielded larger results than those targeted at individual school buildings. The two most effective features were extended learning time and teacher replacement, both of which produced statistically significant improvement in both math and ELA outcomes. Tutoring and wraparound services were shown to have positive albeit modest effects. Positive results began to appear one year after the start of intervention, which stands to reason given the traditional school year calendar. But the longer the duration of the intervention, the larger the effects observed.
Taken as a whole, school improvement efforts had a small but statistically significant average effect of 0.062 standard deviations on math achievement, and a smaller but not significant effect on ELA. While this is not necessarily a reason to jump for joy, small positive effects in a wide range of outcomes and no observed negative effects are a far cry from the conclusion that “it’s all bad.” Unfortunately, there were no data provided on the cost of interventions, nor an analysis of the return on investment. It seems likely that the costs outweighed the benefits observed, even at this level of detail. However, it cannot be denied that improvement is improvement, especially for schools which have been underperforming for years.
Other research, including our own, has shown the benefits of charter schools in terms of both teaching and learning, thus the fact that charter conversion showed strong positive results on math achievement makes sense. Ditto for teacher replacement. In the end it seems as though both schools of thought can be correct at once: Small pockets of positive outcomes clearly resulted from the turnaround effort, and the effort as a whole can still be thought of as not entirely successful.
SOURCE: Beth E. Schueler, et. al., “Improving Low-Performing Schools: A Meta-Analysis of Impact Evaluation Studies,” retrieved from Annenberg Institute at Brown University (August 2020).
