School closure is among the most heavy-handed interventions for turning around chronically underperforming schools. Although evidence suggests that the outcomes are positive for students, the disruption to communities and potential loss of local control can cast a shadow over the benefits. A recent study published in the Economics of Education Review by a trio of analysts from Tulane and Butler Universities revisits the effects of closing and/or restarting low-performing district schools as charter schools. They do this in two Louisiana cities, New Orleans and Baton Rouge.
The Pelican State makes a fine laboratory for testing the impacts of restarts and closures. After Hurricane Katrina devastated New Orleans in 2005, the state took over control of almost all its schools—eliminating union contracts, letting go of all teachers (for potential rehire), and rebuilding the entire education sector from the ruins. Most schools in the city became charters, and remain so today (although now they are run by New Orleans’ elected school board), with no defined attendance zones and parents responsible for choosing schools.
The analysts focus on any school that was restarted as a charter school or closed, regardless of the operator or authorizer. That means that the intervention under study could be a permanent closure, a district-to-charter restart, or a charter-to-charter restart. Although the study doesn’t define what exactly changes in schools that are restarted, it likely includes temporary closure, full re-staffing, and enrollment of new students or re-enrollment of existing students under new management and leadership.
Analysts focus on the period 2009–2012 for elementary schools and 2009–2014 for high schools (when they were still directly accountable to the state). In New Orleans, that timeframe coincided with the identification of eleven elementary schools (seven of which were district-to-charter restarts and four were charter-to-charter) and four high schools (all district-to-charter restarts). Additionally, three elementary schools and six high schools that closed without restarting were included in the analysis for comparison purposes. Baton Rouge had just five high school restarts and closures during the periods under study (two charter-to-charter restarts and three outright closures).
Analysts use a matched difference-in-differences design (starting with 2006 data to provide baseline outcomes, match students, and test parallel trends) and a pooled regression model that increases the number of data points and utilizes a wide range of covariates. In another analysis, they also match treated students to individual students within the comparison schools, calculating effects for students who attended the treated schools at the time of the restart. They examine math and ELA test scores, graduation rates, and college attendance at two- and four-year institutions—although all outcomes aren’t examined in both cities (for instance, they only have ELA/English scores for high schools, since post-treatment math scores were unavailable due to changing the math test in Louisiana).
For students in New Orleans, the effects of closures and restarts are generally positive or null. Two years following restart, test score trends in elementary school English language arts and math for treatment students are higher than for non-treatment students (specifically, math scores are 0.21 to 0.39 standard deviations higher). There were no effects on high school students’ test scores, which were derived from tenth grade ELA tests for students who started at a treatment school in ninth grade. There were also no clear effects on high school graduation or college entry in New Orleans.
It was a different story in Baton Rouge high schools (no elementary schools were included), where restarts and closures reduced both ELA test scores (by 0.07 to 0.21 standard deviations) and high school graduation rates (by 11–15 percentage points).
The analysts seek to determine possible reasons for their results and find that the variation in test score effects is positively related to increases in school-level value added. That is, changes in elementary school quality appear to be driving the results, such that the restarted schools in New Orleans were better at boosting student achievement than the closed schools that they replaced. That’s clearly what we like to see happen.
Why the different findings in New Orleans versus Baton Rouge? The authors believe the answer lies in the former’s unique post-Katrina governance structure, in which decisions were free from union constraints, giving leaders more flexibility to make changes. Baton Rouge, meanwhile, was hampered by its more traditional governance and stronger union influence. I’m not one for oversimplifying, but it’s hard to argue with that hypothesis.
SOURCE: Whitney Bross, Douglas N. Harris, and Lihan Liu, “The effects of performance-based school closure and restart on student performance,” Economics of Education Review (June 2023).
