The findings of Fordham’s latest report, "Do Authorizer Evaluations Predict the Success of New Charter Schools?" suggest at least three takeaways: authorizers should pay close attention to applicants’ education and financial plans; should incorporate multiple data sources and perspectives; and must continue to hold approved schools accountable for their results.
Like everyone else, we education reformers would love to have a crystal ball. Yet, in practice, predicting the performance of schools, like almost every other form of prediction, is inherently challenging.
Still, it’s essential that we do our best, particularly when it comes to forecasting the performance of proposed charter schools. After all, despite the growing pile of research that suggests charter schools outperform traditional public schools on average, the U.S. has too many mediocre or downright bad charters and too few truly excellent ones.
That’s why the National Association of Charter School Authorizers (NACSA) has developed various resources that outline “best practices” for its members (and other authorizers) to use when reviewing the plans of would-be schools. But as sensible as these practices are, they are largely the product of accumulated wisdom and experience. And there are certain questions, such as whether some practices make a bigger difference than others and where those reviewing charter school applications should focus their attention, that they cannot answer.
Thus, the need for empirical research on the evaluation of proposed charter schools by authorizers. Yet there has been strikingly little investigation of this vital subject, with the exception of a 2017 Fordham study conducted by Anna Nicotera and David Stuit, Three Signs That a Proposed Charter Schools is at Risk of Failing, and NACSA’s subsequent expansion of that analysis. For example, to our knowledge, there is essentially no research on one of the most fundamental questions—namely, whether the applications that authorizers rate more highly tend to produce schools that perform more strongly.
One reason for that deficit is the challenge of bulky, non-comparable data. Actual charter applications are rather cumbersome, and their format varies from one authorizer to the next. But an even bigger challenge is sample size. Nationally, only a handful of entities have authorized enough schools to make a rigorous quantitative analysis possible.
Which brings us to North Carolina, where the state’s sole authorizer, the State Board of Education, has now presided over the creation of more than one hundred schools and the closure of more than twenty-five low-performing or struggling schools since the abolition of the statewide cap in 2011, making it the exclusive overseer of one of the largest charter portfolios in the land.
To make the most of the research opportunity presented by North Carolina’s track record, we partnered with Adam Kho, an assistant professor and rising star at the University of Southern California, who is well known for his work on school turnaround and charter schools, and who, like us, was interested in examining how authorizers might increase the likelihood that new schools get a strong start.
That partnership resulted in the Fordham Institute’s latest report, Do Authorizer Evaluations Predict the Success of New Charter Schools?, which Adam conducted with coauthors Shelby Leigh Smith (USC) and Douglas Lee Lauen (the University of North Carolina). With the help of the North Carolina Department of Public Instruction, the team constructed a unique dataset that includes the ratings external reviewers gave to specific portions of proposed schools’ written applications; the votes members of the state’s Charter School Advisory Board took after reviewing those applications (and interviewing the most promising candidates); and the outcomes of students in newly approved schools.
Because the data are limited to the period after North Carolina lifted its charter cap and the pandemic struck, Adam and company were able to analyze the evaluations and votes that determined the fate of four cohorts of applications and then follow those schools that were approved for one to four years after they opened. That amounts to 179 applications, fifty-three approved applicants, and forty-three schools that actually managed to open their doors.
So, what did they find?
First, schools that more reviewers voted to approve were more likely to open their doors on time but no more likely to meet their enrollment targets. In other words, there is some evidence that reviewers were able to identify applicants that had their ducks in a row.
Second, schools that more reviewers voted to approve performed slightly better in math but not in reading. In other words, reviewers’ collective judgment also said something about how well a new school was likely to perform academically.
Third, ratings for specific application domains mostly weren’t predictive of new schools’ success, but the quality of a school’s education and financial plans did predict math performance. Importantly, these domain-specific ratings were based exclusively on evaluations of schools’ written applications (unlike reviewers’ final votes, which also reflected their interviews with applicants and whatever other information was at hand).
Finally, despite the predictivity of reviewers’ votes, simulations show that raising the bar for approval would have had little effect on the success rate of new schools. For example, reducing the share of applications that were approved from 30 percent to 15 percent wouldn’t have discernibly boosted approved schools’ reading or math performance, nor would increasing the number of “yes” votes required for approval.
(Unfortunately, we cannot assess the implications of lowering the bar for approval, since we can’t gauge the effectiveness of schools whose proposals were rejected. Had those schools been in the mix, it’s possible that both reviewers’ votes and their ratings of specific application domains would have been more predictive.)
So, what does all of that imply for authorizing in North Carolina, the seven other states with a single (statewide) authorizer, and the thirty-six states with other combinations of state and local authorizers?
Given the diversity of approaches that states have taken to authorizing—and their geographic and demographic diversity—caution is warranted. But in our considered opinion, the findings suggest at least three takeaways.
First, authorizers should pay close attention to applicants’ education and financial plans. Per Finding 3, the quality of these plans significantly predicts the resulting schools’ math performance (unlike other elements of the application, such as the perceived quality of a school’s mission statement). Our sense is that’s no coincidence, as instructional prowess and budgetary competence are “must-haves” for a successful school.
Second, authorizers should incorporate multiple data sources and perspectives. Like a strong cover letter, a well-written charter school application is a sign that an applicant deserves serious consideration. But of course, the decision to approve should also reflect those intangibles—largely gleaned from face-to-face interviews—and the age-old adage that two heads are better than one.
Finally, authorizers must continue to hold approved schools accountable for their results. After all, we know that the quality of charter schools, like the quality of individual teachers, varies drastically once they are entrusted with the education of children. So if we can’t reliably weed out low performers before they are approved, the only surefire way to ensure that charters fulfill their mission is to intervene when their performance consistently disappoints (meaning, in this case, that chronically-low-performing schools should be drastically overhauled or closed).
To be clear, the latter is not our preferred outcome. But so long as a minority of approved charters underperforms, we see no alternative.
Someday, perhaps, the guidance that empirical research provides to authorizers will make the process for approving new schools more certain and less dependent on human judgment—that crystal ball that so often fails us. Until then, we’ll just have to take it one application at a time.
This is the eighth in a series on doing educational equity right. See the introductory post, as well as essays on school finance, student discipline, advanced education, school closures, homework, grading, and a concluding post on doing educational equity wrong.
One of the ironclad beliefs among education reformers back in the day was the certainty that the achievement gap was caused, at least in part, by a teacher-quality gap. As articulated in the Education Trust’s landmark 1998 white paper Good Teaching Matters, by any measure you could come up with, the most qualified teachers were in the most-affluent schools, while the least qualified worked in the highest-poverty ones. This, more than anything else, is what we meant by “under-resourced” schools and was the result of our inequitable funding system, combined with HR systems and collective bargaining agreements that put the preferences of adults over the needs of kids—especially low-income students and students of color.
It wasn’t hard to figure out what caused this gap. The richest districts had more money and, therefore, could pay their teachers more. Even within districts, experienced teachers had first dibs on open positions. So when teachers retired, these veterans would claim open spots at the more affluent schools, where the job was perceived as easier, while the high-poverty schools were left to hire rookies.
Over the past twenty-five years, a growing body of research has complicated the picture.
Most significantly, we have learned that the attributes of teacher quality we could measure back then, like certification status or even years of experience, are weak proxies for effectiveness. There are some relationships between those markers and student outcomes, but they tend to be small and rather weak. One study estimated that 97 percent of what makes a teacher great is not measured by those sorts of inputs.
Another challenge is that the whole notion of an “effective teacher” might be shaky. It appears from some studies that teachers who are quite effective in some schools are less effective in others. The same thing goes when it comes to teaching students of different races. Lots of research has found that, all else being equal, Black teachers are more effective with Black students than their White counterparts are.
Perhaps all of this is why at least one reputable study found that the teacher effectiveness gap essentially does not exist, at least if we’re defining effectiveness as the ability to consistently boost student achievement.
Furthermore, our definition of teacher effectiveness is constrained by our ability to measure it. Most studies look at teachers’ impact on students’ growth in reading and math scores in grades three through eight. What about teachers that don’t teach reading and math? Or teach in kindergarten through second grade, or in high school? A few studies look at teachers’ impact on students’ grades, behavior, or graduation rates (after they leave their classrooms), but translating those studies into actionable data about millions of teachers is a nut we have yet to crack. So we don’t have an easy way to tell, for example, whether high-poverty schools have systematically lower quality social studies or art or music or PE teachers than more-affluent schools do. To figure that out, you would need a comprehensive and sophisticated teacher evaluation system.
None of this means we should despair. It is possible to attack the question of teacher effectiveness head-on and make sure that poor kids and kids of color get their fair share of the best instructors. Mostly, it means following the footsteps of Washington, D.C., where the IMPACT system has been designed to do exactly that. At its center is a set of sophisticated evaluations of teachers and their instruction—the type that went out of fashion after Race to the Top–era reforms mostly crashed and burned
Perhaps the best part of IMPACT is that it has been tweaked and improved over time. The district and its expert advisors identified the problem with judging teachers in very different schools against one another, an approach that originally disadvantaged teachers in the highest-poverty schools, as it made them look less effective than they really were. They fixed that problem while also maintaining high standards around instructional expectations, both in core subjects and beyond.
It’s not perfect, but IMPACT is miles ahead of anyone else’s evaluation system, and had a clear, positive effect on teacher effectiveness and diversity and student outcomes.
So for folks out there who say they are committed to educational equity and closing teacher effectiveness gaps, I would say: If you’re serious, you need to follow DCPS and put something like IMPACT in place in your district, as well. Otherwise, you are just virtue signaling.
The problem, of course, is that the politics of achieving such a bold reform are extremely difficult. The unions are dead set against it, as it’s their job to protect all their members, including mediocre and ineffective teachers. Unions also tend to be controlled by older teachers, who don’t want to be told that they need to leave their cushy jobs in the most affluent schools and go teach somewhere else. Not to mention that moving the best teachers to the poorest schools is not going to go over well with rich (and powerful) parents.
In the absence of something like IMPACT, then, we might have to settle for a half step in the right direction. Namely: Let the market do its magic. In this case, I refer to the labor market. Districts could pay teachers significantly more to teach in their toughest schools, as Houston is doing under state-appointed superintendent Mike Miles, while also making sure that the high-poverty schools have money in their budgets to pay for those more expensive teachers. Over time, as vacancies come up at such schools, we should expect to see great teachers migrate towards those campuses, at least if the extra pay is significant enough. We may not be able to measure those teachers’ effectiveness without an IMPACT-style system, at least not beyond teachers of reading and math in grades three through eight, but common sense would tell us that smart teachers are going to follow the money.
Which is not to say it’s easy to find the funding to dramatically boost teacher salaries in our neediest schools. But whether we are on the left, right, or center, we need to acknowledge that such efforts are what it takes to accomplish true educational equity. It’s time for all of us to put our money where our mouths are.
The hits just keep on coming: Earlier this month, a motley crew of former Colorado lawmakers helped spur the introduction of a grotesque piece of legislation aimed squarely at dismantling the state’s venerable charter school sector. Under the guise of accountability, the bill—all fifty-five pages of it—would, if enacted, stifle the creation of new charter schools, smother the prospects for renewal of existing schools, and sabotage the systems that have helped them flourish over the last thirty years. Venom and hostility toward charters in some corners of the Centennial State are nothing new, but the sheer volume of anti-charter provisions stuffed into one package puts this particular bill in a noxious class of its own.
Indeed, there are too many policy transgressions to cover in one sitting, but consider five of the most egregious. The first is a provision that would give districts the power to revoke a school’s charter if the district is experiencing declining enrollment. (It also enables them to impose a moratorium on the approval of new schools.) Never mind a charter school that sees an uptick in kids as a result of its popularity or programming; so long as the district is losing students, charters located within it could be put on the chopping block. Suffice to say, this would corner them into an impossible situation in the face of declining enrollments both national and locally. In fact, Colorado’s public school rolls are at a decade low and are forecast to continue declining through 2029—at which time a rebound is expected, but the number of students is not projected to return to pre-pandemic levels until at least 2035.
Second, the bill would injudiciously equip every Tom, Dick, or Harry who lives within a school district’s boundaries with the means to appeal the school board’s decision to approve a charter application. It doesn’t require much imagination to see how this provision could be abused to gum up the works for charters. Former Indiana Governor Mitch Daniels once said, “When it comes to our children’s future, the real devil is not in the details. He’s in the delay.” This clause would give anyone with a beef against charters or choice de facto veto power. If you’re a potential school operator in Colorado, why would you subject yourself to such misery?
Third, the legislation preys upon the lack of access to adequate facilities that Colorado charter schools already face—one of the biggest barriers to growth. Under current law, school districts are required to provide charter applicants, upon request, with a list of vacant or underused buildings. Applicants may then apply to use these buildings and are not required to pay rent—a reasonable stipulation, given that charter schools already spend a considerable amount of per-pupil funding on operations and maintenance (to say nothing of the fact that these facilities were previously paid for by taxpayers). The troublesome new bill would repeal all of this in one fell swoop, effectively kneecapping any aspirations for charter expansion.
Fourth, the State Board of Education—which serves as a backstop against arbitrary and capricious decisions by local school districts—would lose its ability to effectively act as a check against them. A prominent feature of Colorado’s charter law, the appeals process, allows charter applicants to ask the board to serve as a referee when they feel they aren’t being treated fairly by districts. Over time, the decisions rendered by the board have been balanced and unbiased, with half of its rulings coming down in favor of districts and half in favor of charters. Put simply, altering the appeals process is a solution in search of a problem.
Finally, if the bill becomes law, Colorado’s charter schools would be subjected to additional and extraneous bureaucratic red tape from which they are currently spared thanks to “automatic waivers.” This flexibility allows charters, for example, to develop their own school calendar, pick their own curriculum, and hire and compensate teachers independent from the district. The bill’s authors claim that eliminating these waivers will lead to greater “transparency” and “accountability,” but the truth is that the forces behind this bill are attempting to hijack these terms for their own purposes. Here it’s worth noting the draconian consequences that would result if a charter fails to comply with the newly applied regulations (along with the other ticky-tack rules saturated throughout the bill): charter revocation or nonrenewal. It’s hard to read the motivation behind this legislation as anything other than cynical retribution.
Scheduled to be heard in Colorado’s House education committee early next month, the bill’s introduction came a few days before President Biden’s similarly mean-spirited (towards charters) education budget proposal, which seeks an unprecedented cut in the federal Charter School Program. Making matters worse, the National Alliance for Public Charter Schools—the nation’s leading trade association of charter schools and charter management organizations—is currently in a transition period. These are the times that try the souls of charter school advocates.
The silver lining, to the extent there is one, is that after thirty years, Colorado’s charter sector is a well-established constituency: 261 schools strong and serving 135,000 students, including my nine-year-old daughter. This kitchen sink of a bill stands little chance of making it through the legislature, let alone past the governor’s desk, given his background as founder of not just one, but two charter schools. Still, the brazenness of the legislation serves as a reminder that the forces of resistance and repeal are relentless, and will—despite the impressive track record of charter schools—continue their misguided effort to reverse the education tide.
When the TV salesman pitches a beauty product to eliminate wrinkles or a politician promises no new taxes, most of us raise a skeptical eyebrow. If only we afforded that same skepticism to education fads. “Learning styles” have as much grounding in reality as horoscopes, yet twenty-nine states include them in their licensing exams. Seemingly every morning, a new innovation promising to revolutionize the classroom crops up. Thankfully, most of these fads are largely innocuous, if distracting.
In her compelling new book Bad Therapy, Abigail Shrier eviscerates education’s latest fad: an almost neurotic obsession with kids’ mental health. Parents will not punish their children for fear of trauma. Schools meddle with the psyches of their charges. And every child who was once “shy” now exhibits “avoidance behaviors.”
At times, it reads like Shrier herself needs a Xanax, portraying every school as a clinic running experimental trials. She recounts stories of fifth graders crying over distant fathers in glorified group therapy. School days open with mental health check-ins asking if each child has considered slitting their wrists. And she reserves her most vicious rhetoric for affluent parents who wonder if a weighted blanket—never consequences—might keep a toddler from hitting mommy. But such anecdotes, while galling, are hardly emblematic of a science teacher who implements social and emotional learning (SEL) by asking their students to reflect on teamwork after dissecting a frog.
Criticisms out of the way, Shrier’s book levels useful criticism at this therapeutic mindset in the classroom precisely because it receives so little. Students, she points out, are said to be too fragile and traumatized to receive a bad grade. SEL must be a priority in social studies class. Schools should extend the day so students can spend more time attending to their mental health. In each case, these recommendations are emblematic of what Robert Pondiscio has called therapeutic education—the shift of teacher from pedagogue to counselor.
But what if this new focus on mental health in the classroom risks real harm to students?
A central concept to Shrier’s book is “iatrogenesis,” a fancy word for a simple idea: all medical interventions carry trade-offs. Surgery, chemotherapy, anesthesia—each one brings its own perils. In medicine, doctors inform patients of these risks. With therapy, too, psychiatrists and counselors must receive informed consent. But in schools, too few acknowledge that there might be real harms arising from mental health interventions. Shrier provides numerous examples.
Wellness checks may worsen depression or anxiety. Healthy introspection can quickly spiral into a toxic dwelling on negativity, and the best therapies train patients out of these loops of reflection. Few of us are happy all the time, and daily wellness checks or restorative circles keep the difficulties of life ever present in a student’s mind.
Trauma-informed pedagogies risk accommodating emotions in an unhealthy way. Ask any therapist worth their salt and they’ll tell you that avoidance fosters anxiety. No cold calls, no high-stakes exams, only soft deadlines and mental health days. Such accommodation creates fragile students, leaving them less capable and without the coping mechanisms required to face life’s challenges. If we treat a failing grade like an 10/10 crisis and not a minor setback to overcome, our students will respond accordingly.
Another obvious pitfall at the root of this therapeutic shift in schools connects to expertise. The American Psychological Association’s Code of Ethics exhorts mental health providers to only practice within their competence. A therapist specializing in pediatric anxiety should not willy-nilly take on an adult drug addict. Counselors without the pharmacological knowledge of psychiatrists cannot prescribe medications.
Against such exhortations, too many schools have deputized teachers into quasi-therapeutic roles far beyond their training. One adolescent psychology class in a teacher prep program is insufficient to handle the volatility of suicidal ideations or restorative circles, where students are compelled to share their life’s traumas with their gossipy peers.
In one compelling chapter, Shrier takes a wellness screening for children and reworks it for adults, asking “Do you feel emotionally supported by your spouse about the things that matter most to you?” Adults can take a long view in answering that question, overlooking the testy exchange with their spouse the night prior. How might an adolescent respond to questions about their feelings of “hopelessness” after a breakup?
If every child receives regular questionnaires inquiring into their mental state, we risk pathologizing the slings and arrows of daily life. A kid who’s down after a rough day suddenly shows “depressive symptoms.” Nervousness before a test is now “anxiety.” These designations are stultifying, fatalistic even.
Shrier describes the D.A.R.E program as a school-level intervention that famously backfired, fostering curiosity and experimentation with drug use. Regarding various school-based mental health interventions, a handful of high-profile studies recently found several such programs made students more anxious, more depressed, report worse relationships with their parents and peers, and less able to manage their emotions.
These are serious reasons for skepticism that’s sorely absent from the education field. McGraw Hill, a prominent publishing company, insists that every class—especially math class—needs to incorporate SEL. Over half of the country has incorporated SEL into state-level standards, requiring teachers to play mock therapist for their students in between the Pythagorean Theorem and cellular biology. How-tos abound on the topic, with examples of daily “check ins” for teachers and observation checklists for administrators.
Intensive counseling works well for individuals with genuine diagnoses, but we seem to be expecting teachers and school staff to implement therapy-like practices into their classrooms. It’s an error akin to recognizing that chemotherapy works well for cancer patients, and so teachers should administer small doses to all children.
We ought not follow Shrier too far. Without doubt, there’s need for character education in schooling. A few lessons on conflict resolution or emotional regulation are inevitable, especially among younger students. And I worry that readers may take her criticism of permissive parenting too far and never discuss emotions with their children.
Taken as a whole, however, Shrier’s book is a compelling argument against therapeutic education that I hope will encourage districts and advocates to and pump the brakes on this well-intended but risky endeavor.
For many students and teachers, the pivot from in-person to remote learning in March 2020 was a sudden lurch from the known to the unknown. Writ large, research shows the academic impact of that move was devastating. But details matter—and so do exceptions. Little attention has been paid to students who were already familiar with working in remote modes prior to Covid-mitigation school closures. Is it possible that they fared better than their peers, experiencing what amounted to a simple extension of their status quo? New research out of Germany tries to answer that question.
The researchers followed the longitudinal performance of 2,700 students who studied mathematics using an intelligent tutoring system (ITS) called Bettermarks. It is a robust, adaptive online system that can be used in class or at home and covers typical math content taught in grades four through twelve, from basic addition and subtraction to probability and statistics. The “teaching” portion of Bettermarks is mostly overview rather than direct instruction and is primarily used to give students practice problem sets to complete. That said, the system is also interactive—programmed to give feedback when common errors are made (“don’t add the numerators and the denominators,” “find the lowest common denominator,” etc.), provide a finite number of brief hints when requested, and supply students with more practice problems of varying difficulty in areas where they display specific weaknesses. This ITS can be programmed to provide unlimited problem sets upon student demand or to limit available sets to those assigned by a teacher. For this study, students were only working on assigned sets and had to have completed a minimum of five sets in each of the time periods under review.
The analysis covered ITS usage between January 2017 and May 2021 and, significantly, only included students who used the system before, during, and after pandemic-related school closures in Germany. The goal was to get the fullest picture of usage and outcome changes from open schools to closed schools and back again. The analysts particularly focused on potential differential effects for lower- versus higher-performing students, as previous research showed inconclusive and potentially conflicting results for those students. Based on pre-closure performance on ITS problem sets between January 1, 2017, and March 15, 2020, analysts assigned students to either a low- or high-performing group for comparison.
During the first round of school closures in Germany (March 16 through May 31, 2020), all students using the ITS showed an increase in relative accuracy rates compared to their pre-closure performance. Higher-performing students exhibited the highest absolute accuracy in terms of raw scores, but initially-lower-performing students showed larger growth in accuracy in the period relative to the full sample, with their absolute accuracy rising quicker nearer the end of the period.
German schools reopened for in-person learning in the fall of 2020 but closed again on January 1, 2021, for two full months. Looking at ITS performance during this initial return-to-business-as-usual period, the researchers saw a similar pattern to the first closure period, with an overall increase in relative accuracy rates driven by an acceleration of absolute accuracy by initially-lower-performing students. Results during the second closure period (through February 28, 2021) showed a similar pattern. Finally, in-person learning returned for good as of March 1, 2021, and the researchers looked at ITS performance from this point through the end of the school year on May 31, 2021. Perhaps unsurprisingly at this point: The results were the same again.
What to make of this steady pattern of improvement? First and foremost is the evidence that these students were not negatively impacted by the switch from in-person to remote schooling, even through the repeated open-and-close cycles they experienced. In fact, the trajectory for all students in this study was entirely upward, with low-achievers gaining the most. Whatever was going on within their daily class instruction, these findings indicate that students remained engaged and productive. This is not ironclad proof that students became more adept at learning from online resources due to increased use during school closures, although that could certainly be part of the explanation for the results observed. Additional suggestions from the report’s authors include a decrease of “math anxiety” (especially in lower-performing students) due to the lack of face-to-face competition or judgment from peers, and the possibility that use of—and success in—the ITS was being “incentivized” by teachers who made scores part of students’ class grade.
In the end, these findings are part of the complex puzzle of what happened to students as a result of Covid-mitigation school closures. It’s important to know these things, even if we’re just trying to put the pieces together years later—especially the more positive parts of the picture, as small and localized as they may have been.
SOURCE: Markus Wolfgang Hermann Spitzer and Korbinian Moeller, “Performance increases in mathematics within an intelligent tutoring system during COVID-19 related school closures: a large-scale longitudinal evaluation,” Computers and Education Open Journal (February 2024).
Cheers
Jeers
