A brand-new evaluation from the Center for Research on Education Outcomes (CREDO) at Stanford University offers promising signs that Ohio is on strong footing, especially with its brick-and-mortar charters.
A brand-new evaluation from the Center for Research on Education Outcomes (CREDO) at Stanford University offers promising signs that Ohio is on strong footing, especially with its brick-and-mortar charters. At the same time, the analysis reminds us of the hard work ahead to either harness or radically improve the online education model. CREDO examines state exam data from 2013-14 through 2016-17 to gauge the impact of charter schools on student achievement in reading and math.
More trouble in online schools
The bad news first. Previous studies have found that pupils in online charters lose significant ground in math and reading. Sad but true, this study showed the same thing, in spades. CREDO finds large negative impacts, especially in math where online charter students experience losses of 0.23 standard deviations (sd) compared to their “virtual twins” attending district schools—a result equivalent to falling behind almost a full school year. The impacts in reading were also negative, though smaller in size—roughly the same as losing forty-six days of learning in a 180-day school year.
These sizeable losses, in conjunction with the fact that online schools enroll almost a third of the entire charter sector, weigh down Ohio’s overall performance. With the inclusion of e-schools, the average Ohio charter student performed on par with district peers in reading and worse in math (-0.07 sd). Omit the online schools, however, and the picture brightens considerably, as we see in Figure 1. Brick-and-mortar charters are just as effective as their district twins in math, and superior in reading, providing the equivalent of twenty-four extra school days of learning.
Figure 1: Charter impacts for Ohio overall and by delivery model, 2019 CREDO analysis
Note: * significant at p<0.05 ** significant at p<0.01. A result without an asterisk indicates no significant difference between charter and district students’ growth. This note also applies to figures 2 and 3.
To their credit, Ohio policymakers haven’t sat idly by on online schools. While its results are included in CREDO’s analysis, ECOT—the state’s infamously low-performing online school—is now extinct as a result of state action. Moreover, legislators are considering a move to competency-based funding that, by providing dollars when e-schools help students meet academic targets, could lead to improved achievement.
Good news from brick-and-mortar charters
Some will doubtless take the overall charter results, weighed down by e-schools, as evidence that the sector remains stuck in low gear or worse. But in the words of football analyst Lee Corso, “Not so fast, my friend.” Two critical pieces of evidence indicate that Ohio charters—the brick-and-mortar kind—are on the upswing—and doing some needy kids a great deal of good.
The analysis below considers results from both CREDO studies, the new one and the Ohio study released in 2014, which enables us to examine data over a nine-year interval.
Strong academic gains for black students
The most encouraging finding is charters’ positive effects on black students, children who often face the greatest barriers to academic success. The figure below shows that, in the 2019 analysis, the average African-American youngster enrolled in a charter school enjoyed significant gains of 0.10 and 0.04 sd in reading and math, respectively when compared to their “virtual twin” attending district schools. These results translate into an equivalent of fifty-nine additional school days of learning in reading and twenty-four extra days in math, based on a 180-day school year. Also noticeable is the stronger charter impact compared to the previous study, which found no statistically significant difference in the growth of black charter students versus their closely matched peers in both subjects.
Because the vast majority of black charter students attend brick-and-mortar schools—roughly 90 percent of them in 2015-16—we can be confident that these positive findings are attributable to the efforts of brick-and-mortars. Of note, too, while they comprise a very small fraction of charter students (about 5 percent), the results for Hispanic pupils were no different than their peers attending district schools.
Figure 2: Charter impacts for black students, 2014 and 2019 CREDO analyses
Solid elementary and middle charter-school performance
CREDO also finds positive and improving results for charter elementary and middle school students. Consider figure 3, which shows that elementary charter schools in the 2019 study produced positive impacts in reading—a noticeable improvement over 2014—though no effects in math. Meanwhile, middle school charter performance improved against the already strong results reported in 2014. In the new study, their students posted impressive gains of 0.18 and 0.17 sd in reading and math, respectively, equivalent to approximately 100 extra days of learning. Unfortunately, the results for charter high schools still lag, though their students comprised less than 5 percent of CREDO’s charter sample. Finally, we observe large negative impacts in multilevel charter schools that are almost certainly being driven by e-schools, the largest of which serve students across grades K-12.
Figure 3: Charter impacts by grade configuration, 2014 and 2019 CREDO analyses
* * *
To distill the results: Ohio’s charter sector is akin to a football team with an anemic defense and high-flying offense. The online charters can’t seem to tackle anyone, while the brick-and-mortars light up the scoreboard. If this were your favorite team, you’d be screaming for changes to the defense while praising the quarterback to the high heavens.
Something similar should happen in Ohio charter policy. The state needs to find ways to dramatically improve e-schooling. But, in the realm of brick-and-mortar schools, we ought to recognize the great work they’re doing—and do more to help them become even stronger. After years of shortchanging charter students, lawmakers should finally move to fund brick-and-mortar charters equitably, helping to kick-start new-school formation and the rapid expansion of the state’s top-performing charters.
As one of the pioneering charter states, we’ve learned a lot in two decades in Ohio. The progress can seem maddeningly slow at times, but the state is at last on the right track in its turn to higher quality. Let’s work to fix the remaining problems in the charter sector, and build on its strengths. Onwards!
When I visit classrooms with the principals I coach in urban, suburban, and private schools, it’s obvious how hard most teachers are working, how much care they put into setting up their classrooms, and how much gumption it takes to work with young people six hours a day. I see lots of effective teaching—practices that one might rate as level 3 or 4 on a four-point rating scale—and very little that’s horrible, or level 1. But I do see a fair amount of mediocre, level 2 practices—for example, teachers giving out low-rigor, fill-in-the-blanks worksheets, calling only on students who raise their hands, or failing to answer kids’ unspoken question, “Why are we learning this?”
Human behavior naturally falls on a bell-shaped curve, so every school will have some variance in classroom performance. But that inevitably produces inequitable results. Effective practices are especially beneficial to students who walk into school with any kind of disadvantage, and these same students are disproportionately harmed by mediocre and ineffective practices—kids who don’t raise their hands when the teacher asks, “Any questions?”; who haven’t yet learned how to work around their disabilities; who are dealing with a family breakup or other trauma; who are openly defiant or sit in sullen silence.
Truly bad teaching obviously needs to be addressed immediately, but so do mediocre practices, which all too often fly under the radar, and are also not okay. Lesson by lesson, day by day, month by month, denying these kids quality instruction widens the achievement gap and robs them of the American dream.
We know what doesn’t improve sub-optimal classroom practices: school leaders making once-a-year announced-in-advance observations with copious documentation; superficial “walk-throughs” to fill out checklists; interrupting teachers midstream to show how a lesson should be taught; using student survey data as a high-stakes cudgel; mounting cameras in classrooms; and worst of all, using student test scores and value-added data to judge teachers’ performance—a practice that is now thoroughly discredited. We shouldn’t be surprised that these and other practices driven by distrust and compliance have never shown up in the research on effective schools. That dog hasn’t barked.
School board members, superintendents, and union leaders should demand much more of their evaluation policies and rethink their collective bargaining language, driven by these questions: How often are teachers observed each year? Are supervisory visits announced or unannounced; in other words, are visitors seeing daily classroom reality or a dog-and-pony show? How long do supervisors stay, and what are they looking for? Do they check in with students and look at what they’re working on, and afterward is there a conversation with the teacher? Is the process of documenting visits overly burdensome? Are supervisors themselves supervised and coached in a meaningful way, not only on classroom observations, but also on how well they orchestrate teacher teamwork, another key driver of instructional improvement? And what goes in a teachers’ personnel files at the end of each school year, summing up performance on all aspects of their work?
I’ve grappled with these questions over my five decades as a teacher, central office administrator, principal, leadership coach, and research reader, and I’ve concluded that there’s a far better way of supervising, coaching, and evaluating teachers. Here are the key elements:
This approach takes about the same number of educator hours as traditional evaluations, but is vastly more authentic and effective at understanding and improving teaching and learning. (A number of other educators have developed similar approaches.)
It’s also important that teachers feel safe with frequent, unannounced visits. I’ve found that most teachers trust this system if the following are in place:
Principals need to be prime observers, perhaps taking on the most challenging classrooms and orchestrating the work of instructional coaches and peer observers to see high-performing teachers, where a “softer,” more collegial approach would be appropriate. The more observers there are, the better the observer-teacher ratio will be, and the more frequent visits and follow-up conversations will be. The goal is meeting the three essential needs of every front-line professional, which are, according to Douglas Stone and Sheila Heen’s Thanks for the Feedback: appreciation, continuous improvement, and knowing where they stand.
Because this system is structured in a way that avoids bureaucratic nonsense and liberates and develops the skills observers already have, administrators, instructional coaches, and peer observers will, in most cases, have the chops to do these mini-observations well. Here’s why:
For educators who are new to this approach, there will be a learning curve. But the good news is that the mini-observation and feedback skills are eminently coachable by superintendents or their designees through the use of co-observations and discussions of case studies and write-ups in leadership meetings.
In my work in a wide variety of schools, I’ve found that this process brings out the best in observers, helping them continuously grow as coaches of effective instruction. The result: more good teaching in more classrooms more of the time. And that is the key to raising the next generation of well-educated Americans and closing our social-class and racial achievement gaps.
This weekend saw the passing of Siegfried “Zig” Engelmann. He was a giant among educators and a true social justice warrior. By that, I mean that he dedicated his life to improving outcomes for the disadvantaged, not that he was some great wet lettuce who opined about emotional labor and set his critics three books to read before he would discuss anything with them.
In the late 1960s and early 1970s, Engelmann took part in the largest educational experiment of all time—Project Follow Through. The aim was to give disadvantaged kids the best possible start to their education. The planned U.S. government funding was cut by congress, so what was intended initially to be a large-scale intervention was stripped back to an experiment. The idea was that of “planned variation.” Different groups would develop their own programs and implement them in different centers. Outcomes from the different programs would then be compared in order to see which was the most effective.
Engelmann and his colleagues developed the Direct Instruction model. This is not to be confused with just any form of explicit teaching. The Engelmann model had unique features that drew upon a specific theory of instruction and, controversially, employed the use of scripted lessons. Apparently, scripting the lessons was never the original intention, but the first teachers involved in the programs kept going off-piste.
Direct Instruction was the clear winner of the Project Follow Through experiment. Despite being labeled by researchers as a “Basic Skills” program, it was the best performer in terms of higher order cognitive skills, such as reading comprehension and mathematical problem solving, as well as the affective domain of student self-esteem. This is despite the fact that many competitor programs were explicitly set-up to focus on cognitive and affective skills; most of these actually had a negative impact.
This evidence has been largely dismissed by the educational establishment. Because it was so big, Project Follow Through was also messy, and so you can poke holes in the methodology. You might think that this would prompt academics to setup more rigorous trials of Direct Instruction and the alternatives, but they have been strangely reluctant to do so. This means that much of the further research on the method has been conducted by Engelmann and his small group of Direct Instruction advocates centered on the University of Oregon. This then provides another line of attack for those who wish to dismiss the research.
Why is it so important to dismiss Direct Instruction? It is inconvenient to the dominant ideology of educational progressivism or, latterly, constructivism. Who cares about kids’ life chances when there is an ideology at stake.
This means that an independent school like mine will make use of Engelmann’s programs, but his work is either unknown or dismissed in the schools and kindergartens he was trying to help.
I think all of us would like to leave the world a better place than we found it. Zig Engelmann certainly achieved this, but I cannot help feeling that it could have been even better if more of us had listened.
A version of this essay was originally published on the author’s blog, Filling the Pail. The present version was edited slightly per the Fordham Institute’s style guide.
There are two ways to read this report of a first-of-its-kind arts education experiment from Rice University’s Kinder Institute for Urban Research. The first is with a measure of relief. The authors claim measurable academic, social, and emotional outcomes associated with arts education for elementary and middle school students. Bravo! The second is with a touch of vexation, perhaps sadness, or even resignation. Do we really need a randomized control trial study to justify the arts as an essential part of a well-rounded education? If a rock-bottom basic function of K–8 schooling is to expose children to the broadest range of human knowledge, discovery, and expression (that is the function, right?), then a permission slip is hardly needed to include the arts. What would we do if the results had proven no measurable outcomes? Would some argue to remove the arts from the school day?
Alas, that ship has sailed. Arts education has been in decline since the 1980s, note Daniel H. Bowen of Texas A&M, and the University of Missouri’s Brian Kisida, the present study’s coauthors. The emphasis on standardized testing in reading, math, and other core subjects since the passage of No Child Left Behind has also “coincided with notable declines” in exposure to the arts in school, they report.
Bowen and Kisida studied data and survey results from schools participating in the first two years of the Houston’s Arts Access Initiative, an ambitious program that aims to increase access to arts education, particularly for disadvantaged students, through partnerships with local artists and cultural institutions. Because the initiative was oversubscribed in its first two years, Kisida and Bowen had the opportunity to compare outcomes among students at participating schools to those who applied but were deferred—a study group comprising more than 10,000 students in forty-two schools. The results is the first ever, large-scale, randomized controlled trial study of such a program. The arts educational experiences had “remarkable impacts” on academic, social, and emotional outcomes, the duo reports.
Compared to students in the control group, kids exposed to the various arts opportunities in the Houston study experienced “a 3.6 percentage point reduction in disciplinary infractions, an improvement of 13 percent of a standard deviation in standardized writing scores, and an increase of 8 percent of a standard deviation in their compassion for others.” Among elementary school students, arts education also enhanced school engagement, college aspirations, and empathy with others—a finding that should resonate particularly among those focused on social and emotional learning. In an essay for Brookings’s Brown Center, Kisida and Bowen note that school systems rarely collect data on arts programs. Moreover, they write that “the most promising outcomes associated with arts education learning objectives extend beyond commonly reported outcomes such as math and reading test scores.” This is encouraging news and appears to be borne out in their findings, but let’s not be too quick to assume no benefits in general literacy. Kisida and Bowen report positive effects on student writing. A linchpin argument of E.D. Hirsch’s Cultural Literacy and subsequent works is that mature language comprehension rests on a body of knowledge common among writers and readers, speakers and listeners, who are broadly fluent in their shared histories and heritages, including the arts. In a well-rounded education focused on raising readers, art should have no lesser place of privilege than history, science, or literature—it’s all part of the mental fitments that furnishes the minds and enlivens the discourse of literate Americans. This powerful literacy-based argument for arts education has been available to advocates all along, and should weigh strongly against any impulse to reduce arts education “because testing.” Nonetheless, Kisida and Bowen note that “a critical challenge for arts education has been a lack of empirical evidence that demonstrates its educational value”—a sentence that I hope was as hard to write as it is to swallow, given that they also note that an “overwhelming majority of the public agrees that the arts are a necessary part of a well-rounded education.”
There was an interview more than a decade ago with Geoffrey Canada, the founder of the Harlem Children’s Zone, describing education funders’ demands that he demonstrate the value of arts education for his students. Canada grew exasperated noting the $700 billion spent on Wall Street bailouts in 2008. “Then somebody asks me if kids should take violin and do I have evidence?!”
Well, now you have some.
SOURCE: Daniel H. Bowen and Brian Kisida, “Investigating Causal Effects of Arts Education Experiences: Experimental Evidence from Houston’s Arts Access Initiative,” Rice University (February 2019).
One of the ways the U.S. Department of Education is supposed to facilitate student success is by awarding federally funded grants to various state and local programs, then independently assessing the performance of those grantees. Doing the latter job well will be increasingly important and challenging under the Every Student Succeeds Act, which allows for significantly more flexibility on how grantee programs can be designed and ran than under No Child Left Behind, the law’s predecessor.
To find out how well the department has done this, what it has done to correct any issues, and what difficulties lay ahead, the U.S. Government Accountability Office (GAO) conducted an analysis between February 2018 and January 2019. It examined documents related to grantees and ED’s assessments of them, fiscal and annual reports, and reports on management challenges from the Education Department’s Office of the Inspector General. The GAO team also interviewed officials from three other ED offices—the Institute of Education Sciences; the Office of Planning, Evaluation, and Policy Development; and Office of Primary and Secondary Education.
They analysts identified four key challenges.
First, the Education Department has persistently struggled with monitoring grantee performance and ensuring compliance with federal regulations. A GAO report in 2017 found that these duties were regularly performed inconsistently, especially between ED offices, and that key documents, such as those detailing program impacts achieved by grant funds, were missing from almost every file within the sample, leaving $21 million dollars not correctly accounted for. Fortunately, the department under Secretary of Education Betsy DeVos is instituting changes meant to correct these deficiencies. It’s establishing official department-wide procedures for documenting grant records, and improving oversight and monitoring by creating its own resources and examples that state and local agencies can look to when, among other things, collecting data and measuring academic outcomes.
Second, GAO found that persistent issues with effectively and accurately collecting and reporting valid and useful data has limited the Education Department’s ability to assess grantees. The agency’s Office of the Inspector General also reported issues with federal control over state-reported data. In response to these challenges, ED has improved its data systems and offered states support in how to better collect and report their data.
A lack of financial and human capacity is the third issue the department has faced. Fortunately, on the money side, ESSA gives the Education Department authority to consolidate funding for grantee evaluation and monitoring, as well as the ability to choose which programs it evaluates, and what criteria it uses. Unfortunately, human capital is still an issue, as the agency has struggled to fill vacant positions with qualified staff. The department is currently reassessing their hiring procedures for the purpose of improving them and hiring more qualified people.
Finally, the Education Department faces multiple methodological limitations in program assessment, which include determining what constitutes a positive outcome and how to measure it, isolating impact, and quantifying long-term outcomes. Some grantees—like the Promise Neighborhoods program—have submitted data, but the agency has not established how to use this information for program evaluation. Many large-scale federal grants also have vaguely defined objectives that limit ED’s control over their specific goals and outcomes. But the agency is taking steps to fix this. For example, it has improved control over Title I grants by conducting interim measures of progress toward grantees’ long-term goals.
Though the U.S. Department of Education has made many strides in recent years to improve the way it monitors and evaluates recipients of federal grants, the flexibility of ESSA will complicate these efforts and cause new problems as time goes on. It’s a work in progress.
SOURCE: Jacqueline Nowicki et al., “K-12 Education: Challenges to Assessing Program Performance and Recent Efforts to Address Them,” U.S. Government Accountability Office (January 2019).
On this week’s podcast, Mike Petrilli, Checker Finn, and David Griffith discuss Checker’s experience on Maryland’s Commission on Innovation and Excellence in Education, and its bold proposals for improving the state’s schools. On the Research Minute, Amber Northern examines the benefits of arts education.
Daniel H. Bowen and Brian Kisida, “Investigating Causal Effects of Arts Education Experiences: Experimental Evidence from Houston’s Arts Access Initiative,” Rice University, Kinder Institute for Urban Research (February 2019).
