To be clear, I am in favor of building a strong education R & D sector. However, it’s important to acknowledge the serious shortcomings of the current system. It is because of this current state that I am arguing that evidence-based practices don’t work.
I’m making two claims.
- The current evidence-base in education research is extremely thin at best and completely misleading at worse.
- By their very design, studies that result in evidence-based practices discount externalities instead of solving for them. This is not helpful for educators working in real schools and classrooms. The idea that many interventions are effective in some places but almost none of them work everywhere is such a common idea in the education research sector that this phenomenon has its own name—effects heterogeneity.
This post will address the first claim and Part II will address the second claim.
Defining evidence-based practices
School leaders tasked with improving student outcomes have an extremely complex series of decisions to make about the interventions and strategies they are going to use in order to bring about this improvement. These leaders are often encouraged to choose evidence-based interventions. By definition, evidence-based interventions are tools, materials, or sets of routines, typically grounded in theoretical principles, that have been subject to rigorous empirical study.
There are two important assumptions here. The first is that the intervention has been subject to rigorous empirical study in the form of randomized control trials, but the reality is that many education studies do not meet this gold standard of research design. Second, even if the intervention has been subject to well-designed research, we only know that it can work because it has somewhere. However, it is very likely that the “somewhere” in the study is very different in key ways from where my improvement effort is taking place.
Misleading evidence base
An example using Ohio’s Evidence-Based Clearinghouse will illustrate my first point. Let’s say that I am an elementary principal working at a high-poverty school with a high percentage of struggling readers. I go to the clearinghouse in order to find a reading intervention program for my struggling students. The first thing that I have to do is understand the Evidence Level system on the site. Ohio’s Clearinghouse uses the system proscribed in the Every Student Succeeds Act which outlines three primary Evidence Levels for interventions including:
- Level 1 (Strong Evidence): At least one well-designed and well-implemented experimental (i.e., randomized) study.
- Level 2 (Moderate Evidence): At least one well-designed and well-implemented quasi-experimental (i.e., matched) study.
- Level 3 (Promising Evidence): At least one well-designed and well-implemented correlational study with statistical controls for selection bias.
These levels can be misleading. The Level 1 designation of Strong Evidence has more to do with the strength of the research design of the studies used to evaluate the intervention and less to do with the strength of the student outcomes of the intervention itself. I think this would come as a surprise to most people using the clearinghouse.
Thin evidence base
Let’s assume for a minute that as the elementary principal researching reading programs, I can get past the issue with the Evidence Levels. I start searching for elementary reading interventions that meet the Level 1 threshold. One of the programs that meets this criteria is Corrective Reading. On the Ohio Clearinghouse, there is a short description of the program and a note that additional information is housed on the Evidence for ESSA site. So, I continue on this trail to the Evidence for ESSA site and find that the Corrective Reading evaluation was based on a single study that included 206 students in urban and suburban areas surrounding Pittsburgh, Pennsylvania. Significant positive effects were found on the Woodcock Word Attack assessment (effect size = +0.15), and these qualified Corrective Reading for the ESSA “Strong” category. However, most other outcomes (including state reading tests) did not show positive outcomes, so the overall average effect size was only +0.06.
And then I move on to the single study referenced by both clearinghouses (Ohio’s and the Evidence for ESSA site) on which the Strong rating is based. Besides the evidence base relying on a single study with a relatively small number of students, there are a number of other very concerning issues with serious implications for the elementary principal which include:
- The effects of the program did not carry over to outcomes with which schools would be most concerned (e.g., state test scores).
- Teachers in the study received seventy hours of professional development which included three phases: intensive training, practice, and implementation.
- Three primary groups were studied: African American, white, and economically disadvantaged. The program was only effective with the white students in the study.
So, if I am that elementary principal researching reading interventions, and I only used Ohio’s Evidence-Based Clearinghouse to identify my intervention, I likely would have grossly misunderstood the research underlying Corrective Reading. First, the program wasn’t effective in changing outcomes on meaningful assessments such as state reading tests. Second, implementation required significant amounts of teacher professional development, but this expensive training only produced small effects on an obscure assessment tool. And those effects were only demonstrated with white students in the study and not with African American nor economically disadvantaged students.
A better way
There has to be a better way to help frontline educators identify effective interventions. As designed, Ohio’s Evidence-Based Clearinghouse is as likely to mislead and confuse as it is to help. Even under the best of conditions, strong studies that result in evidence-based strategies only answer the question—“what works?” As we saw in the case of Corrective Reading, it is debatable if even that question was answered.
But real educators in real schools need answers to an altogether different set of questions—“what works, for whom, and under what conditions?” This is the evidence of know-how, and the subject of Part II of this series.
Editor’s note: This essay was first published by the School Performance Institute.
