As we were putting the final touches on our new report, The Supplemental Curriculum Bazaar: Is What's Online Any Good?, Amazon unveiled a “new storefront” called Amazon Ignite. The site will allow educators to earn money by publishing—online, of course—their original educational resources (lesson plans, worksheets, games, and more).
The e-commerce titan’s entry into the curricular marketplace is obviously motivated by a perceived market opportunity—and that’s not wrong. The vast majority of teachers are supplementing their core curriculum or don’t have a core curriculum to start with, so it’s no surprise that they often frequent the online arena to obtain the materials with which to meet their instructional needs.
In fact, recent studies by RAND found that nearly all teachers report using the Internet to source instructional materials, and many of them do so quite often. For example, 55 percent of English language arts (ELA) teachers said they used Teachers Pay Teachers for curriculum materials at least once a week. That site reports that one billion resources have been downloaded—a massive number, to be sure.
Yet we know almost nothing about the quality of such supplementary materials. Although several organizations have stepped up to offer impartial reviews of full curriculum products, to our knowledge there’s no equivalent when it comes to add-on resources. Therefore, we set out to answer a simple question: Are popular websites supplying teachers with high-quality supplemental materials?
We recruited University of Southern California associate professor Morgan Polikoff to lead the review. He has conducted numerous studies on academic standards, curriculum, and assessments (including a previous Fordham study on Common Core–era tests), and he co-leads a federal research center on standards implementation. Jennifer Dean, an expert in assessment, standards alignment, and ELA content, served as lead reviewer of materials and assisted with report writing. She was joined by four other expert reviewers with backgrounds in teaching ELA, developing curricula and assessment items, and/or leading instructional teams.
Morgan and Jennifer and their team, with the help of external advisers, developed a rubric that captured both the overall dimensions of quality in curriculum materials—things like rigor and usability—and more discrete dimensions that reflected the key instructional shifts called for by the new generation of states’ ELA content standards: things like regular practice with complex texts and reading and writing tasks grounded in evidence from the text. In all, they examined over three hundred of the most downloaded materials found on three of the most popular supplemental websites: Teachers Pay Teachers, ReadWriteThink, and Share My Lesson.
This crackerjack review team unearthed a wealth of valuable information (encapsulated in nine key findings) that has important implications for district, school, and instructional leaders everywhere, as well as for classroom instructors themselves.
Sadly, the reviewers concluded that the majority of these materials are not worth using: more precisely, 64 percent of them should “not be used” or are “probably not worth using.” On all three websites, a majority of materials were rated 0 or 1 on an overall 0–3 quality scale.
That’s sobering to say the least, particularly given the popularity of these sites and the materials we reviewed. It suggests a major mismatch between what the experts think teachers should (and shouldn’t) use in classrooms and what teachers themselves are downloading for such use—and, in some cases, paying for.
That’s not necessarily a criticism of the teachers. They may be finding value in these materials in ways that we “experts” need to better understand. In interviews, teachers told us that they use the materials to fill instructional gaps, meet the needs of both low and high achievers, foster student engagement, and save them time. They rarely use the materials as is. Much adapting goes on as they choose and modify items to fill specific needs—needs that likely take precedence day to day over whether particular materials are aligned to state standards or incorporate high cognitive demand (or some other quality valued by experts).
We’re not suggesting that teachers’ views and judgments should yield to those of experts. Why not weigh both? Consider how this works on Rotten Tomatoes, the popular website that reviews the quality of movies and other entertainment. Their Tomatometer is based on the opinions of hundreds of film and television critics and is a trusted go-to for millions of viewers. When at least 60 percent of the critics’ reviews of a movie or TV show are positive, it receives a red tomato, meaning it’s “fresh.” Less than 60 percent and it gets a green splat, meaning it’s “rotten.”
Those could reasonably be termed expert judgments. But Rotten Tomatoes also provides Audience Scores, which are just that. When at least 60 percent of viewers give a movie or TV show a star rating of 3.5 or higher, a full popcorn bucket indicates that it’s “fresh” from the audience’s perspective. When less than 60 percent, a tipped-over popcorn bucket reveals it’s “rotten.”
So the moviegoer and television watcher can readily access two different ratings—one from professional critics and another from the audience. Often they’re similar, but not infrequently, they diverge. It’s hard to say who is “right,” but potential viewers get more information by seeing both ratings than they would from just one.
Same thing here. By definition, we looked at materials with high “Audience Scores,” which is to say these were materials that had been downloaded the most. Yet in a majority of cases, our expert critics gave them a green splat, even though teachers rewarded them with a full popcorn bucket.
What then? Should we search for ways to block or deter teachers from using materials that experts don’t like? Some on our team would welcome such a heavy-handed approach to monitoring supplemental resources, perhaps by empowering district leaders to enforce stringent policies about which supplemental resources would be allowed in their schools. We understand that impulse. It recalls an argument we often have with libertarians over school choice, wherein we think it’s sometimes necessary to close really bad schools even though parents may like them.
In this case, however, we think a better solution is simply to provide teachers with more information, Tomatometer style. In addition to proving user reviews or comments to teachers, or highlighting and promoting the most popular lessons, the platforms should also make expert reviews available.
Two additional points are worth mentioning.
First, as our title indicates, the online marketplace is a bustling bazaar of cacophonous activity with myriad offerings of every sort. We cannot claim that our results apply to the thousands of other online resources out there for educators nor even to everything on the sites that we did evaluate. There’s no way to evaluate it all, and undoubtedly, much of what’s on offer is worth using. Yet we can state with some confidence that most of the most popular items leave much to be desired.
Second, not everyone will agree with our criteria and methods for assessing these materials. Even within our review team, not everyone was satisfied with every part of the process or with the conclusions about some materials. In some cases, we may have been too easy on the materials. In evaluating alignment, for instance, we simply asked whether the materials aligned to the standards that the teacher developers said that they aligned to. Similarly, a key expectation with assessments was that they cover the key content of the lesson.
In other cases, maybe the bar was too high. For example, we looked for cultural diversity by seeking the inclusion of multiple authors from diverse groups and/or topics of diverse cultural importance. Whether that’s a reasonable expectation for any one supplemental item (versus a full-fledged curriculum) is certainly debatable. Ditto in expecting supplementary lessons to offer supports for most or all student subgroups, given how inadequately many full-bore curricula handle differentiation.
Regardless of their quality, one of the things that can get lost when teachers go trawling for supplemental materials is curricular coherence. As such, we agree with Morgan and Jennifer that school leaders and department heads should pay more attention to what’s actually taught in classrooms by way of supplemental materials. What they learn could inform an array of subsequent strategies for improvement, from offering teachers training in how to identify high-quality materials to publishing a list of curated supplemental resources and addressing shortcomings and gaps in their core curriculum (the work of the Louisiana Department of Education may be instructive here).
Teachers are understandably hungry for instructional stuff, but the sites they’re turning to are often providing subpar versions of it. We hope that they make improvements going forward. And we also hope that Amazon, the “most valuable company on the planet,” will learn from its predecessors and strive to beat them at the quality game.
