Over the past nine months or so, our Moonshot for Kids initiative, a collaboration with the Center for American Progress, has given me an opportunity to think a lot about research and development in K–12 education.

It’s made my head hurt. Like, a lot.

Where I keep getting hung up—and based on the majority of proposals to our Moonshot for Kids competition, I don’t think I’m alone—is over the distinction between “research and development” (R & D) and “school improvement.” These are very different concepts. Yet bringing them into productive contact with one another might be the key to getting student outcomes moving in the right direction once again.

First let’s discuss why they should be viewed separately, and then examine what could happen if they were brought into purposeful engagement.

What’s the purpose of R & D?

In any field of human endeavor, the rationale for research and development is the assumption that we can always find a better, more effective, more efficient way to get a job done, or as one frequently-played ad on NPR says, to “solve previously unsolvable problems.”

And to be sure, the problem we want our schools to solve is hard. Consider what we ask a typical high-poverty elementary school to achieve:

Take a group of fifty to one hundred five-year-olds, most of them illiterate and innumerate; many of them lacking in pre-literacy and pre-numeracy skills, plus limited in self-control and socialization experiences; some of them facing severe behavioral and emotional challenges and more than a few with troubled home and neighborhood environments; and, over the course of six years, prepare most of them to succeed in middle school and beyond.* That preparation must include mastery grade-level expectations in reading, writing, and math, plus history, geography, and science, plus essential social and emotional skills, including a growth mindset; perseverance; attention to detail; and teamwork. And do all of this while forming habits of good character and citizenship; creating an appreciation for art and music; providing opportunities for physical exercise; in a community of kindness and respect; and while the children continue to face the various stresses of poverty.

You can see why we likened our education challenges to a moonshot.

A proper research and development enterprise would work at attacking the problem head on. It would focus on the components that schools can control, like the types of people they hire; the roles those people play; the tools they are given; how student time is allocated; how student learning is assessed; how community and parent resources are tapped; etc. The “R” in R & D would focus on discrete questions related to these slices of the schooling enterprise. And the “D” would take insights from the R to build out new tools, technologies, and other solutions, perhaps bundled in completely new and innovative ways.

Thankfully, we have some examples of successful R & D in elementary education already. Decades of NIH-funded research bequeathed us important scientific insights about how to teach children to read, for example that phonics and phonemic awareness must be taught systematically. That science has been baked into high-quality curricula (such as Core Knowledge Language Arts) and inspired helpful tools (such as DIBELS). There are other promising examples in the areas of reading comprehension, math, writing, student discipline, and more. But there are also plenty of blind spots. Can we really say that we know a sure-fire way to teach little kids about history, geography, and science? What about developing good character or understanding our system of government? 

But let’s stipulate that, for elementary schools especially, the R & D process has given us some useful science and evidence-based tools. Why aren’t more schools using them?

That’s where “school improvement” comes in.

What’s the promise of “school improvement”?

The R & D process is inherently top-down. That’s not to say that governments or venture capitalists or philanthropists shouldn’t solicit input from “the field.” Of course that’s essential. But when these folks fund research and development projects, they are making bets on ideas and people that show promise. They are exercising expertise.

But for the fruits of R & D to have an impact in the real world, they must be embraced by some sort of bottom-up process. Someone needs to take a bunch of different ideas and a set of cutting-edge tools and combine them in new and innovative ways. As a hammer needs a carpenter, or a blueprint needs a foreman, R & D needs a receptive audience to bring it to life in the real world.

In the private sector, this task is handled largely by the likes of Silicon Valley, via the profit-motived-fueled phenomenon of creative destruction, as start-ups and their investors attempt to apply R & D to real-world problems and create value at scale.

That motive is curbed almost into oblivion in public education, thanks to our “system” of 14,000 government-run local monopolies. Sure, there’s plenty of money in their combined $650 billion budget that could pay for new products and tools and technologies. But there’s a reason that the VC crowd has been relatively uninterested in K–12 education: the inclination of our schools to spend any marginal dollars on additional staff instead of tools or technologies that might make their current staff more effective or efficient. In fact, as Mike Antonucci pointed out last month, the latest data indicate that we are doing it again, hiring staff like there’s no tomorrow.

Without a profit motive, then, we need a different kind of process that might lead schools to search for, find, and successfully implement promising tools and technologies that result from R & D. And we call that process “school improvement.”

This is what’s advocated by Tony Bryk and the Carnegie Foundation, in the form of “networked improvement communities,” or as the Bill & Melinda Gates Foundation calls them, Networks for School Improvement. They encourage schools or systems to work together to identify solutions to common problems, implement those solutions thoughtfully, collect and analyze data on how things are going, and iterate.

And if done well, the bottom-up School Improvement Networks should find the fruits of the top-down R & D investments and figure out how to make various solutions fit together in a coherent way.

Is goodwill enough?

Is this theory of action naïve? You don’t have to be a cynic to think that “the system” is never going to focus on doing “what works” if “what works” goes against the political interests of “the system.” Which, in many communities, is employing as many people as possible.

That’s one reason why I believe that accountability-for-results remains an essential element of tomorrow’s education ecosystem, why at the very least we need A–F systems or the like to shame local systems into balancing their small-p political interests against the urgency of getting better results for kids. It’s not enough, but at least it should nudge schools to consider investing in better tools and strategies instead of just hiring more people. Maybe it will help if those tools or technologies are cheap or free.

It’s also why I support charter schools, which face the dual incentive structure of accountability for results and the obligation to satisfy their customers. They have to get good results or die. And they have the autonomy to think creatively about how to best use their staff, time, and financial resources to best serve their kids. It’s no coincidence that high-poverty charter schools tend to get better outcomes than their district peers, given their incentives, agility, and ability to search out what works and implement it in their classrooms. If you want to see what that looks like, go visit an Achievement First, or Success Academy, or IDEA, or KIPP elementary school, or any of the other “Whatever it Takes” (“WIT”) charter schools nationwide.

—

In sum, R & D is not the same thing as school improvement. R & D is top-down; school improvement is bottom up. R & D is about creating solutions to thorny problems; school improvement is about taking those solutions and fitting them together in ways that work for particular children, communities and circumstances. R & D is about tools and technology; school improvement is about how to best implement them in the real world.

R & D and school improvement are not the same. But they need each other like bees need flowers, and flowers need bees. We must invest in both for them both to flourish.

* This actually understates the level of difficulty, since most high-poverty elementary schools must deal with enormous student turnover as well.

Among several disturbing elements of Christina Cross's New York Times op-ed, “The Myth of the Two-Parent Home,” is the author’s seeming belief that unless growing up with married parents has the same effect on black children as on white youngsters, it is not worthy of endorsement. Otherwise, asserts the Harvard postdoctoral fellow, “blindly promoting the merits of marriage and the two-parent family is not the answer.” Her purpose appears to be to show how black families do in comparison to white families, as opposed to what the absolute impact of strengthened family structure would be on black children. Take this excerpt:

Although in general, youths raised in two-parent families are less likely to live in poverty, black youths raised by both biological parents are still three times more likely to live in poverty than are their white peers. Additionally, black two-parent families have half the wealth of white two-parent families. So, many of the expected economic benefits of marriage and the two-parent family are not equally available to black children.

Yet the research Cross herself cites from the National Center for Education Statistics tells a far more hopeful story about the impact of family structure. Consider what we see in figure 1:

Figure 1. Percentage of children under age eighteen in families living in poverty, by child’s race/ethnicity and family structure, 2017

Source: Characteristics of Children's Families, National Center for Education Statistics, U.S. Department of Education. Last updated: May 2019.

In 2017, only 5 percent of white children under eighteen residing with their married parents lived in poverty, versus 12 percent of black children living with their married parents. That’s a relative multiplier of 2.4, not the exaggerated “three times” that Cross estimates with her obsession with black-white comparisons. But far more importantly, we see that the proportion of black children under eighteen living in poverty decreases from 45 percent for mother-only households and 36 percent for father-only households to just 12 percent for married-couple households. That’s a huge difference and an enormous improvement that seems not to interest Dr. Cross.

To repeat: Being raised in a married-couple household led the poverty rate for black children to go down 73 percent compared to mother-only households and 67 percent compared to father-only households. And as evidence of the power of family structure to transcend race, 31 percent of white children raised in mother-only households live in poverty, versus just 12 percent of black children living with their married parents. That is a stunning realization.

Yet Cross narrowly cites evidence that seeks to denigrate the two-parent household as a marginal thing for black children, while ignoring the data that actually cement the difference family structure plays in the lives of children of all races.

—

Another strange statistical technique practiced by Cross is lamenting the fact that black Americans have disproportionately high levels of family instability while not disclosing figures that would illuminate the extent of the issue. I would urge that it is better to share the raw data, so that people can assess the actual numerical explosion in non-marital birth rates among blacks and then reasonably conclude for themselves whether these statistics do actually matter. Take this excerpt:

America has a long and troubled history of viewing racial inequality primarily through the lens of family structure. Recall the publication of the controversial Moynihan Report in 1965, in which government officials pointed to higher rates of “female-headed households” and “out-of-wedlock births” among black Americans as key drivers of the disadvantages they face.

When Moynihan produced “The Negro Family: The Case for National Action,” he was drawing attention to what he saw as a crisis in a segment of the black community that was mired in poverty. According to the 1965 report, 23.6 percent of all black babies were born outside of marriage.

If the same attention had been paid to that glaring statistic then as has been paid to structural barriers around race, steps could have been taken to arrest the continuing decline in family formation that has occurred over the last five decades, not only in the black community, but across all races. Indeed, according to the most recent Census data, in 2018, the non-marital birth rate among black women reached 69.4 percent and the overall non-marital rate was 39.6 percent. As Moynihan himself warned again almost thirty years ago in 1991, “Illegitimacy levels that were viewed as an aberration of a particular subculture twenty-five years ago have become the norm for the entire culture.”

—

This raises yet another peculiar aspect of Cross’s argument. By singling out black Americans as the “group whose family structures have long been used to explain their social and economic disadvantages,” she obscures the fact that the black community isn’t the only place that’s suffering from the consequences of family breakdown. Consider another excerpt:

Black Americans have the highest rates of single-parenthood and nonmarital births in the country, and this divergence from the two-parent family ideal is routinely implicated in the lower levels of educational attainment and higher rates of poverty and unemployment within the black community.

Family breakdown is unarguably bad for black children. But what is happening with other Americans? Again using birth data for 2018, table 1 shows the non-marital birth information for women age twenty-four and under. Yes, it’s staggering that 91 percent of the babies born to black women under age twenty-five were outside of marriage. But what of the 61 percent of babies born outside of marriage to white women age twenty-four and under? Their more than 238,000 children dwarf the numbers in other racial categories. 

Table 1. 2018 U.S. births to women age twenty-four and under, by race

According to ChildTrends, the percent of births occurring to unmarried women has grown most rapidly among white women. And there is increasing evidence that opioid deaths and other issues facing particularly white working class men are linked to a breakdown in family stability.

Perhaps if there wasn’t such a narrative insisting that the breakdown of the family is a “black problem,” more agreement might be reached around the idea that the decline in family structure is an existential challenge facing communities of all backgrounds and one that all should tackle together.

—

Ultimately, Cross makes her argument clear:

[W]hat deserves policy attention is not black families’ deviation from the two-parent family model but rather structural barriers such as housing segregation and employment discrimination that produce and maintain racialized inequalities in family life.

Does it have to be one or the other? Why it is so difficult to have a nuanced conversation that both structural barriers and factors like family structure matter to determining child outcomes? Most importantly, what do we teach the next generation about their ability to develop the personal agency to overcome those barriers? If you are twelve years old, do you have the power to solve housing segregation?

Perhaps the next generation of students should learn that a staggering 97 percent of millennials who obtained at least a high school degree, worked full-time, and married before having children are not poor. Rather than pummel young people over barriers for which they have no control, maybe it would be good to share the power over decisions that they do have to determine their own destiny.

—

In his powerful Washington Post essay, “Don’t deny the link between poverty and single parenthood,” Robert Samuelson says, “If, magically, a third of America’s poor escaped poverty, the change would (justifiably) be hailed as a triumph of social policy.” Similarly, in her New York Times op-ed, Ms.Cross’s own analysis suggests that, while strengthening family structure is no silver bullet, there is much to be hopeful for if family structure among blacks and kids of all races were more stable. That is not a myth.

Fordham has produced The Supplemental Curriculum Bazaar: Is What’s Online Any Good? Worth reading!

Are popular materials offered on Teacher Pay Teachers, and similar sites, useful?

Not really, say the authors. They mostly suck. They looked at English materials. One big concern was the lack of directions:

For example, one poorly rated writing task asks students to select five important events from their life and write an autobiography but offers little guidance beyond that.

Stacey Childress responds to Bazaar. She correctly cautions us to wait a minute. While she likes the idea of curriculum reviews—she provided seed funding to what became EdReports—she asks why do teachers use this “low-quality” stuff. It’s because, she explains, teachers trust other teachers—not expert raters or principals.

So let’s dig in.

1. Why don’t teachers trust “the technocrats”?

One reason, which Robert Pondiscio points out in his most recent opus, is that “we” technocrats often lack humility.

Those of us—and I include myself—who seek more creation and use of high-quality curricula…we need to look ourselves in the mirror before we critique what individual teachers use and create.

The “Good Stuff”—those highly rated lesson plans? They don’t seem, when adopted by districts, to drive up student achievement.

Tom Kane’s recent study hit that point in math.

David Steiner’s recent essay reluctantly makes the same point more broadly. He writes:

I predict that in the coming months, we will see more such findings to the effect that new high-quality curricula aren’t achieving much of anything for our less prepared students—and the critics are already vocal.

The top down approach to assigning expertly-designed-and-validated curriculum hasn’t yet worked in real life. We haven’t exactly, as a field, copped to that failure. It doesn’t mean we shouldn’t try. Just that we should admit where we are. If the stuff we say is “good” doesn’t actually drive student achievement, we should be really careful about saying somebody else’s stuff isn’t good.

Individual teachers may not know the data about a particular curriculum, but they do know that lots of “highly-rated things” have failed in their personal experience.

2. Engagement versus achievement

Tom Arnett chimes in with a recent paper for AEI. He argues that another reason which teachers reject district-provided “high-quality curricula” because they are often picking a different “Job To Be Done” than trying to vault achievement. Sometimes they just want to increase student engagement, or make their own teaching more enjoyable.

Tom is right. “The Job” many teachers are “hiring” curriculum supplements to do is student engagement. Not achievement. The Supplemental Curriculum Bazaar seems to reject that quest on its face. They’re not even seeking to measure if a lesson might increase engagement.

An analogy: As parents, we don’t only give kids healthy food (at least this one). We might say: “I want my kid to enjoy this museum. We just pulled into the parking lot. He’s hangry. Here are cheese crackers and a Capri Sun. Yum. Now he’s not hangry. We’re enjoying the museum.”

Curriculum “purists” essentially demand parents give that kid carrot sticks and organic apple juice, always, leading to a parking lot showdown. Those purists might tell American parents to be more like French parents. Just ignore the crying when it comes to food, and everything will work out, goes the French conventional wisdom.

I’m skeptical. Most teachers realize that when they download “Sight Word Scramble” (fifth most popular on TPT) that it’s meant to be fun. They realize when they download the comic-book-illustrated-style “What if the World had 100 People?” (fourth most popular lesson plan on TPT) that it appears more engaging than their district-approved social studies lesson.

Engagement is a fully legitimate goal! When a reviewer sees a movie with Jonah Hill or Seth Rogen, we know the Job To Be Done is stupid laughs—i.e., engagement. They have to evaluate whether the movie accomplishes that job. They can’t say, “Wow, really not even close to Citizen Kane.” Teachers want to know if material is good for its genre, not to be told that only one genre, achievement gain, is worthwhile.

Similarly, to answer one of Bazaar’s metrics, when thousands of teachers repeatedly download lessons “without clear directions,” it may be because they’ve already got a workable set of directions in their heads.

New composers need cheat sheets for orchestrations, the “clear directions” for things like pizzicato, overblowing, bowing at the bridge, flutter tongue. But experienced composers have that stuff in their heads already. Materials for experienced composers don’t need explicit directions for that reason.

Mike Petrilli and Amber Northern acknowledge both these points in the foreword to Bazaar:

They (teachers) may be finding value in these materials use the materials to fill instructional gaps, meet the needs of both low and high achievers, foster student engagement, and save them time. They rarely use the materials as is.

But that message doesn’t quite get picked up. Bazaar got headlines like this from Hechinger Report: “Most English lessons on Teachers Pay Teachers and other sites are ‘mediocre’ or ‘not worth using,’ study finds.”

3. Watching movies versus reading scripts

Petrilli and Northern offer a great idea:

Just as the movie review site Rotten Tomatoes offers both critical evaluations and “audience scores” from viewers, these curriculum web sites should provide both experts’ opinions and information about the popularity of various lessons.

But one important distinction is worth noting.

The Rotten Tomatoes reviews are not written by people who read the script. They’ve watched the movie.

Consumer Reports does not rate a car solely by sitting inside it or studying its specs. Yes, they do that. But they also (and more importantly) actually drive the car.

Yet, typically, curriculum reviewers simply read lessons and imagine them in their minds’ eyes. That is a very difficult way to gauge what happens in real life when that lesson is taught.

Hollywood and Broadway executives know this. Sure, they pay people to “evaluate scripts,” but that’s not the main way they decide what movies to greenlight.

I would urge curriculum reviewers to rate curricula by watching several teachers actually use it. They need to have the experience, as I have, of watching teachers do exactly what you ask them to do and seeing it suck.

I realize that watching lots of lessons sounds logistically implausible. But it’s not!

I worked at a large education organization in Africa, with several hundred schools in five countries. We created our own curriculum. Our original review process was to hire experts to examine it, such as department heads at ed schools and so forth.

Yet if our team would visit the actual schools, often the most ambitious, “highest-rated-by-experts” lessons fell flat with real-life teachers and students. Meanwhile, some of the simpler-seeming lessons actually led to better results.

Even though these schools operate with 1 percent of the per-pupil spending we have in the U.S., the organization’s CEO was willing to invest in creating a new qualitative observation system to improve the curriculum. We hired ten experienced local teachers in our five countries. They wandered the country, visiting a different school each day. They’d observe class after class, sending detailed notes and ratings, not about what was in the lesson plan, but what actually transpired in the classroom from the lesson plan. We switched from inputs to outputs.

Little by little, we were able to grind our way to better lessons, both from teacher perspectives and from the achievement data. I have to believe that if that sort of feedback loop can be created in some of the poorest schools in Africa, back here in the U.S. we could figure out a way to provide EdReports and others with the money they’d need to have raters watch actual lessons across an array of schools and school types.

Then we'd have the chance for a “Tomatometer Expert” who is more trusted by teachers…because the feedback would be grounded in real life outputs (how the class actually goes) rather than inputs.

More importantly, we’d create a path for a series of grind-it-out, long-additive improvements—A/B testing of little things in lessons—that would perhaps allow us to one day see scholars publish studies of curriculum implementation that leads to huge gains for students.

Mike Goldstein disclosures: I am a volunteer board member for Match Education, which creates curriculum that is (highly) rated by EdReports. I am narrowly expressing my individual view here and speak for nobody else. I spoke briefly to the creators of Fordham’s Curriculum Bazaar study in its early stages. My kids do not eat enough vegetables.

1. Our first piece today is yet another look at the EdChoice voucher program from the perspective of a suburban school district leader. That is, a district who has had just four students ever leave the district for a private school via a voucher. Our own Chad Aldis is interviewed here making the point about magnitude. But why, I ask you, should the district’s perspective on this matter as much as (or more than) a parent’s perspective? Why are the parents never interviewed? (News 12, Cincinnati, 1/6/20) The matter of perspective is raised loud and clear in this opinion piece from Ed Post, responding to a previous national voucher-bashing piece. Fordham’s annual Ohio By the Numbers report is quoted and linked here, discussing vouchers’ minimal share of our state’s education expenditures. (Education Post, 1/6/20)
    
2. The interim CEO of Lorain City Schools held a convocation of staff this week, his first in the new job. He had a lot of positive stuff to say, but I was struck by this: “I am not here to bring a bunch of change. But I think it's important to see what we currently are doing, what's working, what's not working and how do we tweak it to make it better or maybe it falls off the table.” It is noted that he is planning to keep former CEO David Hardy's "Big Board," or "excellent schools dashboard," a system used to track academics, student attendance and behavior, and building atmosphere. You know, the data that Hardy used in his successful effort to bring the district’s report card up from and F to a D last year? For some reason, though, continued use of this data was characterized in the story as something of a worry (for whom, you may well ask). Luckily, the interim CEO figured it would be and thus added: “My concern is that we move forward and we keep moving forward and we keep doing better things in terms of achievement and get out of academic distress.” Yeah. Like was happening before you got here. (Elyria Chronicle, 1/7/20)
   
   

3. The Columbus City Schools board of education met for its first meeting of the year this week. Among other things, the board voted to approve the sale of five unused buildings in the district. Sounds like the board, under new leadership for the first time in a while, will have its hands full tussling with the historic preservationists on the fate of one of those facilities. But here’s hoping that there are lots of awesome charter schools champing at the bit, cash in hand, to snatch up any and all of those buildings. (Columbus Dispatch, 1/7/20)
    
4. A new report from the Ohio Department of Higher Education indicates that fewer Ohio graduates in the Class of 2018 needed remedial courses when they enrolled in college. ODHE officials couldn’t really say why the reduction occurred (continuing a downward trend over the last few years), and a group of college professors interviewed were downright dismissive of the findings: “It remains to be seen whether there has been real improvement in college readiness or simply some gimmicky addressing of the problem.” Oooh. Academia burn! (Dayton Daily News, 1/8/20) Speaking of data at odds with other data, Dayton City Schools announced this week that it is seeing progress in its efforts to increase attendance for students (and staff) in the district. While officials noted five very specific points of improvement, this data is at odds with what was reported mere weeks ago regarding the effects of their new busing program on attendance. (WDTN-TV, Dayton, 1/7/20)
   
   

5. Finally today, good news for Parma City Schools as they were removed from “fiscal caution” status, and a certain amount of oversight by the state as a consequence, after four years. How’d they do it? Oddly enough, by being more fiscally responsible. Or, as the district supe put it: “If you look at our salaries compared to some neighboring districts, we’re still below where we want to be, but leaders of our negotiating teams understood the fiscal parameters that we have to deal with and really look to the better of the district as opposed to sometimes the better of just themselves. That’s an important consideration in being fiscally solvent.” In other words, not spending money you don’t have helps keep you out of debt. Wonder if anyone else has stumbled on this secret formula? (Cleveland.com, 1/8/20)

Did you know you can have every edition of Gadfly Bites sent directly to your Inbox? Subscribe by clicking here.

We are endlessly tempted—and strongly encouraged by OECD’s Andreas Schleicher—to infer policy guidance from PISA results. If a country’s score goes up, maybe other countries should emulate its education practices and priorities, as they surely must be what’s causing the improvement.

The temptation is obvious, yet is best resisted, for the same reason that one ought not suppose that someone’s elevated temperature is the result of their fitness regimen (or lack thereof). A fever is generally the result of a bug—bacterial or viral—and is not caused by a surfeit or dearth of weightlifting and treadmilling. Similarly, a rising PISA math score among a nation’s fifteen-year-old population might be the consequence of new curriculum—or better teacher preparation—or longer school years—or heightened accountability pressure—or school choice—or stiffer college entrance requirements—or greater prosperity that enables parents to keep more of their kids in school longer—or changing societal mores regarding the education of girls. Or whatever. Schleicher and his team, as well as innumerable wonks and policy makers, around the world, just love to detect their favorite policy at work in a country with rising scores and then seek to propagate it elsewhere on the planet. They’re even happier to finger places with sagging scores for doing things they don’t approve of.

Yet much like a thermometer can yield an important clue that one’s health is not good—and a rising or plunging barometer usually signals changing weather—PISA scores are a valuable source of data on matters that any sane education policymaker wants to know: Is my country getting better or worse on the three core subjects that PISA tests? How does it compare with other countries that matter to us? And—digging in a little deeper—what do we see about the relative performance of boys and girls, of advantaged and disadvantaged youngsters, and of other important demographic groups?

PISA scores fluctuate, of course, and small bumps up and down don’t reveal anything significant. But when a country’s scores dramatically or steadily rise, or persistently decline, it’s at least a signal to those responsible for that country’s education that something good or bad is happening. And—let’s admit it—big ups and downs in Country A’s scores are also sure to catch the attention of policy wonks in Countries B, C, and D, whether or not their scrutiny lead to sensible inferences about causation.

Recent rounds of PISA have illustrated all these things at work. Why are China’s scores so high? Is it great schooling or totalitarian manipulation of who gets tested—or maybe some of both? What is it about Singapore that causes that wee nation’s lofty scores to keep rising? What’s the secret sauce in Estonia these days?

Back in 2001, German officials were appalled by how poorly their young people scored on the first round of PISA, and the resulting “PISA shock” catalyzed multiple changes in the FRG’s education system.

Without doubt, however, the most famous case of what we might term PISA envy is Finland, whose unexpectedly lofty scores back then—on three consecutive rounds of testing through 2006—triggered widespread interest in how that small Nordic land was handling K–12 education. The flames of fascination were expertly fanned by Pasi Sahlberg, who wrote and spoke convincingly (and in English) of the Finnish education miracle and whose Harvard Ed School affiliation lent added credibility to his insights.

Ed-policy tourists trooped to Helsinki and most of them loved what they saw there, because it wasn’t about school choice or test-based accountability, the kinds of policy reforms making major inroads in the U.S. and elsewhere on the planet, but about tolerant, well-educated, well-compensated, high-status teachers with plenty of planning time, hard at work in a uniform system of government-run schooling. It’s no exaggeration to say that, for a decade or so, just about every progressive educator on earth yearned to work in (or lead) a system like Finland’s.

What happened thereafter, however, cooled some of that ardor.

Beginning with the 2009 assessment round, Finland’s scores sagged in all three PISA subjects, a decline that continued in 2012, 2015, and in the 2018 results released last month. Not only did average scores sink but rich-poor gaps have widened and gender gaps are sizable.

Why this fall from Finnish education glory? That’s an excellent question, nicely explored by The Economist in a recent article titled “The parable of Finland.” The most interesting insight to be found there—at least in my eyes—is that Finland practiced a more traditional form of education (textbook based and teacher driven) in the 1990’s and the kids sitting for the early rounds of PISA had attended such schools. “By the time the results came out [however], many Finnish schools had started to move in a very different direction,” far more student-centered, discovery-oriented, and with teachers serving as guides-on-the-side. The uber-progressivism of recent years at least correlates with the sagging scores that Finland has lately notched on PISA.

Fascinating, yes, but also speculative. It could also be climate change, alterations in the reindeer migration, shifts in the Russia-Finland relationship, or perhaps the amazing new library in central Helsinki distracting kids from their lessons. Further analysis is definitely called for, yet we are unlikely ever to know for sure. What is certain is that, like the body temperature on that thermometer, PISA scores can helpfully signal that something is changing, but in and of themselves they cannot explain what’s causing it.