NAEP is by far the country’s most important source of information on student achievement, achievement gaps and so much more, even though it’s invisible to most Americans. Yet NAEP is far from perfect—and could do so much more than it does. It’s time to wrestle with its challenges, shortcomings, and possible future scenarios.
Tuesday marked the formal debut of the book that’s consumed most of my work life over the past two years, the book I generally think of as a “biography of NAEP” or “the past, present and future of NAEP.” I’m sincerely grateful to the Harvard Education Press for publishing it, to Fordham’s Pedro Enamorado for helping in so many ways to get it done, and to the Smith Richardson Foundation and Carnegie Corporation of New York for helping make it happen.
This has mostly been a labor of love, as NAEP has intersected my own life and career in many ways. That started in 1969 when, at the unripe age of twenty-five, I found myself behind a desk in the Old Executive Office Building as a very junior participant in Nixon’s White House staff, brought there by my graduate school advisor and mentor, the late Daniel P. Moynihan, who had just become a senior member of a multi-headed domestic policy team back in those heady pre-Watergate days.
NAEP was a toddler then, recently entrusted to the then-new Education Commission of the States, whose executive director evidently thought he should explain it to this green education staffer at the White House, probably because the federal government was paying for it—and wasn’t paying enough. That visit introduced me to the National Assessment, and it’s been in and out of my life ever since. The busiest phase came in the decade after 1985 when, first as Bill Bennett’s assistant secretary for research and then as first chair of the new National Assessment Governing Board, I had a fair amount to do with nudging NAEP onto what might be termed its modern trajectory. In the years since, I’ve kibitzed, advised, scribbled, and occasionally nudged some more, but from outside. (You’ll find a bit more kibitzing below.) Still, if I were hit by a truck tomorrow and someone tried to figure out if there was anything worth putting into an obituary, I’d hope that my recurring connection with the Nation’s Report Card might make it into the third or fourth paragraph.
In a couple of short recent pieces, I’ve termed NAEP “the most important test you’ve probably never heard of.” That’s not as true of Fordham readers, most of whom have at least one foot in the ed-policy or ed-research worlds where NAEP is well known and much used. But in the broader education universe, and certainly in the lives of ordinary parents and citizens, the “Nation’s Report Card” is essentially invisible.
In fact, when people asked me of late what I was working on and I said “a book about the past, present, and future of NAEP,” except among fellow ed-wonks, the response was often a blank look and polite smile. Educated and worldly though my interlocutor probably was, maybe even a close relative, he or she usually had no idea what I was talking about.
The main reason is that NAEP has no immediate impact on most Americans. It tells them nothing about their own child, their own school, or (with a handful of exceptions) their own community. From the standpoint of a parent or grandparent, local taxpayer, teacher, or principal, it’s just not very relevant.
Yet NAEP is, and for decades has been, America’s premier gauge of whether its children—all our children—are learning anything in school, whether they’re learning any more today than years ago, and whether the learning gaps among student groups are narrowing or widening. Moreover, it supplies that information not just at the end of high school, not just among college-bound students, not just among fifteen-year-olds, and not just for the entire country. Nor is NAEP confined to reading and math. Rather, it (usually) reports on pupil achievement in grades four, eight, and twelve. It does that for as many as ten different subjects. And it does so (at least some of the time for some of those subjects in some of those grades) not only for the nation as a whole, but also for every state and for several dozen big-city school systems, as well as some categories of private school.
Also important: NAEP is far more than a simple barometer gauging changes in education performance. It also enables users to determine whether what their state expects of its fourth graders, say, in math, or eighth graders in English, is up to a widely accepted version of national standards for those subjects, namely the trio of “achievement levels” devised—and faithfully defended—by NAEP’s governing board.
Because NAEP makes accommodations for children with special needs while requiring schools to include most such children in the tested population, it both supplies accurate information on how they’re faring academically and sets a good example for states’ own assessment regimes. Because it employs a variety of test questions, not just the multiple-choice kind, it provides sophisticated information about how well young Americans can perform important advanced assignments such as writing coherent sentences and paragraphs that explain things and cite evidence.
Besides all that, NAEP subtly causes U.S. education to improve by providing educators with feedback on what groups of pupils can and cannot do in the subjects that matter most, and it provides this in ways that illustrate how many are good (or not) at various elements of those subjects.
That’s why it’s important, even if it’s invisible to most Americans. Yet NAEP is far from perfect—and the second half of the new book wrestles with current challenges, shortcomings, and possible future scenarios.
Some of NAEP’s issues were also laid out in a recent review by the National Academies of Science, Engineering, and Medicine (NASEM), and some have been acknowledged in a long “we’re working to fix it” blog by NCES Commissioner Peggy Carr and NAGB executive director Lesley Muldoon. Others will doubtless turn up in an audit spearheaded by IES director Mark Schneider.
NAEP today is creaky in its technology, expensive, cumbersome in its decision-making, and getting shaky in its governance. For me, its biggest single shortcoming is the failure to systematically supply state-level data at the twelfth-grade level. That’s the end of high school, the point at which the products of our K–12 system emerge into college, career, citizenship, and adulthood. One might expect state leaders to demand systematic NAEP-style data at that point above all others. Yet it isn’t happening.
Looking ahead, my top anxiety for NAEP is the possible politicization of its governing board, known as NAGB. There’s a special place in my heart for this body, and fond—perhaps too rosy—memories of how, despite its meticulous, bipartisan, “Noah’s Ark” composition, over the years its twenty-six members have almost always managed to come together in pursuit of a shared vision of NAEP’s role and function. That capacity for consensus at the top—and the board’s independence from Education Department dictates, partisan tensions, and interest group hang-ups—has arguably been NAEP’s single greatest asset and the most important guarantor of its integrity and respect-worthiness.
But as culture wars permeate just about everything else nowadays, there are signs that they’re making their way into NAGB, too. The board nearly came to blows over its recently-concluded new reading framework and is currently heading into the stormy waters of science, likely followed by the even stormier territory of civics and history. Can the center hold? Can consensus still prevail? I wish I were confident.
Meanwhile there’s this book you can now read. The first half is history, which I expect most everyone will generally agree with. The second half, dealing with current issues and future scenarios, may prompt more controversy.
But please have a look and see for yourself. More importantly, please do all you can to preserve and strengthen the most important test that most Americans have never heard of.
Georgia is the latest on a growing list of states that make financial literacy courses a requirement for high school graduation. Call me a curmudgeon, Scrooge, or “griper,” but I’m skeptical.
From the outset, delivering financial instruction to students of any grade feels a bit like “cargo cults.” Kids wave about mock budgets and shift around fake money in the hopes of mastering financial knowledge, the utility of which for them lies over the horizon. At best, they’ll retain a vague understanding of compound interest. More likely, they’ll forget whatever instruction they received by lunchtime, let alone remember it years later when they actually need it.
The Federal Reserve has a comprehensive analysis of financial literacy interventions. Their top-line conclusion reads: “Improved financial behavior does not necessarily follow from financial information.” Other research confirms such lackluster results.
The terms “behavior” and “information” are essential to understanding why. Financial security requires not just knowledge, but action, self-control, and life habits. Regarding physical health, we understand that a donut is unhealthy—this is common knowledge—but we choose to eat it anyway. “Don’t buy stuff you cannot afford” is not some abstruse dictum, a piece of occult knowledge waiting for instruction to reveal it. In both cases, the knowledge necessary for physical and financial health are commonly known, but the behavior is the lacking component.
Not all evidence points to wasted effort. The same review from the Fed references studies that linked state-mandated financial ed curricula to “increased asset accumulation” in adulthood, and other surveys associate instruction with a higher likelihood of having a bank account. On the whole, however, school-delivered financial literacy courses seem largely ineffectual; personal habits, life experiences, and access to financial knowledge as an adult are far more important factors—all factors that schools cannot influence.
Honestly, though, whether financial literacy classes are effective is the wrong question to ask. Even if research found them increasing rates of saving or decreasing the rate of loan defaults, I’d still be skeptical of their inclusion in schools. The proper question to ask about any educational practice isn’t if it works, but how it compares to other interventions—and uses of the schools’ limited time and leverage.
Right now, the average school day lasts about 6.5 hours. With core classes—math, science, literature, and history—plus time to eat, engage in some physical activity, and gain a passing exposure to the arts, what does that leave time for? On top of that, schools are also de facto counseling centers, welfare distribution hubs, and soup kitchens. In the past decade, they’ve effectively been asked to solve every societal issue from racial disparities and drug abuse to obesity.
How much are we asking schools to achieve? Do we want to heap yet another adult problem onto their endless to-do list of things to solve? Can we continue to ask our schools to bear the weight of society’s problems? How long until Atlas shrugs?
That’s not to say that schools are ineffectual in every endeavor that they pursue. Time and again they’ve proven that through targeted academic interventions—implementing “the science of reading” or a knowledge-rich curriculum for example—they can, have, and do drastically improve student achievement. Yet when schools go beyond their remit and try to solve all of society’s problems, they accomplish little, even as such interventions cut into standard academic instruction.
What’s more, if we want our students to be financially successful, perhaps more important are the academic skills and knowledge that we can impart to them now, setting them up for career success. As such, a better intervention to address adult financial woes would be improving the core academic functions of the system while also incentivizing alternate career paths through CTE. Then, maybe, fewer students would sail off to college, take out loans, and find themselves in the very debt we’re trying to avoid, opting instead for practical careers.
Schools, when they’re good, are very good at academic instruction. However, when society sees a problem and gestures at the classroom, suggesting “let the teacher deal with it,” they more often than not do a middling job of solving the problem while giving short shrift to their core academic focus.
Should public schools strive to teach character to students? One group in Texas says no. Mary Lowe, of the Tarrant County chapter of Moms for Liberty, recently spoke before the state board of education in opposition to SB 123, which would expand character education in Texas schools. Hollie Plemons, a parent, also spoke to the board. They’re mine,” Plemons said of her children. “They don’t belong to the government. They belong to me. I don’t want the government to teach my children character. That’s my job, and it’s a job I want to keep… Please stay in your lane… Focus on academics, and let the parents parent.”
This sentiment—which draws a hard line between the responsibility of the parent to raise a child and that of the school to educate a student—is part of a larger trend. As Rick Hess has noted, “Social and emotional learning (SEL) has become a new flashpoint in our educational culture wars.” For some families, teaching a social curriculum takes precious time away from academic subjects like reading and math. For others, like Lowe and Plemons, SEL seems to be tantamount to indoctrination. There’s a risk of that, to be sure, as some SEL programs are bunk. Just as some ELA curricula do a terrible job of teaching reading, some SEL curricula are heavy on questionable content and light on research documenting their effectiveness. And proponents of SEL have not been as transparent as they should be about what’s included in curricula, often disagreeing about what belongs in the tent. But that should not disqualify the entire project of SEL, much less efforts to teach character.
Both tend to get framed improperly—and incorrectly—in the current political debate. Conservatives tend to favor character education because it seems in line with traditional virtues and values, while progressives seem to favor SEL because it’s seen as a forward-thinking way to teach interpersonal skills and self-awareness. Although this distinction gets brought into sharp relief in op-eds and social media, it blurs in classroom practice.
Look to Responsive Classroom—a widely used classroom management system that weaves SEL practices into daily instruction—as an example. When done with fidelity, the first six weeks of the school year are dedicated to community building, setting norms, and learning routines. Thereafter, the first thirty minutes of each school day are dedicated to a morning meeting that reinforces SEL concepts like active listening and identifying emotions.
It is through these practices—not in spite of them—that virtues are taught. When children practice taking turns while speaking, they are learning patience. When they listen intently to a peer share a story during carpet time, they are learning empathy. When they identify their own negative emotions before erupting in a tantrum, they learn forbearance and moderation. And while some enviable children exhibit these character traits naturally, most need to be taught through explicit instruction. That means character education, and that means at least some versions of SEL.
Jennifer Frey has written extensively—and excellently—on the importance of character education. In a recent article, she writes, “From an early age [...] young children need to be exposed to stories that show them what good and bad characters are through plot that reveal the narrative dimension of their lives.” Frey is correct, and much of the SEL I led as a teacher began in texts like When Sophie Gets Angry, Big Al, and kid-friendly translations of Aesop’s fables. But the lessons rarely ended there. They were followed up with practice, practice, practice. Most children, especially young children, learn by doing. You can explain to children how to ride a bike, but they master it only when they hop on the seat and ride—and maybe fall once or twice and then get back up and ride some more. The same holds for empathy, courage, and perseverance.
Sure, one can point to examples of SEL gone awry and stories of teachers inserting their own political biases into classroom instruction. But we should not fall into what Nate Silver terms “nut picking”: singling out the worst or most offensive aspects of a group to discredit the whole. Properly understood and practiced, SEL is politically neutral because the children of Democrats and the children of Republicans both need to learn how to share crayons and use kind words. If an SEL lesson seems to be leading students to eventually vote a certain way, the teacher has crossed a line and parents are right to call it out. But these examples would be the “nuts” that fail to represent the whole.
Nor should SEL turn into “permission structure to downgrade the academic mission of schools.” As a teacher, I found that the time invested in SEL paid dividends later in the year. Children who cannot regulate their emotions, children who are in conflict with their peers, and children suffering from trauma won’t be doing much learning. In these cases, SEL makes academic learning possible, and teachers are courting disaster if they skip it. Talk to any teacher who has had a student suffer from abuse, neglect, or the death of a parent: A child in crisis is a child who is not ready to learn sight words or multiplication tables. Most children do not come to school as blank slates ready to receive knowledge. They bring with them the residue, good and bad, of their home lives, and this needs to be addressed by any teacher who cares enough to see it.
This does not diminish parents’ role in the development of children’s character. Surely, the family is the “little platoon” that is the foundation for society. Ms. Plemons is correct when she states that she is fulfilling a parent’s duty by teaching her children character. However, the family is not the entirety of society, and children have public facing lives in the same way adults do. The set of character traits that are introduced in the home need to be elaborated and developed for a larger sphere. Society is broad, diverse, and sometimes chaotic. The classroom, and by extension the school, is a natural place to develop these traits before children enter adulthood. It’s much easier to practice sharing and taking turns in a group of twenty-five than in a nation of 330 million.
Parents should not recoil from all forms of SEL. At the same time, those promoting SEL need to do a better job about engaging those parents. Families have a right and responsibility to know what’s being taught in the classroom. Teachers would do well to focus on the lessons that have buy-in from home, and perhaps avoid the squishier, New Age–inspired content around happiness and self-actualization.
Mostly, teachers just need to be explicit about which character traits they are teaching to students and be consistent with their follow-through. There’s even a time and place to teach edgier concepts like social justice, provided there is parental buy-in. (That’s easier to achieve in a school of choice, which should cause progressives to reconsider their opposition to charter schools.) When there is mutual trust, families and schools can work together to teach children the social skills required for success in school and life.
School reopening decisions following emergency pandemic closures have been sources of much parental angst, pundit caterwauling, and political controversy. Now that most of the dust has settled, new research gauges what was really happening in U.S. schools during this turbulent time.
The analysis comes to us from the IZA Institute of Labor Economics and was conducted by two economists from Drexel University and the Université du Québec. They begin with scheduling data from the school-calendar software provider Burbio and link them with the pandemic-era tracker Return2Learn to determine whether public and private schools reported being open for fully in-person or hybrid learning for every individual school day during the 2020–21 school year, which is to say the school year most thoroughly affected from beginning to end by the plague that is still with us. They cover nearly 70,000 traditional district, charter, and private school buildings across the country.
What schools say they are doing and whether students are actually in buildings could be two different things. The analysts thus match anonymized data from more than 40 million cell phones via Safegraph to individuals actually traveling to these school buildings during times they were meant to be open for teaching and learning. “Dwell time” at school sites is used to eliminate people visiting briefly for meals, homework pickup, and the like. The study uses the term “effective in person learning” (EIPL) to denote provision of on-site education that was actually attended by students—with “effective” meaning “being in effect” rather than “being successful at a task”—but it is important to note that this research does not determine the percentage of enrolled students in their building when opened. It is likely to be far less than 100 percent of pre-pandemic numbers.
Although EIPL had dropped below 20 percent of its pre-pandemic level in most places during spring 2020 when Covid first hit, it rose to move than 50 percent on average during the 2020–21 school year. However, large differences were evident. Some of these were regional. For instance, some cities in Florida (such as Jacksonville, Orlando or Tampa) showed EIPL averaging more than 75 percent from September 2020 to May 2021. Conversely, in some western cities (such as Los Angeles, Portland, and Seattle), EIPL remained at an average of 20 percent or less more than a year into the pandemic.
The analysts also find clear cut disparities by school type. Briefly: Public schools provided substantially less EIPL than private schools, with charters ranking below traditional district schools and private nonreligious schools ranking below private religious schools. For both public and private schools, EIPL was lower in more affluent and educated localities with a larger share of dual-headed households, but also for schools with a larger share of non-White students. For public schools, EIPL was negatively related to pre-pandemic school test scores, school size, and school spending per student. Additionally, and perhaps alarmingly, EIPL was negatively related to federal Elementary and Secondary School Emergency Relief (ESSER) funding per student, despite this effort being expressly aimed at helping schools reopen in person learning. The researchers don’t have a plausible explanation for this inverse relationship, although they note that previous research suggested that only a tiny fraction of ESSER funds had been spent on anything as of August 2021. It has also emerged that schools have used Covid-relief money for lots of initiatives that have very little to do with students actually being in their buildings.
In looking at non-school factors that might correlate with EIPL patterns, the analysts found that most of the inverse relation of EIPL with local affluence, education, parental structure, and school funding can be explained by—wait for it—whether the state’s governor was a Republican and the margin by which the local county voted Republican in the 2020 presidential election. Covid vaccination rates also predict higher EIPL (more vaccinations, more in-person learning), while a state’s teacher unionization rate predicts lower EIPL (more union teachers led to less in-person learning, everything else being equal). In contrast, local Covid case and death rates and whether a school is located in a city, suburb, or town/rural area do not have significant predictive power. Explanations for these observations are not offered. The report’s authors are also clear that no causal relationships can be determined with these data. But they do strongly suggest that school reopenings were “driven in large part by policies that align with local political preferences.” Or, more briefly: All the Twitter memes you read or posted were accurate.
SOURCE: André Kurmann and Etienne Lalé, “School Closures and Effective In-Person Learning during COVID-19: When, Where, and for Whom,” IZA Institute of Labor Economics discussion paper (January 2022).
One common refrain in debates around education is that standardized exams negatively impact applicants from disadvantaged backgrounds. In recent years, many colleges and selective K–12 programs around the U.S. have been limiting or abandoning these measures—a trend only accelerated by the Covid pandemic.
The use of standardized exams, however, may function to replace other, more-discretionary criteria like subjective personality assessments or personal and family connections when judging applicants. Indeed, these assessments were devised in the twentieth century partly as a way to minimize the influence of prestigious boarding schools, social class, and geography in college admissions. This meritocratic impulse built on an earlier push in the late nineteenth century to use standardized tests for awarding government jobs.
A new NBER study draft from University of California, Davis economists Diana Moreira and Santiago Perez explores the social and equity impacts of the 1883 Pendleton Act, which mandated that federal civil servants be hired on the basis of competitive exams, rather than the “spoils” system of political patronage used before. What can we learn from studying its effects?
The analysts constructed a new dataset over the period 1871–93 by matching federal employee names to census records, including their sex, race, immigration history, and geographic origin. Moreira and Perez also managed to match thousands of employee files to information related to socioeconomic class, including the individual civil servants’ former careers, families’ careers, and approximate household wealth. They then divided careers—both individual and family—into five categories based on education levels: professional, white collar, farmer, blue collar, and unskilled laborer.
They then created a “social background index” aggregating information on parental wealth, parental occupations, and demographics like race and immigration history, such that a low score corresponds to a disadvantaged background. Then they divided employees into a “treated” group, those subjected to testing beginning in 1883, and a “control” group of federal employees still hired by discretion, who were usually at the top and bottom of the pay scale.
Moreira and Perez find that after the introduction of testing, the median social background index score for treated federal employees rose by 0.17 points. In real terms, this captures the researchers’ observation that, out of a pool of thousands of employees, from 1881 to 1885, the adoption of standardized testing for mid-level federal civil servants is associated with a shift in their background away from immigrant and upper- and lower-class Americans and toward native-born and middle- and upper-middle (professional-) class Americans.
The researchers interpret this to mean that the Pendleton Act reforms increased the share of middle-class “educated outsiders” in the federal bureaucracy. While the legislation reduced the share of individuals from disadvantaged backgrounds entering the federal workforce, Moreira and Perez suggest that this may have been a consequence of the reduced influence of the “spoils” system for awarding federal jobs, which favored political connections like Irish-immigrant urban machines such as New York’s famous Tammany Hall. The researchers also mention that, although the Civil Service Commission argued that the Pendleton Act reforms would increase the number of African Americans employed, statistical estimates find no impact or change in racial representation in the federal workforce.
As for how this may relate to standardized testing in American education, Moreira and Perez argue that their findings point to an equity-efficiency tradeoff, in that the use of standardized exams may improve the quality of the bureaucracy at the cost of “equitable” or diverse representation across social backgrounds. Yet this is dubious. To validly constitute such a tradeoff, you’d have to somehow lose out on “fairness” to gain efficiency. If the researchers believe that adopting standardized exams in 1883 cost society fairness, they are embedding several assumptions into this welfare analysis: that the federal bureaucracy in 1885 was better than in 1881; that innate talent was uniformly distributed along class, education, and immigration lines; that the “spoils” system better reflected this uniform distribution than standardized tests; and that the federal government should have targeted “innate” talent rather than demonstrated credentials.
Nevertheless, Moreira and Perez’s exploration of the social impacts of early standardized testing in the United States is thought-provoking, in that it points to the way hiring (or admissions) decisions are never made ideally, but are instead made under limited information, using “noisy signals” of candidates’ abilities like credentials and recommendations. Deciding which sets of information to include or exclude from such decisions may have powerful social implications. If nothing else, Moreira and Perez’s study should offer some context to the public policy debates around the use of standardized testing for K–12 and higher education admissions, affirmative action, equality, equity, race, and class in America today.
SOURCE: Diana Moreira and Santiago Pérez, Who Benefits from Meritocracy?, NBER (2021).
On this week’s Education Gadfly Show podcast, Ashley Jochim, a principal at the Center on Reinventing Public Education, joins Mike Petrilli to discuss her research on pandemic learning pods, what parents and educators liked about them, and whether they might outlive the Covid emergency. Then, on the Research Minute, Amber Northern reviews a study of how New York City’s charter schools are affecting diversity in traditional public schools.
Recommended content:
Feedback welcome!
Have ideas or feedback on our podcast? Send them to our podcast producer Pedro Enamorado at [email protected].
Cheers
Jeers
