Standardized tests and test-based accountability have come under serious criticism in recent years. One of the most important questions is whether improving student learning, as measured by test scores, helps improve students’ opportunities later in life. It’s a tough issue to study, but the weight of the evidence says: yes.
State standardized achievement tests were a central component of the federal government’s outcomes-based policies of the 2000s and 2010s. Policymakers and education leaders used tests to monitor the system’s progress toward performance goals and to inform decisions about accountability, including interventions in low-performing schools.
Since then, standardized tests and test-based accountability have come under serious criticism. And one of the most important questions is whether test scores measure something important that is predictive of later in life outcomes for students. After all, if standardized tests—the cornerstone of test-based accountability—are not measuring skills and knowledge connected to students’ longer term life outcomes, then the incentives created by outcomes-focused policies in the 2000s and 2010s were arguably misaligned with what most people would agree is one of the school system’s ultimate goals: preparing students for success after they leave K–12.
So, do test scores predict later success?
At first glance, the answer here seems clear: Students who do better on tests also tend to do better in college and work. Ample evidence suggests that test scores predict a range of student outcomes after high school. James J. Heckman, Jora Stixrud, and Sergio Urzua, for example, find that they are significantly correlated with educational attainment and labor market outcomes (e.g., employment, choice of occupation, work experience) and negatively correlated with risky behaviors (e.g., teenage pregnancy, smoking, illegal activity). More recently, Dajun Lin, Randall Lutter, and Christopher J. Ruhm find that cognitive skills at the end of high school are associated with rising labor market returns as people age. But, as the well-worn adage says, correlation is not causation.
The relationship between test scores and life outcomes will also reflect other, unmeasured student characteristics (e.g., diligence) and conditions (e.g., family characteristics, the systemic effects of racism and social inequality, adverse environmental exposures [such as lead]). The bottom line is that there is little doubt that a range of individual, social, and environmental factors affect both students’ opportunities and success in school and also later in life. The deeper question then is whether learning outcomes measured by tests tell us anything else? If a school improves student learning as measured by test scores, should we conclude that it is helping improve its students’ opportunities later in life? This is a key question for test-based accountability. And convincingly answering it is not easy. The underlying issue is that non-school factors that are unobserved to researchers, such as the degree to which students receive encouragement in the home, may influence both test scores and lead to better adult outcomes.
Researchers have generally relied on quasi-experimental methods to investigate the causal effects of interventions on learning (which is not observed) as measured by their effects on test scores (which are observed). The question is whether test scores capture the longer-run effects of changes in learning independent of other factors that also affect longer-run outcomes. Researchers have tried to tackle this problem by examining the causal effects of other educational inputs on both test scores and later-life outcomes. The reasoning for this approach is that if we find that these inputs have effects in the same direction on both tests and later life outcomes, then there is plausible evidence that test scores are measuring something that affects later outcomes.
Unsurprisingly, not all these types of studies reach the same conclusion. Some studies of the effects of school choice, for example, find that effects on test scores and later outcomes do not point in the same direction. However, a larger body of studies on teachers, peers, small class sizes, finance reform, and some school choice programs support the idea that there is a causal link between what test scores measure and life outcomes. Will Dobbie and Roland Fryer, Jr., for example, exploit oversubscribed charter schools to compare lottery winners and losers and find that attending a high-performing charter school increases test scores and college attendance and decreases the likelihood of risky behavior. Elsewhere, researchers have used the plausibly random timing of school finance reforms (often driven by lawsuits) to assess the effects of funding on short- and long-run student outcomes. Using this approach, Kirabo Jackson and colleagues, as well as Julien Lafortune, Jesse Rothstein, and Diane Whitmore Schanzenbach, find that changes in school spending affect test scores and adult outcomes. Consistent with these quasi-experimental results, Susan Dynarski, Joshua Hyman, and Diane Whitmore Schanzenbach leverage data from an actual experiment (the famous Tennessee Student-Teacher Achievement Ratio [STAR] class size project) and find that the experiment’s test score effects were an excellent predictor of postsecondary outcomes.
Widespread critiques of testing notwithstanding, these studies suggest that the measured outcomes on standardized tests do indeed capture aspects of student learning that matter for both assessing student progress in school and predicting student success after schooling ends. Research aside, this claim has some face validity: To the extent that your experiences later in life depend on literacy, numeracy, or some other specialized knowledge (e.g., chemistry), measures of your knowledge and skills in school should tell us something—again, not everything—about the opportunities you may have later in life.
Editor’s note: This is a modified excerpt from a report by the authors: Dan Goldhaber and Michael DeArmond, “What Does Empirical Research Say About Federal Policy From NCLB to ESSA?” CALDER Center American Institutes for Research (2023), in Looking Back to Look Forward: Quantitative and Qualitative Reviews of the Past 20 Years of K-12 Education Assessment and Accountability Policy, U.S. Chamber of Commerce Foundation (2023). For a more in depth discussion of the predictive power of tests and whether the relationship between tests and postsecondary outcomes is causal, see Dan Goldhaber and Umut Özek, How Much Should We Rely on Student Test Achievement as a Measure of Success? Education Researcher, 48(7): 479–483, https://doi.org/10.3102/0013189X19874061.
While national school-choice advocates crow about recent legislative victories in states like Iowa, Utah, South Carolina, and West Virginia, setbacks and struggles simmer in Illinois, Montana, and Idaho. The promise of school choice for families and children continues to face threats from adult interests and the stale argument that all we need to improve learning is more money for traditional public schools.
Illinois’s Invest in Kids Tax Credit Scholarship Program—which grants tax credits to taxpayers who fund scholarships that allow 9,700 low-income students to attend private schools—is set to sunset at the end of the calendar year unless lawmakers approve an extension. This is unlikely to happen in the current legislative session. Program opponents claim that “the state can’t afford a private school choice program because public schools are underfunded by billions of dollars.” Illinois currently spends $15,129 per public school student, while fewer than one in five Chicago third-graders are proficient in reading and math. If legislation isn’t passed in the next few days in Springfield, then the funding for Invest in Kids goes away permanently.
Montana Governor Greg Gianforte signed HB562 in May, which created a workable charter school law that would allow “community choice schools” to operate independently from local school districts, including important exemptions from certification and other bureaucratic requirements. In June, the Montana Quality Education Coalition and the League of Women Voters of Montana filed a lawsuit challenging the law. Opponents claim that the new law “threatens to divert critical taxpayer dollars away from public schools—part of what the Montana Quality Education Coalition and other state associations characterized as a broader agenda to privatize education spearheaded by out-of-state interests.”
Montana passed its law thirty-two years after Minnesota passed the nation’s first charter school law, yet opponents’ arguments harken back to those first made against public charter schools when George H. W. Bush served as President and Bryan Adams’s “(Everything I Do) I Do It for You” topped the music charts. It’s still about the money and adult interests.
Even in politically-bright-red Idaho, where I live and where the legislature and governor are unabashed charter school supporters, parents find their choice of school at risk. In rural Fruitland on the Oregon border, the Treasure Valley Classical Academy has been serving 588 students in grades K–10. This charter outperforms the local district on state assessments, mostly by double digits. Its year-over-year student retention rate averages over 95 percent, and it has a burgeoning waitlist.
Yet it’s stuck in a facility situation that prevents it from growing. In fact, if a second facility can’t be built, the school will have to reduce its enrollment to less than 400. School leaders, knowing this challenge was coming their way, have worked with the city of Fruitland for three years to identify an appropriate property for an upper-school campus, while also assembling a responsible financing package that would allow the school to serve 702 students as a K–12 program. Everything needed to finance and build a second building is in place, save for final approval from the city.
On June 13, the Fruitland Planning and Zoning Commission rejected the school’s Conditional Use Permit (CUP). Idaho Ed News reported that, “In its denial, the planning commission cited traffic issues, written and public testimony received, project costs, and impact to the area, according to minutes from the meeting. One nearby business worried about losing its liquor license because of its 300-foot proximity to a school.” TVCA has agreed to mitigate its fair share of impacts.
This decision, if upheld by the city council later this summer, would prevent hundreds of students from completing their education at the Hillsdale Barney Charter School Initiative American Classical Academy, and push back many into the under-performing district schools their families fled.
Nationwide, we’ve seen many school-choice gains, yet far too many American families and children still struggle to find and keep school opportunities that work best for them. And like democracy itself, keeping and protecting freedom and the right of parents to choose where their children are educated is a constant struggle that requires vigilance. My own childhood hero, Ronald Reagan, warned in 1964: “Whether we believe in our capacity for self-government or whether we abandon the American revolution and confess that a little intellectual elite in a far-distant capitol can plan our lives for us better than we can plan them ourselves.”
Parents should have the final say in where their children go to school, not self-interested adults or tame bureaucrats.
This month, New York City students received their offers to the city’s eight specialized high schools. As has been the case in recent years, Asian students form over half of the admittees, followed by White, Hispanic, and Black students. Critics often cite Stuyvesant High School, the most selective of the eight schools, as the example of how admissions—determined by scores on the Specialized High School Admissions Test, or SHSAT—are not racially proportionate. This year, Stuyvesant offered seats to 762 students: 489 went to Asian students (64 percent), 158 to White students (21 percent), 51 seats to students of unknown ancestry (7 percent), 36 to multiracial students (5 percent), 20 to Hispanic students (3 percent), and 7 to Black students (1 percent). Stuyvesant admitted a higher percentage of Asian students and a lower percentage of Black students than the average percentages for all the other specialized high schools.
The argument that the schools are problematic because they are not demographically proportional to the city’s population rests on two related claims: that the SHSAT isn’t a good measure of student performance, and that the test must be wrong if it doesn’t yield proportional outcomes.
The SHSAT clearly works, however. Its outcomes provide the proof: The schools populated by students selected by the test are the best STEM schools run by the New York City Department of Education. Try as it might, the DOE has created no admissions systems that result in better STEM schools. For more than 100 years, a single objective academic test has worked, generating talented alumni who have made tremendous contributions to the world. The specialized high schools have graduated fourteen eventual Nobel Laureates in STEM—more than what most countries can claim. And because of the recent expansion of the Discovery Program backdoor, under which disadvantaged students who miss the SHSAT cutoff score gain admission via teacher recommendation and completion of a summer catch-up program, we now have evidence that students admitted by the SHSAT perform significantly better than students admitted otherwise.
By design, tests should not aim for proportional outcomes. If proportionality were the goal, then everything now measured by tests or contests—from blood tests to pilot evaluations to NBA tryouts to musical auditions—should be done instead by simple lottery. Tests are intended to provide information, not to produce proportional demographic outcomes. And this is exactly what the SHSAT does.
Once you have measured performance, then you can begin to consider why the outcomes are whatever they happen to be. If nonproportional outcomes bother you, then you have three questions to consider: Is the test administered in the same way to all takers, what can be done to improve learning for all, and does proportionality matter?
As to the first question: The SHSAT is identity-blind, meaning that it is administered to all students in the same way. As to the second: How do we improve learning for all? In aggregate, students who perform better have usually spent more time and effort on academic pursuits than those who perform worse. That should be self-evident: It’s not just the time spent studying, but the interest, self-discipline, commitment, and determination that gets the students to spend such time. Indeed, the Brookings Institute conducted a study that found that Asian students on average spend twice as much time on homework as White students, and four times as much time as Black students. There is a simple lesson from this.
What we need to do to improve learning for all groups has been clear for decades. The city’s public schools have pursued a race downward, however, with declining test scores, rampant grade inflation, and a dismantling of basic education in favor of ideological indoctrination. Time is wasted not only on pitting our so-called oppressor and oppressed students against one another but also on a kind of learned helplessness and loss of agency. The return of true gifted and talented programs in every district, with strong academics all around, would enable children to perform better on the SHSAT. The city council, state assembly, and state senate were all offered good legislation that would mandate these programs, with multiple entry points and different admissions criteria that would help nurture students, but the city instead put in place a flawed, in-name-only gifted and talented program. There has also been discussion of opening more top-notch academic high schools that admit by policies other than a single test, especially for non-STEM; these should be piloted.
Does proportionality matter? Progressive critics insist that it does, and they demand proportionality by race, ethnicity, and sex (but only in areas where proportionality suits them). Race, ethnicity, and sex, however, are attributes, not achievements. The SHSAT is deliberately structured to disregard attributes, to measure only potential for achievements, and as we’ve seen, it does so exceptionally well. The test rewards merit. Proportionality, by contrast, rewards not merit but the checking of a box.
The SHSAT works, as do New York City’s outstanding specialized high schools. Let’s congratulate the kids who got in, teach more kids to replicate their hard work and ethics, and grow better schools throughout New York City.
Editor’s note: This was first published by City Journal.
It should be common knowledge by now that all charter schools—like all district schools and, for that matter, all private schools—are not created equal. Nor do they produce equal outcomes. Specifically, we’ve seen time and time again that treating the charter sector as a monolith can obscure the strong results of brick-and-mortar charter schools. That’s because virtual charters, for the most part, don’t do nearly as well. And that’s part of the reason why serious scholars who study outcomes in charter schools get aggravated when other presumably serious scholars lump together in their analyses brick-and-mortar and virtual schools. Efforts to study them separately should be the norm. Thankfully, Temple University’s Sarah Cordes knows that and proceeds to examine the short- and long-term outcomes of Pennsylvania charter schools—including student achievement, attendance, high school graduation, and college enrollment and persistence—with a specific eye toward the differences between brick-and-mortar and virtual schools.
Roughly 8 percent of Pennsylvania students are enrolled in charter schools, and about one-third of those are enrolled in grades 9–12. About half of charter high school students are in urban districts, with about 32 percent in suburban districts and 15 percent in rural districts. Pennsylvania is also home to one of the largest virtual charter sectors in the country: As of 2020, these schools enrolled 175,000 kids, or 2.1 percent of the total public school population, i.e., roughly a quarter of all charter pupils. Note that, in Pennsylvania, virtual charters are authorized by the state, while brick-and-mortar charters are authorized by school districts.
The sample for Cordes’s study includes all members of the ninth-grade cohorts of 2012 or 2013, as long as those students had been enrolled in Pennsylvania public schools in eighth grade and have sufficient baseline data. They must also be continuously enrolled in the fall of every year from ninth through twelfth grade and be able to be matched along a number of dimensions with at least one traditional public school (TPS) student, as well as one charter student. These “matched cell fixed effects” essentially mean that charter students are compared to district students from the same school in the same cohort, and share the same race, gender, and baseline test scores. Cordes is evaluating the intent-to-treat effect of charter school enrollment since treatment status is based solely on enrollment decisions made in the fall of ninth grade, with those students who enroll in a charter school after grade nine assigned to the comparison group.
Brick-and-mortar charter students in her sample are much more likely to be Black, economically disadvantaged, and eligible for special-education services than their district peers, as well as lower performing and more often chronically absent. Virtual charter students are more likely to have repeated a grade, but share similar characteristics with charter students in physical locations.
Brick-and-mortar charters show positive or null effects across all outcomes. For instance, their students see an increase in algebra scores of 0.04 standard deviations (with smaller increases in English language arts and biology) and an increase in attendance by 1.2 percentage points. Brick-and-mortar charters also tend to shift postsecondary enrollment from two- to four-year institutions, specifically making it more likely that students enroll in four-year institutions by roughly 6 percentage points over their district peers. But they have no effects on high school graduation or initial postsecondary enrollment. Positive effects are largely concentrated among Black and economically disadvantaged students and those living in urban districts—results which mimic prior studies. The effects for White students are null or just slightly positive or slightly negative, depending on the outcome.
Virtual charters, on the other hand, are associated with mostly worse outcomes. Compared to students who enrolled in district schools at the start of ninth grade, virtual school students are 9.5 percentage points less likely to graduate, nearly 17 percentage points less likely to enroll in college, and 15 percentage points less likely to persist in a postsecondary institution beyond one semester. The only measure that is positive is attendance: Virtual charter schooling is linked to a 1.8 percentage point increase in attendance and a 4.5 percentage point decrease in chronic absenteeism. On the one hand, these attendance results may reflect the preference of families opting for virtual charters because they need more flexible school schedules due to work or family obligations. But they could also reflect less rigorous definitions of attendance at virtual charters, which might mean simply logging on at any point in the day for any amount of time, or stipulated participation or contact with teachers, without thought to the extent or quality of involvement.
Cordes’s findings replicate previous research on virtual charters—both the good and the bad—and reinforce what should now be accepted wisdom that these schools don’t work equally well for all students. But they do work for some, and they remain popular options in states across the country. So it’s vital that we keep working to improve virtual schools themselves, as well as the policies and infrastructure on which they rely. Meanwhile, however, serious evaluations of charter school effectiveness should always distinguish between brick-and-mortar and virtual environments.
SOURCE: Sarah A. Cordes, “Cyber versus Brick and Mortar: Achievement, Attainment, and Postsecondary Outcomes in Pennsylvania Charter High Schools,” Education Finance and Policy (February 2023).
Since the 1980s, education reform efforts have sought to shake up the stodgy, traditional landscape of public schooling in the United States. One way to do that is to start schools from scratch that can introduce innovative new education models and push traditional systems to improve. A new report from the Carnegie Corporation of New York takes a closer look at new public schools that have opened in the last three decades to identify whether that aim was achieved and define any patterns of change.
The authors select 1990 to 2020 as their timeframe, an appropriate choice given that the first charter school law was passed in Minnesota in 1991 and that the charter movement has provided the most obvious source of competitive pressure on the district establishment. Their analysis considers only district and charter public schools, both old (opened prior to 1990) and new (opened in 1990 or after). Every school they include is encompassed within one of those two categories. Certain school models, such as those in correctional facilities, are expressly excluded in the methodology described. However, while the methodology makes no mention of virtual schools either way, an interactive map of schools included in the analysis lists no virtual schools in any stat. Thus, it seems safe to assume the analysis considers only brick-and-mortar schools.
Of the 93,502 public elementary and secondary schools operating across the country in the 2019–20 school year, the authors find that 31,912 (34 percent) did not exist thirty years earlier. Of those, 7,079—more than one in five—were charter schools. All together, these new public schools are attended by nearly 17 million students, or one-third of all public school students in the country.
As to patterns, the analysts identify three waves of new school creation. The first, in the 1990s, reflects overall increases in student enrollment occurring at the time. Just 4 percent of those created in this wave were charters, as the movement was just getting started. That means most of these “first wave” schools were district schools created in high-growth states and suburbs. The second wave, in the early 2000s, shows new schools being established at a higher rate than school enrollment trends and a boost in the percentage of charters (24 percent) among them. The third, in the 2010s, shows a declining number of new schools being created, even though student enrollment was still increasing, albeit modestly. Charter schools made up 40 percent of schools created in this wave. Nevada, Arizona, and Florida led the nation in new school formation, while fifteen states (including Ohio) saw an overall decrease in total schools by 2020, despite schools continuing to be created. New schools of both types are more likely to enroll Black and Hispanic students than their older counterparts, while White students are more likely to attend older schools. These trends have accelerated in the last decade and are attributed to charter school proliferation in more and more states, which tracks with other data.
The authors surmise that all or nearly all of the new charter school creation over the period was a result of “systems change”—deliberate efforts to build something new and/or operationalize education reform principles—due to the intentionality required and the funding and regulatory hurdles that must be overcome to launch a charter. By contrast, they believe that at least one-third of new district schools created during this period were a response to enrollment pressures—either growth that required additional buildings or geographic shifts that required closing an old building in a declining location and building a new facility in a burgeoning one. And while an additional third of new district schools do appear to have been created for some reason other than enrollment concerns, the authors won’t go so far as to label that other reason as systems change. Seems wise, since systems change and districts are not typically synonymous terms.
New York City, where Carnegie is based, saw the largest number of new schools created (1,050) compared to all other districts/county systems. By 2020, 57 percent of all public schools in the city were new, serving 45 percent of the student body. The report includes a case study of the Big Apple, finding that a majority of public high school students there attended new schools in 2020 while a majority of elementary students attended schools opened prior to 1990. The average new school enrolled 452 students, compared with 744 students enrolled in older schools. For high schools, the difference was even more pronounced: 495 high schoolers attended the average new school versus 1,416 at older New York high schools. The report aligns this finding with former Mayor Michael Bloomberg’s effort to prioritize small schools. The peak of new school creation in the city occurred during the Bloomberg administration (2002–2013) and the majority in that period were new district schools. However, between 2016 and 2020, 71 percent of new schools created in the city were charter schools.
What does all this mean? Carnegie folks are very bullish on the prospect that most of these new schools reflect efforts at innovation and improved opportunities for students, as per research findings on NYC’s small school project. But traditional schools are traditional for a reason, and it is highly likely that, building-wise, districts have created more new facilities to house the same old programs than to experiment with new teaching and learning models. Luckily, we know from research that competitive pressure almost always improves district school outcomes. So to the extent that charters are growing and competition from voucher and ESA programs is increasing—and as the seismic changes from pandemic-era innovations settle into the mainstream—perhaps that bullishness is warranted.
SOURCE: Carnegie Insights, “New Schools in the United States: A Quantitative Review of Public Schools Opened During the Last Three Decades,” Carnegie Corporation of New York (May 2023).
On this week’s Education Gadfly Show podcast, Kevin Teasley, of the Greater Educational Opportunities Foundation, joins Mike Petrilli to discuss his charter high schools where students take dual-enrollment courses at local colleges and earn associate degrees. Then, on the Research Minute, Amber discusses a study that investigates how states’ early literacy policies affect reading and math achievement.
Recommended content:
Feedback Welcome:
Have ideas for improving our podcast? Send them to Jeanette Luna at [email protected].
Cheers
Jeers
