Our team at NewSchools recently released a report titled, Using Expanded Measures of Student Success for School Improvement. In it, we share some on-the-ground lessons from innovative public schools in our portfolio. Some aspects of it caused Checker Finn angst, prompting what he dubbed a “crotchety old guy” blog. As usual, crotchety or not, his observations are smart and useful. In the spirit of dialogue, I wanted to respond to some of his points.

In 2015, we launched an innovative schools portfolio, looking for teams starting new district and charter schools designed to help every student build a strong academic foundation and important mindsets, habits, and skills correlated with success in young adulthood. Back then, we weren’t following the crowd, because there wasn’t one yet. But we thought conditions were ripe for it to develop into a larger trend, which it has in a big way. So much so, we use “fever pitch” to describe the current level of interest among big funders, policymakers, and think-tanks. The term is meant as a caution, not a celebration.

If all the talk and attention doesn’t lead to more clarity about how schools can support both academic and social emotional learning, it won’t matter much. Given our role as a funder of lots of new schools—more than one hundred since 2015—we’re in a position to roll up our sleeves with them to try out various approaches, learn quickly, jettison things that don’t work, and share lessons broadly. So far we’ve published two pieces about our work in this area, available here and here.  

Checker flagged three important issues: how non-academic indicators are measured, whether schools can help improve them, and whether they should be part of accountability systems. Below are some thoughts on each:

1. In terms of measurement, self-report surveys have limitations, and we acknowledge those. And yet, it’s possible to assess whether they are valid instruments for specific SEL and culture measures. We selected items that had been validated in similar contexts, most of which were already in use with hundreds of thousands of students. Every year, we validate them again in our specific implementation context. However, over the long haul, we need stronger tools for this purpose—ideally ones that generate information about how students are developing based on what they do, not solely on what they say. We’ve conducted small pilots of a couple of such tools, but they aren’t mature enough to roll out further. Like Checker and Rick Hess, we want to see more R & D capital brought to bear on the development of more sophisticated approaches to this measurement challenge. In the meantime, we’ll continue to use the best tools we can find and do our best to be transparent about what they tell us and what they don’t.


2. Can educators help students develop social emotional competencies? In other words, are they malleable in a school context? The short answer: Some are, some are not. As we wrote in our recent report, we asked schools which non-academic indicators they cared about most. They collectively named around sixty things—a mix of cognitive skills, habits, values, traits, and SEL competencies. In order to select a smaller number they could follow together, we used a 3M filter, asking if each indicator was meaningful, defined by a correlation with academic and other longer-term outcomes; malleable in a school context; and measurable in a valid and reliable way in a school setting. The seven social-emotional competencies we selected met a threshold of evidence for these three criteria. Even so, we don’t know nearly as much as we would like to about which interventions and practices help foster improvement for which students. Students in our schools take the survey at the beginning of the school year and again near the end, so we can see if there is growth over time. Throughout the year, teachers receive advice and coaching about relevant research-based practices and interventions they might try. One of our goals is to understand more every year about the most effective ways for educators to support student growth on these dimensions.


3. Should these kinds of measures be incorporated into existing accountability systems? Our answer is a resounding no. The lack of consensus about which non-academic indicators deserve attention, the weak knowledge base about effective practice, and the nascent state of measures and instruments make it a bad idea from a technical perspective. But there are other sound reasons to resist calls to use them for accountability, even as we continue learning from them. For instance, while feverish bandwagons can indeed lead to problems, there’s also something to like about the grassroots interest from parents and educators. Inserting these types of indicators into accountability frameworks will almost certainly shift them into a compliance paradigm, potentially squelching the positive energy and organic demand for them. And finally, like many things, it’s likely that the best and most enduring decisions about which mindsets and habits are valued most will be made in local communities, rather than in distant education agencies. As schools and their communities wrestle with this topic, we hope our work (and that of many others) generates practical guidance about valid approaches, practices, and tools that they can use to implement their decisions.

Our venture philanthropy model means we provide significant supports in addition to grants. When we set out on this path in 2015, we decided we couldn’t wait for others to develop perfect solutions our schools could use to develop and measure their students’ non-academic growth. Instead, we’ve selected the best options available, knowing that what we learn might lead us to commit more deeply to some measures and tools and to discard others. As we continue to learn and share, we’re eager for more dialogue and debate on the best way to support an expanded definition of student success.

Educational testing is under attack. In New York City, according to one recent poll, a majority of voters support removing the Specialized High Schools Admissions Test (SHSAT)—the standardized test used for admission to the city’s eight most selective high schools. In higher education, the “test-optional” movement has gained momentum, with prominent schools like the University of Chicago no longer requiring the American College Test (ACT) or Scholastic Aptitude Test (SAT) for admissions. In fact, according to fairtest.org: “More than 1,000 four-year colleges and universities do not use the SAT or ACT to admit substantial numbers of bachelor-degree applications.” And in graduate admissions, the “GRExit” movement continues, with Princeton University announcing that 14 of its graduate programs no longer require the GRE for admission.

Criticism of testing is not new. In 1975, educational psychologist Lee Cronbach observed that, while support for testing would come and go, critics came “largely from those who speak for the poor.” What Cronbach described over 40 years ago is replaying itself today. Testing is being deemphasized or eliminated, though the evidence for its reliability and validity is substantial, especially compared with other selection criteria.

Standardized tests like the SAT have been shown to measure general cognitive ability, or g. By extension, the SHSAT and similar standardized mental or achievement tests likely also measure general mental ability, which has been shown to be predictive of academic performance, career potential, creativity, and job success. Even after accounting for socioeconomic status, standardized tests are predictive of long-term outcomes. They have “predictive validity,” a key statistical measure of the degree to which the test score is associated with some valuable outcome of interest in the future, such as grades or graduation rate.

In undergraduate admissions, the predictive validity of standardized tests ranges from 0.51 to 0.67 (with perfect validity being 1.0) for predicting cumulative GPA and individual course grades. Similar findings are observed in graduate admissions. Results from over 82,000 graduate students revealed validities ranging from 0.34 to 0.41 for cumulative graduate GPA and 0.26 to 0.51 for comprehensive exam performance. Standardized admission tests at both the undergraduate and graduate level are also related to achievement in life, such as high incomes or creation of patents. These effects are statistically significant: a correlation of just 0.30 translates roughly into a 67 percent chance that a student scoring in the top quintile will be successful (versus 33 percent of those in the bottom quintile). A study of more than 22,000 middle school students in New York City found that the SHSAT had a predictive validity of roughly 0.45 for freshmen high school grades. Though still preliminary, these findings suggest that the SHSAT functions similarly to undergraduate and graduate admission tests.

Though critics are often unimpressed by the validity evidence, no method of evaluation currently exists that predicts future success as well as standardized tests. Studies of other admission-related criteria have found substantially lower validities compared with such testing. Letters of recommendation show predictive validities ranging from 0.13 (medical school GPA) to 0.28 (undergraduate GPA). Personal statements fare much worse. Critics point to the effect of social class on test performance as a primary reason for not using the test, but for many talented students from families of low socioeconomic status, the standardized test offers a platform on which to compete with students from wealthier backgrounds.

Yes, being born into a wealthy family brings advantages—but are those advantages most pronounced in proctored, timed tests of logic and reasoning? The evidence on the predictive validity of the tests, even after accounting for socioeconomic status, suggests not. An individual born into a working-class family of five who works extra jobs to help with the bills will be at a much worse disadvantage if the admissions process places more weight on extracurricular involvement or volunteer-research experience. If standardized tests are removed, even more talented and capable students without the means to obtain the credentials of “well-rounded” students—on factors with low to zero predictive validity—may simply be overlooked.

Standardized tests are by no means perfect. Financially disadvantaged students might be strained for finding money and time to take the tests, but more equitable solutions exist than to throw them out. Universal screening, for example, may remove the undue financial burden of the test on disadvantaged students. Despite what appears to be a fundamental disagreement between test critics and proponents, we believe that common ground is evident. The key is to seek fair and equitable admission processes that best evaluate human potential from all backgrounds. Despite their imperfections, standardized admission tests remain an effective way of doing this.

Editor’s note: This essay was first published by City Journal.

Lorain City Schools is no stranger to negative headlines. As one of three districts currently under the control of an academic distress commission (ADC), it has been a hotbed for controversy.

Fortunately for the students in the district, this autumn seems to be bringing a nice change of pace. First, what you’ve probably heard about: September’s state report card release revealed that Lorain earned an overall D (up from an F last year), raised its Gap Closing grade all the way up to a B, and improved its performance index scores.

But here’s what you probably don’t know: Earlier this month, the U.S. Department of Education announced that the district, along with several other industry and education partners, won a federal grant to modernize workforce training.  

To understand the importance of this grant, some background is necessary. Last summer, President Trump signed the Strengthening Career and Technical Education for the 21st Century Act into law. The legislation, referred to as Perkins V, is the long-awaited reauthorization of the federal law that governs how states fund and oversee career and technical education (CTE) programs. Tucked into the pages of the new law are provisions creating the Innovation and Modernization Grant, a competitive program aimed at creating, implementing, replicating, and taking to scale “evidence-based, field-initiated innovations to modernize and improve” CTE, with the goal of improving student outcomes.

The U.S. Department of Education announced in mid-April that $2 million dollars would be available for the program. To be considered for the grant, applicants had to propose a plan that would modernize CTE, develop the effectiveness and alignment of CTE with labor market needs, and improve student outcomes. The law also identified several competitive preference priorities: promoting STEM and computer science education, serving students from low-income families, and serving students in qualified Opportunity Zones.

Earlier this month, the department announced nine winners out of sixty-four nationwide applicants. Among the victors was an application from Lorain County Community College, which partnered with—you guessed it—Lorain City Schools to win a three-year grant worth just under half a million dollars. Lorain City Schools is one of nearly a dozen partners, including:

• Lorain County RITE Council

• NextFlex

• AgriNomix

• Lorain County Community College’s Early College High School

• Lorain County Joint Vocational School

• Elyria City Schools

• Keystone Local Schools

• Midview Schools

• North Ridgeville Schools

There isn’t a ton of information available about the specific project itself, but the information that is available seems promising. The project aims to “engage and support” students in computer science CTE pathways that lead to applied associate’s degrees in computer science and result in employment in in-demand fields. Working together, the partners will work to achieve five goals:

1. Implement dual enrollment pathways in all partnering schools

2. Support students to meet minimum math assessments standards for postsecondary program admittance

3. Expand earn-and-learn opportunities in targeted computer science pathways

4. Expand the STEM Success student support services program to include students in computer science pathways

5. Recruit and credential more teachers in computer science

If accomplished, each of these goals could be a boon for the students in Lorain City Schools and other partnering districts. Supporting students to meet postsecondary admittance standards, for example, ensures that students will avoid expensive remedial courses when they get to college. Earn-and-learn opportunities have become increasingly popular both nationally and in Ohio, and for good reason—they offer students meaningful opportunities to develop academic and career skills. It’s also important that the project will focus on computer science. According to Code.org, a nonprofit dedicated to expanding access to computer science, Ohio has thousands of open computing jobs, and average salaries for computing occupations in Ohio are significantly higher than the average salary in the state.

As with all new projects, the devil will be in the details. Rigorous implementation and a focus on meaningful outcomes for students should be priority. But kudos to the folks in Lorain County for taking the initiative to create a project that could have such a significant impact on the students in several districts—including Lorain City Schools.

Call me a crotchety old guy and you have a point, but I need to vent my angst over a new report from the NewSchools Venture Fund, authored by the very capable Stacey Childress and her colleagues. They’ve landed with many feet on the social and emotional learning (SEL) bandwagon. Indeed, their new “insight brief” proudly declares that “enthusiasm for social emotional learning has reached a fever pitch among policymakers and funders.” That may well be so, but readers should bear in mind that while some fevers do good—the bodily kind can boost one’s immune system, not to mention getting overcome by election fever, a feverish desire to end poverty, etc.—other fevers can kill (typhoid, dengue, etc.) and still others simply drive you nuts (cabin fever).

SEL fever may be all of the above—and it may already be time to apply an ice pack and antibiotics. Months ago, Rick Hess and I wrote a cautionary piece showing how SEL could do much good if deployed in pursuit of academic learning, but also runs multiple risks of going off the rails, much as happened to earlier “whole child” education movements such as “self-esteem” and “values clarification.” We offered seven cautions lest overheated SEL boosters nudge it in the latter direction.

The sixth of our cautions was about metrics. Here’s what we had to say:

Make It a Priority to Develop Valid, Reliable, Intuitive Metrics for SEL—and Be Honest About Their Limits. It will be important to identify and insofar as possible develop bona fide outcome measures that feel credible to a broad swath of parents and educators. This will lend legitimacy to the insistence that SEL be evidence based, while enabling its measures to be viewed alongside measures of academic performance.... School climate surveys are a start, but let’s not kid ourselves. They share the vulnerabilities of all subjective “how do you think things are going” polls and questionnaires, including selective answers by adults who want their school to look good (or bad!) and plain old game-playing by students. SEL needs more reliable instruments. Just how practical it will be to develop them remains an open question, which should be addressed with a transparency and modesty too often absent in recent years in high-profile efforts to promote other sorts of novel measures such as student assessment and teacher evaluation.

It’s also vital to resist overselling the instruments that we do have. Some of what we most value in SEL may ultimately prove difficult to measure systematically or credibly.... Transparency and a willingness to continuously solicit feedback from skeptical students, parents, and teachers—not just supportive ones—will prove invaluable in addressing these tensions. A relentless commitment to evidence will help guard against goofiness while also helping reassure parents and educators. When the evidence is shaky or uncertain, SEL advocates need to forthrightly acknowledge the fact—not duck it or downplay it....

The NewSchools team, like many others in education today, is all about “expanded measures of student success” and how these can be used for school improvement. Others are especially keen about how such expansion can be used for improved school accountability that gives a fuller picture of school performance that’s less dependent on reading and math test scores. There’s considerable wisdom and common sense in the new brief’s advice about “non-academic indicators,” which they’ve been working on for two years in conjunction with TransformEd and some forty district and charter schools. Along the way, they whittled some sixty potential indicators (“skills, traits, mindsets, and values”) down to fourteen, half of which they term “social emotional competencies,” the other seven “culture/climate factors.” As they gather data, they look for (and don’t always find) correlations between these indicators and standard academic measures, and this quest is yielding a few useful initial insights. For example, the brief reports that “growth mindset” and “perceptions of school safety” are the two SEL indicators “most strongly associated with academic performance” while “sense of belonging” isn’t. Good to know, and even better when turned into formative guidance for school people.

I wanted to know a bit more about how these fourteen indicators work, in particular how they generate the data that enable correlations even to be sought. It’s well known that measuring this sort of stuff is difficult and lots of smart people are working in various places to get better at it. When I lifted the veil on the GreatSchools metrics, it was not too surprising to find that they’re based entirely on surveys administered to schoolkids, i.e., on self-reporting by students. (Also popular in SEL—and some ESSA accountability—circles are gauges of “school climate” resulting from surveys that include teachers.).

You can learn a lot from asking kids questions, and lots of survey questions and scoring scales have been tested and validated in various ways. TransformEd has done a lot of that and can cite much research—as did the big Aspen Commission on SEL that prompted Rick and me to write. One can learn a lot about the kids’ preferences, their opinions about themselves and their peers and their schools, and much more. But how reliable are they when used to gauge changes over time in a child or a school, particularly when seeking correlations with other things that are also changing over time? How many of these indicators are actually malleable and to what extent? And—especially—to what extent can they be altered by actions undertaken by schools and educators?

Consider, for example, what kids are asked that is meant to reveal the extent of their “growth mindset,” which we know from Carol Dweck, Angela Duckworth, et al.—and are again being told—correlates with, if it does not actually enhance, academic achievement over time.

They’re given statements such as “There are some things I am not capable of learning” and “Challenging myself won’t make me any smarter.” They’re given a scale of responses from “not at all true” to “completely true.” On questions like those, of course, the less true the statement in kids’ eyes, the stronger their growth mindset. (You can see all fourteen indicators if you turn to Appendix A of a TransformED working paper on the topic.)

Makes sense, sure. But now ask yourself to what extent are these things susceptible to change as a result of things that teachers and schools do or don’t do, or do well or poorly? And to what extent does this sense of self among kids get far more powerfully shaped by home and neighborhood, by the cultures they grow up within (including different ethnic and gender attitudes and stereotypes), by prior experience with anything from riding a bike to baking a cake? And how much are they contingent on context and subject matter?

On the first of those two illustrative statements, for instance, I would have displayed zero growth mindset because I probably would have imagined learning astrophysics, Amharic, or skydiving. This would have happened on the second statement, too, as I would have thought of how challenging myself in one realm—high jumping, say—wouldn’t have made me any “smarter.”

Now think about placing statements like these before both eight-year-olds and fifteen-year-olds. Before A students and D students. Homeless kids and upper-middle -class kids. During math class or P.E. On Monday morning or Friday afternoon. Think about it. Of course you’ll learn stuff from the results, and of course that stuff may help school people understand what they’re dealing with, what challenges they need to try to help kids with, and which kids have which challenges. But just how reliable and valid are these data? How accurately do they trace change over time? What inferences can you legitimately make about what causes changes in them? To what extent are they malleable by schools?

Some things in the SEL realm should certainly be malleable, especially on the “school culture/climate” side of the indicators. School safety and “rigorous expectations” are obvious. It gets more problematic on the “social-emotional competency” side, but over time—assuming kids remain there for multiple years—a first-rate school can probably deepen one’s curiosity, encourage perseverance, and foster self-regulation. Yet it’s really hard to imagine schools and teachers having great influence over those things—and especially over kids’ self-perception of those things—when compared with the other forces that shape and influence children. Yes, I’m harkening back to a tiny bit of “original research”—it took about five minutes—that I did several decades ago when I calculated that the typical American child spends only 9 percent of their time on earth between birth and age 18 under the school roof—and 91 percent elsewhere. (And that assumes full-day kindergarten and perfect attendance for thirteen years.)

Two risks loom. Most troubling—and I’m not saying NewSchools is pushing this—is what happens if indicators like these make their way into accountability systems and are used for judgments that get made about schools and teachers. It’s simply wrong to hold people and institutions to account for things over which they have such limited control—and which rely on indicators that more than a few parents and politicians will find unacceptably nosey and personal. Perhaps the risk is small—Angela Duckworth herself has warned against doing it—but in the quest to widen the accountability net, it shouldn’t be ignored.

Also troubling, and far more likely: What happens when school leaders, teachers, and ed reformers get so preoccupied with the SEL indicators themselves that they neglect English, math, science, and history or forget how much those matter in the real world? Recall the research showing that kids in Asian countries that were trouncing us on achievement measures actually didn’t feel all that positive about themselves and their work. Remember, too, the research showing that many young Americans who were bursting with self-esteem could neither multiply two numbers nor write a cogent paragraph.

To repeat: Resist overselling. Don’t get run over by bandwagons. Avoid catching most fevers.

Just weeks away from what could be a watershed school board election, Denver hosted a community town hall last Friday under the auspices of journalist Roland Martin and The 74. Titled “Is School Choice the Black Choice?” the event was the sixth in a series being held across the country with the goal of engaging local communities on education issues impacting communities of color. The panel discussion, featuring six local advocates, school leaders, and elected officials, was a long and winding one, punctuated by lively moments that underscore the broader tensions and complexities currently facing school choice and education reform writ large.

Racial strains figured prominently into the evening’s dialogue, and Martin kicked things off by saying, “You cannot have a movement that involves black and brown children if you do not have black and brown folks leading it and in charge of it.” One panelist described his work as disrupting the “preschool to prison pipeline” and working with black parents to navigate Denver’s “white supremacist school system.” Setting aside for a moment the verbal arms race that has led to the casual trading of such potent terms, the sentiment behind these words illustrates the brutal racial politics involved in communities like Denver when the student population is majority minority and the decision-makers are mostly white.

The emphasis on race is a far cry from the education reform of yesteryear, when the question of school choice being the black choice would have looked and sounded much different. The “soft bigotry of low expectations” was also about race, though by not exactly saying so, reformers were able to keep more people in the big tent. Back then, the focus on race was less tribal, but now it’s unmistakably aggressive, and the town hall reflected the polarization and balkanization of today’s state of affairs. In some instances, the reductionist politics that has infected our broader discourse reared its ugly head on stage. Nowhere was this more obvious than in the lack of consensus among the panelists of what school choice means for black communities.

This disagreement can be traced to several thorny and overlapping issues. For starters, “school choice” can ostensibly mean whatever one wants it to mean. For Martin, choice is about supporting whatever “works”—defined in part by student performance on state tests—regardless of governance structure, and opposing whatever doesn’t. According to him and some on the panel, choice is a tool for affording greater agency to students and families. For others, choice does exactly the opposite. Given the controversial school closings in the city, some now view choice as an anger accelerant and a constant reminder of indiscriminateness and injustice. Fueled by an even broader skepticism of standardized tests and perceptions of inherent racial bias, they argued that the focus on choice is symptomatic of a broader failure to heed and engage the black community on all issues of schooling. Here in Denver, this has been a protracted dispute with no resolution in sight.

In watching the hour and a half of tortuous back and forth, the frustration revealed on stage seemed less about left versus right, reform versus union, or privatizers versus status quo—though all three were certainly in the mix. Instead, there was a populist bent to the conversation, what political strategist Mo Elleithee has described as “in” versus “out.” Elleithee argues, “The real struggle is between the elites and the streets, people and the powerful. We have been increasingly moving in that direction for the past twenty-five years.” Similarly, the resentment expressed by some on the panel and in the audience that night turns on a feeling that they’ve been victims rather than beneficiaries of Denver’s school reforms. 

The pushback to yesterday’s reform playbook feels in some respect unfair given the dire straits many of Denver’s schools were once in—when taking no action would have been unconscionable—but it’s fair to say that reformers were guilty of too easily overlooking how schools can be a significant community asset regardless of academic performance. It’s worth remembering Andy Smarick’s outstanding piece, “Can Bad Schools Be Good for Neighborhoods?” In it, Smarick observes:

It might be the case that the school—notwithstanding its persistent low academic achievement—acts as an important strand in the invisible web of social connectivity that helps to hold a community together despite all the malign forces trying to pull it apart… those pursuing school-closure strategies should be mindful that every school, even the lowest-performing, is woven into the fabric of its neighborhood—and tugging on that thread affects the entire cloth.

What cities like my Denver hometown have seen—and which was palpable during the town hall—is a subset of those from marginalized communities reacting and responding to what were well-intentioned and overdue efforts to improve low-performing schools. This doesn’t mean reformers in Denver were wrong to close certain schools and replace them with new ones, but the effort to bring the community along clearly fell short. As the pendulum continues to swing, the stage may have been set for a shift away from choice in the Mile High City.

What happens to initially high-achieving students from high-poverty families as they move through elementary school? In the opening of his new book, How the Other Half Learns, Robert Pondiscio worries about these students while teaching fifth grade in the South Bronx. Tiffany is bright, pleasant, and earning good grades. But his assistant principal instructs him that top students like Tiffany will be fine; it’s the low achievers he needs to focus on.

As a result, Pondiscio views any reform from the lens of what he calls “The Tiffany Test”: “will this make it more likely or less likely that kids like Tiffany—promising low-income children of color—will get what they need to reach their full academic and life potential?”

We can begin to answer this question now that a majority of states have been giving assessments aligned to college and career-ready standards. Eleven states, educating 9.9 million students (20 percent of the U.S.) participate in the Smarter Balanced consortium. The Tiffanies from California to Washington, to Connecticut and Vermont are doing worse, and better, than one might hope for.

To see this, we can’t compare this year’s fifth graders to last year’s, as most states do. Instead, we need to follow the same groups of students and see how their achievement changes over time. We can take publicly available data to create groups of students by when they will eventually graduate from high school. Last year’s fifth graders will graduate in 2026, sixth graders in 2025, and so on. While classes can rise and fall within schools, as families move across neighborhoods, the size of the groups remains stable across most states and districts.

Across three classes of students—those who will graduate high school in 2024, 2025, and 2026—in the Smarter Balanced states, the slide is “only” 8 percentage points, as Figure 1 illustrates below. But this is a substantial number. 181,000 fewer students in fifth grade are meeting and exceeding math standards than they were as third graders. In California, Washington, and Connecticut, where subgroup data are available, 86,000 of these students are from economically disadvantaged families.

Figure 1. Slide in mathematics achievement, as determined by meeting or exceeded state standards, classes of 2024, 2025, 2026

It’s not just that a few elementary students like Tiffany who had been successful in third grade have fallen just below the proficiency cut score. In every class of economically disadvantaged students in California, Washington, and Connecticut, more students are scoring at the lowest levels in math as they progress through elementary school. For example, Figure 2 below shows the math achievement of low-income students in the Class of 2025 as they move through elementary school. The gray bars (Levels 1 and 2) are increasing, and the blue bars (Levels 3 and 4) are sliding downward.

Figure 2. Math slide for California’s economically disadvantaged students (class of 2025)

What might explain the slide in math?

First, the “mile wide and inch deep curriculum” problem William Schmidt identified takes time to undo. Too many textbooks in use still haven’t made the shift in focus and coherence expected by the new math standards. Forty-two percent of math teachers the Fordham Institute surveyed in 2016 reported that the math materials available to them are not well aligned with the Common Core. While EdReports.org is helping to raise the quality of learning materials, many of the textbooks in the field still cover less than 50 percent of the major work expected of the grade.

Shelbi Cole of Student Achievement Partners says there is still a substantial mismatch between the questions used in textbooks and the tasks on the assessment. As an example, many of the texts have a large number of word problems in the K–5 Number and Operations in Base Ten (NBT) domain, but this is not the case on the Smarter Balanced test, which emphasizes conceptual understanding and procedural skill aligned to the expectations of the standards. 

Second, the new standards lay out a progression for teaching fractions that’s still taking teachers time to get adjusted to. Before the Common Core, addition and subtraction of fractions were introduced anywhere from third to seventh grade. Now unit fractions are introduced in third grade and developed in a careful sequence that extends partially into sixth grade. Teachers and their students may still struggle with understanding the logic underlying the algorithms of multiplying and dividing fractions.

Math knowledge and skills are cumulative, writes Joel Rose in his recent report, “The Iceberg Problem.” Two-thirds of sixth graders in their urban schools have considerable unfinished learning from earlier grades. They enter math classrooms at least two years behind.

Third, there may still be too much test prep crowding out learning time, says Patrick Callahan, former co-director of the California Mathematics Project and a consultant to several school districts. “For example,” he says, “there are too many weekly or daily practice test items that are never discussed or analyzed. This over-practice can reinforce misconceptions and misunderstandings rather than address and correct them.”

Research by David Blazar and Cynthia Pollard finds that math teachers’ test-prep lessons are of lower quality than when the same teachers aren’t prepping students for the test.

These are only initial hypotheses. States and districts would be well served by gathering expert elementary math teachers to investigate the causes of the slide more thoroughly.

We also need to build cases studies of individual student success and failure. Investigate a group of students who started out meeting standards in third grade and determine why they have slipped as fifth and sixth graders. Assemble a complete picture of their academic and social foundations. Similarly, we need to review a group of students who initially weren’t meeting standards and have grown to reach them. This will help us ensure future Tiffanies get all they need to be academically successful.

Part II, to be published next week, will summarize the surprisingly good news in reading and writing. Stay tuned.

There’s a not-so-secret tension that separates frontline educators from ed reformers, policymakers, and even district office poobahs. This tension, and the cost of top-down initiatives disrupting what’s working on the ground, form the through line of Eric Kalenze’s important new book, What the Academy Taught Us. I’ve made no secret of my admiration of Kalenze and his work as the American face of England’s ResearchED movement. His book shows off his gifts to good effect: He’s one of the few working educators who isn’t reflexively hostile to school improvement and education reform efforts—even when those efforts go sideways.

The academy of his book’s title is “Sophomore Academy,” an initiative circa 2004 at Osseo Senior High School in suburban Minnesota, just north of Minneapolis, where Kalenze taught English. A few years into his tenure, his principal, Dr. Robert Perdaems, approached Kalenze about an issue he’d noticed while studying the school’s data. “Dr. Bob” recognized that kids who dropped out or failed to graduate on time had all ended their sophomore year behind in credits. “In essence, every incoming sophomore class contained a group of invisible dropouts,” he writes. “Kids who were on their way to dropping out but didn’t know it yet.” The book also details how Dr. Bob built other school-wide improvements, turning much of the design and execution work over to a nucleus of teachers—a kitchen-cabinet group called “CSI” (for “Continuous School Improvement”)—which brought greater focus to each year’s improvement plans and played a key role in building staff buy-in and participation.

Kalenze describes how teachers on the CSI team, which he was enlisted to join, worked together, set priorities, and carried them out to create a sense of “individual belonging and shared community” among at-risk students, moving them academically from “stable to able.” In his telling, Kalenze and his colleagues all gained as professionals through this “bottom-up” teacher-led initiative. And it bore fruit. “Within the structures that Dr. Bob created when he commissioned the Sophomore Academy—not an off-the-shelf, comprehensive program—we were given the freedom to make a real difference when it mattered most,” Kalenze writes.

But it’s not a story with a happy ending. The book morphs into a cautionary tale of “how educational leadership is now done across much of the U.S. and how the already challenging job of continuous school improvement has become incredibly constrained and difficult.” A few years into its run, CSI and Sophomore Academy suffered deaths by a thousand cuts—not just budget cuts, but the inability to protect the initiative from the effects of “transformational system change,” detailed in a chapter drily titled, “The District Knew Best.” School improvements that actually work, Kalenze writes, “are never quite sexy or bold enough to attract the same degree of attention” as personalized learning, education technology, or even yoga breaks.

The district’s vision and goals were “decidedly disruptive,” Kalenze writes. Its word-salad strategy was “to create transformational system change to ensure equitable student achievement.” In sum, the district “had clear ideas about what needed transforming, and they ordered these ideas up for everyone,” including standards-based grading practices, a “digital-learning plan” featuring iPads for everyone, and standardized behavior practices “informed by racial-equity focused professional development.” None of these “had strong enough evidence to justify their adoption,” Kalenze writes, but “all of them had large impacts on schools’ operations, required high costs to enable, and kicked up considerable animus from the districts practitioners and parents.” Predictably, none of them did much to improve student achievement.

To his great (and bewildering) credit, Kalenze tells all this without resentment, but the upshot is clear enough. Losing the “bottom-up” improvements midwifed by Dr. Bob “eroded the overall professional culture” at Osseo High School. “Over time, teachers went from feeling that their impact was not valued to suspecting district leadership did not trust them, period.” Kalenze’s purpose is to offer some clarity around continuous school improvement before it becomes the Next Big Thing that everyone talks about but nobody does (or even understands) very well. He concludes with a list of basic improvement principles that he notes would fit onto a single PowerPoint slide, including identifying problems accurately, matching the right people to each improvement initiative, giving those people active roles in those improvements’ design and communication, and respectfully building and maintaining school-wide harmony. Kalenze’s valuable book is intended to rescue “continuous improvement” from cheerleaders and champions in the world of think tanks and Ted Talks. But it is equally useful as a cautionary tale. When something is improving, continuously stay out of the way.

SOURCE: Eric Kalenze, What The Academy Taught Us: Improving schools from the bottom-up in a top-down transformation era (John Catt Educational, 2019).

New and fascinating research uses a creative study design and a unique data set to address whether a thriving local economy leads to better student outcomes. Specifically, it examines how the Texas boom in shale oil and gas drilling, which brought with it large and localized effects on wages and the tax base, impacted district schools.

Researchers Joseph Marchand and Jeremy Weber use the variation in shale geology across school districts to model the potential impact of the boom. Specifically they use data on the shale depth—the average distance in kilometers from the surface to the shale formation—as a measure of shale richness and a proxy for the district’s resource endowment. (Deeper shale tends to have greater pressure, so it is more productive and profitable.) They compare results in the districts located at the shale formations versus those outside the formations.

The sample includes over 1,000 school districts for which shale geologic data were available and focuses on shale oil and not the natural gas formations. Districts are followed for fourteen years, from 2001 to 2014. The district outcomes of interest are school finance, spending, labor market outcomes, composition of the student and teacher body, and student achievement outcomes. Analysts use a couple different models, including a district fixed effects model, but both are essentially looking at how outcomes change over time based on shale depth. They also look only at the “within-shale” sample because “non-shale” districts may be quite different than those with shale.

In a nutshell, they find that the percentage of students passing standardized tests in the average shale oil district declined relative to districts outside of any shale formation, even relative to districts with below-average shale geology. There were no clear differences relative to college completion rates or participation and performance on college entrance exams. Attendance rates also slightly declined as a result of the boom.

These declines occurred despite that fact that, over the entire period, the total tax base of an oil district with average shale depth grew by over a staggering $1 million more per student, relative to non-shale districts. This led school districts to lower property tax rates, borrow more, and spend more.

Analysts found that most additional spending went to capital projects or to service debt, with none going to payroll or teacher salaries, despite the fact that two-thirds of total school spending goes to payroll. Analysts say that the capital spending may stem from a state’s focus on equalizing operational spending—but not facility spending—across districts. Unlike operational spending, districts must fund facilities almost entirely through local property taxes, but with an expanded tax base, they can issue bonds and service them without increasing property taxes. (Plus if and when the boom goes bust, they aren’t on the hook to maintain higher salaries this way.)

The boom also widened the gap between private and education sector wages. Specifically, the average shale district experienced nearly a 20 percent increase in the private sector wage. Student composition also changed, but in a way that moderated the decline in achievement since there was a decline in economically disadvantaged students—most likely as a result of extra family income. Finally, the boom was linked to increased teacher turnover, and it led to more inexperienced teachers in the classroom. The overall negative effect of shale development on student achievement likely stems in part from this turnover and the decline in teacher quality.

So in the end, the boom strengthened the local economy, raising wages and adding new jobs. All good. But it didn’t improve student achievement, perhaps because it lured more experienced teachers to leave the classroom. Moreover, it no doubt resulted in some ecstatic kids and teens who are now enjoying stunning school buildings, decked out science labs, fancy gyms, and professional-grade football fields.

SOURCE: Joseph Marchand and Jeremy Weber, “How Local Economic Conditions Affect School Finances, Teacher Quality, and Student Achievement: Evidence from the Texas Shale Boom,” Journal of Policy Analysis and Management (August 2019).

Note: This is the fifth in a series of blog posts on school funding in Ohio; for the previous posts, see here, here, here, and here.

This week, the Ohio House Finance Committee begins hearings on the high-profile school funding plan crafted by Representatives Robert Cupp, John Patterson, and an array of district superintendents and treasurers. As a refresher, the fully implemented proposal would:

• Add about $1.5 billion per year, a roughly 15 percent increase, in state outlays on K–12 education (though how to pay for it remains unknown).

• Spread new money widely to districts (instead of targeting specific needs or rewarding performance).

• Retain funding guarantees (but, to its credit, removes caps).

• Likely cost even more than advertised because of its call for several additional “cost studies.”  

If these issues weren’t enough to give reform-minded legislators pause, the Cupp-Patterson plan’s icy approach to school choice might tip them over the edge. It should dishearten advocates of school choice, whether public charter, independent STEM, or private schools, who have long sought fairer, more equitable funding. And not only advocates. Ohioans more broadly should be worried about a plan that shows too little concern for the needs of 250,000 students—five times the size of the Columbus school district—who exercise these options. Consider the following three problems.

First, the plan fails to eliminate funding disparities between district and charter schools

Ohio has shortchanged charters since the first of these independently-run public schools opened two decades ago. My recent analysis, released earlier this year, indicates that urban charters receive approximately $4,000 less per pupil in overall funding (state, local, and federal) than districts nearby that serve children of similar demographics. Although both Cupp and Patterson have acknowledged this severe funding gap, their plan does not rectify it. According to the latest estimates released by the Cupp-Patterson workgroup from July, charter schools in the Ohio Big Eight—the cities where most charters are located[1]—are expected to receive an average bump of $758 per pupil by FY 2021 (relative to FY 2019).[2] At first blush, this doesn’t sound like a bad start. But we must remember that Big Eight districts would also see extra state aid. Under the Cupp-Patterson plan, these eight districts on average are projected to receive of $383 per pupil in additional funding by FY 2021. Thus, the charter-district funding gap is estimated to close by a less impressive $375 per pupil by FY 2021, barely denting the $4,000 per student gap that currently exists.

Second, the plan will widen charter funding gaps over the long run

Due to the staggering costs of a new funding model for districts, their increases are phased in over a six-year period, with FYs 2020–21 being the first two years of implementation. Districts, therefore, should continue to see rising funding levels in FYs 2022–25. Charter funding amounts, however, are frozen after FY 2021. While no projections exist beyond FY 2021, this likely means that charter funding gaps will widen as the implementation of district funding continues.

Not only will the phase-in cause further rifts in charter-district funding, but so will the plan’s proposal to periodically adjust district funding for inflation. On the charter school side, the plan maintains “fixed” base amounts that determine the bulk of state aid. In FY 2019, charters received a fixed $6,020 per pupil base amount,[3] which under the Cupp-Patterson plan climbs to $6,179 and $6,338 per pupil in FYs 2020 and 2021, respectively. But then, the plan maintains charters’ FY 2021 base amount at the same level for perpetuity. Unless legislators make regular changes to charters’ base amounts, their base per-pupil funding will remain stuck at FY 2021 amounts, even as their costs rise with inflation.

In stark contrast, the Cupp-Patterson plan provides school districts with regular funding adjustments. One of the major shifts proposed in this plan is to move districts from a fixed base amount[4] to an incredibly elaborate inputs-driven calculation that is supposed to gauge the cost of educating an average student (no effort was undertaken to estimate charter costs). While this new approach would take pages to describe, it’s important to know that these “inputs” are based on variables such as statewide average salaries and benefits. These inputs costs would remain constant throughout FYs 2020–23. But in FY 2024, districts’ input costs would be adjusted for inflation, specifically using the CPI-U measure. Hence, the average teacher salary, for instance, used for the purposes of calculating district teacher costs would rise. In the years thereafter, the state would again make cost adjustments. Every fourth year the input costs would be reassessed based on actual data about salaries, etc., reported two years prior. Because these costs are sure to rise over time, districts would in turn enjoy increased state funding.

In sum, because charters’ per-pupil base amounts remain static after FY 2021 while districts’ are adjusted, the Cupp-Patterson plan will over time widen the gap between charter and district funding.

Third, it does nothing to bridge the enormous gap between voucher and public school funding

Ohio has also significantly underfunded scholarship, or “voucher,” programs that enable needy students to attend private schools. The current amounts for the EdChoice (both the failing-schools and income-based model) and the Cleveland scholarship are just $4,650 for K–8 students and $6,000 for high school, funding levels that fall short of those for even charters, and miles behind district funding. While some private schools can provide a quality education to students on such meager amounts, many others can’t fundraise enough to cover those costs. For families seeking private school alternatives, this results in a more limited range of choices.

Unfortunately, the plan does nothing to improve this woeful funding picture, and in fact worsens it. By freezing current scholarship amounts—while escalating public school spending considerably—the voucher funding gap will only widen.

***

The architects of the Cupp-Patterson plan have dubbed it the “Fair School Funding Plan.” Seen through the eyes of school districts, that may be true. But it doesn’t create a fair model for school choice. It perpetuates the chronic funding gaps between districts and choice programs, both charters and vouchers, and even more worryingly, is sure to widen them over the long run.

A truly “fair” funding plan would show concern for all schools, no matter what banner they fly under. It would also work toward fairness to Ohio parents who, for many various reasons, long for something different for their children. On school choice, then, the Cupp-Patterson plan can only be viewed as a failure. The plan preserves—even further advantages—the status quo, rather than putting Ohio families, and their preferences and choices, at the heart of education funding.

1. The line of demarcation between the board and the teachers of Columbus City Schools was sharply drawn at yesterday’s school board meeting. To wit: the board insisted they were trying to boost all third graders to proficiency by setting a high bar (the highest in the state, it seems) for passage of the beginning-of-the-year MAP test. The number they chose was, they said, based on the “top tier” of student scores last year, all of whom ended up proficient on state tests. Union reps continued to insist the move would create a ton of paperwork for them and that time would be taken away from students who really need reading help. Oh, and that the board was trying to game the state report card by doing so. Probably should have led with that one, shouldn’t I? (Columbus Dispatch, 10/15/19)
2. From the Will Wonder Never Cease department: Dover City Schools pulled its levy off the November ballot this week. Why? Because the triennial update of property values in the county showed an unexpected—and I’m sure very well received by homeowners—increase in property values. The district is estimated to receive an additional $858,160.80 annually, just by doing nothing. Nice. (The Times-Reporter, 10/14/19)

3. Gadfly Bites subscribers with good memories (what was your name again?) will recall that Ohio’s school report cards were recently lambasted by a district supe as being far too complex because they came with a 36-page guide to help readers parse the data included. Well, guess how many pages are included in the guide to Ohio’s social and emotional learning standards? That’s right – 36. It is probably fair to say that school counselors and superintendents see the world a little differently most of the time, but this is very telling. The counselor at Dover High School (what up, Dover?!) suggests that the length of the guide is indicative of thorough, thoughtful, and well-researched SEL standards. And these standards are not even in state statute. “It’s not a law. It’s not a mandate,” she says. “They are recommendations of the appropriate social and emotional skills.” Kinda makes that report card explainer look downright skimpy if you ask me. (The Times-Reporter, 10/13/19)
4. Here is the heartwarming story of an entrepreneurial young man from the Youngstown suburb of Canfield. Prabh Dhaliwal runs a nonprofit that works to assist fellow high schoolers in the area. His aim: to recruit volunteers who have recently taken and done well on the ACT and SAT to tutor other students to boost their scores and their attractiveness to colleges. Adults as tutors just don’t cut it for Dhaliwal, since they haven’t taken the tests in a while. “Our instructors – students – have put in the work, figured out the tests and now want to share their knowledge with others,” he says. In fact, the divergence of commentary from students and adults in this piece is pretty striking. Not to mention the mindset difference between suburb and city. Take a look and see what you think. (Youngstown Business Journal Daily, 10/16/19)

Did you know you can have every edition of Gadfly Bites sent directly to your Inbox (in case you want to cut out the games and sign up for such a newsletter)? Subscribe by clicking here.