1. In case you missed it this week, the long-threatened extension of lowball, non-academic graduation requirements moved forward in the General Assembly. Ohio’s War on Knowing Stuff will continue for at least another two graduating classes. You can check out coverage in the following articles, all of which include fiery quotes from our own Chad Aldis. You know – the person who’s been publicly pointing out the folly of this effort for the last two years. Gongwer was first out of the gate, of course. (Gongwer Ohio, 12/5/18) Coverage of the grad requirements issue in the Dispatch also mentions some shenanigan-like business regarding the EdChoice Scholarship. Hadn’t even heard about that one in all the other hubbub, but it sounds like it turned out OK. (Columbus Dispatch, 12/5/18) Chad provides today’s title quote for the Bites in this coverage from public radio in Columbus. (WOSU-FM, Columbus, 12/6/18) And DDN coverage is here. (Dayton Daily News, 12/6/18) Never too late to make an important point, our own Aaron Churchill published a letter to the editor in the Blade just this morning in which he explains—for the umpteenth time in this long, sad process—that the putative graduation rate “apocalypse” being touted by the War on Knowing Stuff’s foremost champions is wrong, wrong, wrong. And provides receipts. (Toledo Blade, 12/7/18).

2. With his usual thoroughness, DDN’s Jeremy Kelley followed up when the bill containing the lowered grad requirements passed out of the legislature and headed to the governor’s desk. There was only one vote against the bill at any point along the way. Those aforementioned adults, no doubt, did rejoice as predicted. (Dayton Daily News, 12/6/18)

3. Here is a little more detail on those putative changes to the EdChoice Scholarship program which almost happened in the Senate Education Committee this week. Sounds like the opposite of the “serious, in-depth discussion of the EdChoice program” which the Chair would like to have. (Gongwer Ohio, 12/5/18)

4. An interesting take here on the recent report released by the House Speaker’s Task Force on Education and Poverty. In it, some general recommendations are provided to help address the obvious (thanks to test scores and other data of course) academic achievement gap between students living in poverty and their higher-income peers. From those recommendations, school funding gadfly Howard Fleeter—an employee of the Ohio Education Policy Institute and not part of the task force (I don’t think)—extrapolates a need for immediate replication of Akron’s I Promise School everywhere that the poverty-based achievement gap exists. There really is very little evidence for this leap from what I can see, but I was definitely struck by Fleeter’s rhetorical coup de grâce on the topic: If that school isn’t successful, he says, then it’s really time to worry. Is there any way we can we hold you and the State of Ohio to this, sir? Please? (Columbus Dispatch, 12/5/18)

5. Why do I humbly ask this of Mr. Fleeter and our leaders? Call me a pessimist if you must, but I have no confidence that the entrenched education establishment in our state (and you know who I’m talking about) has much will, let alone ability even with ironclad data and a crap ton of additional resources, to close any sort of achievement gap of the kind the House Speaker is pointing out. I am bolstered in this personal opinion this week by an unlikely source: Columbus City Schools board member Mary Jo Hudson. Hudson, former cabinet director in the Strickland administration and former member of Columbus City Council (you know what I mean), submitted her letter of resignation from the board this week and offered a scathing indictment of…well, everything the education establishment is currently doing. “Our completely inadequate model of governance is broken,” she wrote. “Until it is fixed, our schools will continue to fail, students will be housed in sub-standard facilities, our board will be unresponsive and our families will be kept in poverty rather than given pathways out.” Harsh but true, and an insight that the establishment should not ignore. You can see coverage of the resignation here… (Columbus Dispatch, 12/6/18) …but I would encourage the curious and the justly horrified to read the letter in full first, without her colleagues’ commentary around it. (Columbus Underground, 12/6/18). It is to be hoped, of course, that once Hudson is clear of her commitment to the Columbus school board, she will provide receipts for her charges—examples of all of the ills of which she speaks in her letter. Until that time, I give you the ongoing saga of Trotwood-Madison Local Schools as Exhibit A (or should that be Exhibit ZZZ900?) for “brokenness” and “inadequacy”. As my long-standing (and long-suffering) Gadfly Bites subscribers will recall, Trotwood avoided a declaration of Academic Distress this year by the narrowest of D-minus-minus margins. And now that the clock is effectively reset, the pressure to return to status quo is beginning. To wit, the contract extension for Interim Superintendent Tyrone Olverson was strongly opposed by the Trotwood teachers union for reasons that seem to be at odds with his efforts to turn the district around. The board did not listen, extending the interim contract for three years (!) this week, but I’m sure the opposition won’t end. (Dayton Daily News, 12/6/18)

Did you know you can have every edition of Gadfly Bites sent directly to your Inbox (in case you want to join the rejoicing and sign up for such a newsletter)? Subscribe by clicking here.

The use of standardized tests as a measure of student success and progress in school goes back decades. This practice was formalized by the 2001 passage of the No Child Left Behind Act (NCLB), which established the broader use of test scores as a measure of school quality nationwide. The 2009 Race to the Top federal grant program promoted teacher evaluation reforms that also included the use of standardized tests as a component of a teacher’s evaluation.

But there has been pushback against the use of tests. Some academics and advocates, prominently including the teachers’ unions, have raised various concerns about the consequences of reliance (or overreliance) on test scores for school and teacher accountability purposes. And while there is certainly academic and policy disagreement about the efficacy of using test scores for accountability purposes, there is no doubt that policymakers are scaling back the mandated use of tests. The 2015 passage of the Every Student Succeeds Act (ESSA), for instance, continues NCLB’s requirement that students be tested annually from third to eighth grade, but eliminates much of the federal role in enforcing test-based accountability.

More recently, however, policy scholars have begun to question whether test scores are a metric that we should really care about, pointing out that test score gains are not always associated with changes in other schooling outcomes. But this critique of tests as a measure has gone beyond a narrow academic policy debate. As recent headlines in Forbes show—for instance, “How Much Do Rising Test Scores Tell Us About a School?” and “Is the Big Standardized Test a Big Standardized Flop?”—this debate about tests as a measure of success is now reaching a much broader audience.

A recent report by Collin Hitt, Michael McShane, and Patrick Wolf (herein cited as HMW) is the latest warning from a number of prominent education policy researchers about using test scores, though their focus is the use of tests as a measure in school choice initiatives. The report focuses on a number of studies that examine the effects of different school choice programs on both student test scores and long-term outcomes (such as high school graduation and college enrollment), and examines how well test score impacts in these studies align with the attainment impacts. The authors find very little correlation between the two and conclude that “test scores should be put in context and should not automatically occupy a privileged place over parental demand and satisfaction as short-term measures of school choice success or failure.”

The measured statement that test scores should not automatically occupy a privileged place as a measure of success is at odds with the much stronger claim by Peter Greene that “test scores do not tell you what they claim they tell you. They are less like actionable data and more like really expensive noise.” But we think this is premature, given the evidence connecting student success on tests with the later life outcomes we could probably all agree are important, such as going-going and labor market success

Test scores and later life outcomes?

The idea that tests measure knowledge and that it’s beneficial for students to acquire more knowledge in schools seems pretty uncontroversial. But it also isn’t crazy to argue that test scores are only an intermediate measure of what we really care about: the extent to which students are gaining knowledge in school that enhances their later life prospects. This view raises the question whether this intermediate measure does a good job of capturing what we really care about: the underlying learning that really is important for later life success.

There is a vast amount of literature linking test scores and later life outcomes, such as educational attainment, health, and earnings. Hanushek provides an excellent review of the extant literature on the relationship between cognitive skills, as proxied by test scores and individual incomes in developed and developing countries, and concludes that there is considerable evidence that test scores are directly related to later life outcomes. For example, in the U.S. context, students who score one standard deviation higher on math tests at the end of high school have been shown to earn 12 percent more annually, or $3,600 for each year of working life in 2001. Similarly, Heckman, Stixrud, and Urzua find that test scores are significantly correlated not only with educational attainment and labor market outcomes (employment, work experience, choice of occupation), but also with risky behavior (teenage pregnancy, smoking, participation in illegal activities).

The idea that students who learn more in school—and hence perform better on tests—have better later outcomes because of that learning has a good deal of face validity. It is no great leap to imagine that students who score in the ninetieth percentile on a science test are more likely to be successful scientists than those who score in the tenth percentile, and that this score reflects a better understanding of science. But we might be less sure that smaller differences in student test achievement are meaningful; and as Hanushek notes, these observed correlations do not necessarily reflect causal effects of schools on later life outcomes.

Maybe students who do well on tests are the same students who wake up in the morning, go to work on time, and work hard. Test achievement is also likely to largely reflect learning opportunities outside of school—the supportiveness of families or the communities in which students live. This is why scholars doubt that static measures of test performance alone are reflective of contributions that schools or teachers make toward student learning—a popular critique of those who doubt the use of tests for school accountability purposes.

Both of the above arguments highlight why it is difficult to attribute the observed associations between test scores and later life outcomes to the causal effect of schools. The key question is whether interventions that boost students’ test scores are also likely to lead to better future outcomes for students. Certainly, a lack of an underlying causal link between test scores and long-term outcomes should lead us to consider downplaying (or perhaps eliminating) tests as a measure of student success. Unfortunately, definitively establishing such a causal link is challenging, given that it would be unethical to design an experiment where we randomly provide better education to some students, measure their test scores, and assess whether improvements in test scores lead to better life outcomes. Therefore, what we know about the causality of this relationship comes from a limited number of studies that examine the causal effects of different educational inputs (e.g., schools, teachers, classroom peers) on both student test scores and later life outcomes. If a study finds test score impacts and adult outcome impacts that are not in the same direction, this might be regarded as evidence that test scores do not affect the later life outcomes we care about.

So, what does the literature say about whether there is a causal link? While there are certainly studies that find test-score and long-term-outcome effects that are not in the same direction (as cited in the HMW report), our reading of the broader literature in this context seems to indicate that they are outnumbered by the studies finding evidence of a strong causal link between test scores and later life outcomes. Perhaps the most influential study of all in this strand was conducted by Chetty, Friedman, and Rockoff. Examining the long-term effects of teacher quality (assessed based on their effect on student test scores), the authors find that students who are assigned to highly effective teachers in elementary school are more likely to attend college and earn higher salaries.

Another study by Raj Chetty and co-authors examines the long-term effects of peer quality in kindergarten (once again proxied by test scores) using the Tennessee Student Teacher Achievement Ratio (STAR) experiment, and finds that students who are assigned to classrooms with higher quality peers have higher college attendance rates and adult earnings. Similarly, using the Tennessee STAR experiment, a recent study by Susan Dynarski and colleagues looks at the effects of smaller classes in primary school and find that the test score effects at the time of the experiment are an excellent predictor of long-term improvements in postsecondary outcomes. Lafortune, Rothstein, and Schanzenbach and Jackson, Johnson, and Persico investigate the effects of school finance reform on test scores, educational attainment, and earnings, and find significant benefits of an increase in school spending on both test scores and adult outcomes.

Finally, there are a number of studies in the school choice context (cited in the HMW report) that show certain school choice programs having positive effects on both test scores and later life outcomes. For example, Angrist and colleagues examine the effects of Boston’s charter high schools and conclude that charter effects on college-related outcomes are strongly correlated with gains on earlier tests. Dobbie and Fryer find that attending a high-performing charter school not only increases test scores, but also significantly reduces the likelihood of engaging in risky behavior.

These two studies also highlight an important concern regarding the HMW report, which presents these studies as evidence of misalignment between test scores and long-term effects. While both studies find statistically insignificant, albeit positive, attainment effects (presented as evidence in the HMW report), they both find significant effects on other long-term outcomes that we care about (a shift from two-year to four-year college enrollment in the former study, and a significant effect on risky behavior in the latter).

Overall, all of these studies suggest that interventions that move the needle on test scores also improve later life outcomes. Thus, we contend that the weight of empirical evidence lends support to the argument for using test scores as a measure of success in education systems. This does not mean that the test score effects of educational interventions will always align with their effects on adult outcomes. It is easy to make the case that interventions can and do improve later life outcomes without affecting the cognitive skills of children. In short, test scores will not encompass the full impact of schools and teachers on students, and hence we should not expect them to fully capture all the contributions that schools and teachers make toward influencing long-term student outcomes.

But we need to think carefully about what that might mean for education policy and practice. From a practical perspective, we can’t wait many years to get long-term measures of what schools are contributing to students. This does not mean that test scores ought to be the exclusive or even primary short-term measures, but if one believes in school accountability and that test scores ought to be down-weighted, it is important to consider what alternative measures of success are out there and how reliable they are. For instance, there are concerns that non-test outcomes, such as attendance, grades, suspensions, and high school graduation rates, are arguably more “gameable” than test scores. And we certainly know less empirically about the causal connections between these types of outcomes and long-term student success.

Where one lands on the use of test scores to measure student or schooling success is clearly a matter of subjective judgement and policy debate. But, importantly, that debate should be framed by the right interpretation of the empirical evidence, most of which does suggest that test scores are a good intermediate measure of student success.

Dan Goldhaber is the director of the National Center for Analysis of Longitudinal Data in Education Research, and the director of the Center for Education Data and Research at the University of Washington. Umut Özek is a senior researcher at the American Institutes for Research and an affiliated researcher with the National Center for Analysis of Longitudinal Data in Education Research.

Editor’s note: This essay was adapted from a CALDER policy brief of the same title.

The views expressed herein represent the opinions of the author and not necessarily the Thomas B. Fordham Institute.

Ever since the supposed “graduation apocalypse” was first declared two years ago, we at Fordham have been vocal about the dangers of mischaracterizing Ohio’s graduation rates, passing permanent laws without data, and lowering expectations for students.

Of particular concern were the alternatives proffered by the State Board of Education and passed by the legislature for the Class of 2018. These softball alternatives allowed students to graduate based on things like attendance, capstone projects, and volunteer hours. Since the start of the school year, district officials have been calling for these alternatives to be extended to the classes of 2019 and 2020. This week, the Senate Education Committee buckled under pressure: House Bill 491 includes an amendment that, if passed by the full Senate and House (and ratified by Governor Kasich), would extend softened requirements to this year’s juniors and seniors.

As outspoken opponents of the watered-down requirements, we are disappointed that they may once again be used to award Ohio students a diploma. For twenty-five years—since 1994—Ohio students have had to demonstrate some level of objective academic competence to receive a diploma. Not now, nor for the foreseeable future.

Unfortunately, none of this was necessary. The need to extend these alternatives is a direct result of the failure of adults.

Since January, the state board has repeatedly complained about the inaction of the legislature. That’s ironic, perhaps even hypocritical, considering that the board is legally responsible for setting the number of end-of-course (EOC) exam points required to get a diploma. If they believed that students were struggling because the required number of points was too high, they could have easily lowered that number. Instead, the board did nothing. They abdicated their responsibility and wasted the majority of the year—including the entire fall semester of the class of 2019—complaining about a policy they had the power to change. Now they’ve laid it at the feet of the legislature to lower expectations for students.

But that’s not the only issue that’s led us to this point. A lack of clear data has also contributed. It’s been nearly six months since students from the class of 2018 received their diplomas, but the state still isn’t sure—or isn’t saying—how many students used the alternatives to graduate, or how many students met career-technical or IEP requirements instead. Graduation requirements have been the most hotly debated issue over the past two years, yet this debate has proceeded with incomplete data. This has led to wild speculation about whether the state really is withholding diplomas from too many students.

Misinformation being provided to students seems to be another issue. Despite clear language from legislators that the softened requirements were a temporary solution only for the class of 2018, thousands of students and families seemed to have missed the memo. As a result, students like this one in Springfield were “surprised” to learn that the 2018 alternatives wouldn’t be available. Her stepmother told the media: “She had that same pathway that the kids before her had, and then now, they’re going to pull it away with no safety net at the last minute.” That is, quite simply, untrue. The alternatives were never available to the class of 2019, and they certainly weren’t suddenly taken away. It’s almost as if students and parents were misled.

Finally, these requirements were originally passed when the class of 2019 were just completing seventh grade. How much transition time is necessary? Do districts have early-warning systems that flag when students are off track so that schools can provide additional supports and help them catch up? The whole point of higher expectations was to help prepare kids for a more challenging twenty-first-century workplace. There’s little systemic evidence this has occurred. Moreover, the career option for students who wanted to focus on entering the workplace directly doesn’t seem to have been utilized by many, although the data available is insufficient to know for sure. If students are struggling to demonstrate grade-level academic competence, they should have been assisted to both improve their academic skills and earn an industry credential that could lead to workplace success. Yet there is no evidence that suggests this occurred in meaningful numbers.

All in all, adults have created quite a mess for the classes of 2019 and 2020. Legislators have once again declared that this extension is only temporary. Reporters and editorial boards need to be crystal clear about this in their reporting, and district officials, school leaders, and teachers need to communicate to parents and students that these alternatives will not be available for the class of 2021 and beyond. We’re skeptical that will be the case. Next year’s state report card will show a big jump in the graduation rate and many folks will speak of the numbers with pride. But it’ll be at best a pyrrhic victory, with more students receiving a diploma with little evidence that they actually know more.

I’m in the middle of a series of posts looking at how we might usher in a “Golden Age of Educational Practice” now that big new policy initiatives appear to be on ice. Last week I claimed that all of the possibilities that might work at scale entail various investments in innovation and R&D. Such efforts will only be successful, though, with exponentially better insight into what’s actually happening in the classroom.

That’s because, right now, key decision-makers are flying blind. Consider just a few examples of questions that have been raised in recent weeks that we simply cannot answer:

• Is student achievement flat because teachers are implementing Common Core, and it’s not working? Or is it flat because are teachers mostly ignoring Common Core? Or is it neither of the above? We have no idea.
• Has “balanced literacy” served as a Trojan Horse that allowed whole-language reading instruction to continue unabated in our elementary schools, instead of a scientifically-based approach with a big emphasis on phonics and phonemic awareness? Is this an issue in relatively few schools or lots of schools? We have no idea.
• Are most high schools teaching a Howard Zinn–inspired version of U.S. history, with an overwhelming focus on our country’s injustices, as opposed to its triumphs, too? Or is this just happening in deep-blue bubbles? We have no idea.

And it’s not just policy wonks or education scholars that lack information; leaders at state and local levels have too little insight into classroom practice as well. Whereas the world outside of our schools has been transformed by information technology, the data we collect on classroom practices is somewhere between nonexistent and laughably rudimentary. In other words, we know almost nothing about almost everything that matters.

To be sure, education research improved dramatically starting in the early 2000s with the creation of the Institute of Education Sciences, the federal mandate for annual tests in grades three through eight, and the concurrent development of longitudinal data systems in most states. Scholars suddenly had the money and the data to examine a variety of educational interventions and their impact on student achievement, significantly increasing our understanding of what’s working to boost student outcomes.

Yet the vast majority of such studies rely on state “administrative data”—information that is collected to enable our systems to keep humming along, but that can also be happily recycled as markers of various inputs or programs whose effectiveness might be studied. Lots of this is related to teacher characteristics—their years of experience, race, training, and credentials. Other data captures bits of the student experience—their attendance patterns, course-taking habits, family background—and that of their peers.

This is all well and good but it’s still very limited. We end up studying the shadow of educational practice rather than the real thing. What we don’t see is what’s actually going on in the classroom—the day-to-day work of teachers and their students—the curriculum, the assignments, the marks students receive, the quality of instruction itself. We simply don’t know what kids do all day: the books they read, the tasks they’re asked to perform, the textbooks teachers use—or even whether they’re used at all or sit unopened in the closet, whether programs are implemented with fidelity, haphazardly, or not at all.

Examining practice has always been a difficult and expensive proposition. The most respected approach involves putting lots of trained observers—often graduate students—in the back of classrooms. There, they typically watch closely and code various aspects of teaching and learning, or collect video and spend innumerable hours coding it by hand. This is incredibly labor-intensive and costs gobs of money, so it is relatively rare.

Alternatives to observational studies are much less satisfying. The most common is to survey teachers about their classroom practices or curricula, as is done with the background questionnaires given to teachers as part of the National Assessment of Educational Progress (NAEP). Though useful, these types of surveys have big limitations, as they rely on teachers to be accurate reporters of their own practice—which is tough even with positive intentions. It’s also hard to know whom to survey about some information; for example, Morgan Polikoff, associate professor of K–12 policy at the University of Southern California, has been trying to understand which textbooks schools are using, and is finding that, in many districts, nobody can give him a straight answer.

So that’s the challenge: We lack the systems to collect detailed information about classroom practice that might help us learn what’s working and what’s not, and inform changes in direction at all levels of governance.

Thankfully there are potential solutions. I see three:

1. Take advantage of data already being collected by online learning providers and services, such as Google Classroom, to gain insights into our schools;
2. Systematically collect a sample of student assignments, complete with teacher feedback, to learn more about the “enacted curriculum,” its level of challenge, and its variation; and
3. Use video or audio recording technology in a small sample of schools to better understand instructional practice in America today.

The first possibility is a cousin of using administrative data to power research studies. Online learning platforms like Khan Academy and services like Google Classroom are already collecting reams of data about teaching and learning, but to my knowledge, these data remain largely proprietary and locked away. Surely it would be possible to protect student privacy and any trade secrets while allowing researchers to gain insights into what’s working in our schools.

Google Classroom seems particularly promising, given that, by some accounts, more than two-thirds of districts use it today. Imagine if scholars could view, anonymously, student essays and other assignments. With the help of machine learning, we could figure out how much variation there is in the level of challenge of the assignments and in the grading standards. And we could glimpse if schools with tougher assignments and higher grading standards were getting better results in terms of student learning, after controlling for background factors. We might also be able to tell which curriculum a given teacher or school was using and the degree of alignment between student assignments and grade-level standards.

This approach would be particularly useful for middle and high schools, given that many assignments are now completed online. But what about elementary schools, where paper-and-pencil worksheets still largely rule the roost? That brings us to our second big idea. Imagine if we could identify a nationally representative sample of elementary schools where researchers would collect a sample of student work on a regular basis—worksheets, quizzes, tests, etc. The research initiative would develop an easy-to-use mechanism for digitizing these materials, adding value to the teacher and the school. For example, a scanner could be provided that makes copies of marked-up worksheets and quizzes, automatically enters the grades into teachers’ electronic grade-books, puts an electronic copy in a students’ online portfolio, and sends the image to parents’ emails. Meanwhile the information is sent to the researchers, connected securely to each students’ profile, and anonymized. (Xerox’s XEAMS initiative experimented with some aspects of this; Class Dojo has some of these functionalities, too, with students scanning their work with iPads.)

Just as with Google Classroom, we’d make a quantum leap in our understanding about the curriculum our schools are using, the level of rigor in student assignments, teachers’ grading standards, and much more.

The third big idea, and also the most controversial, is to record what’s happening in a sample of our classrooms. (Audio is less intrusive than video, and gives you just as much information.) Using a smart speaker like Amazon’s Echo or a Google Home Mini, researchers could capture the play-by-play of instructional practice and then train algorithms to make sense of what’s going on. As I explained in my Education Next column earlier this year, this is no longer the stuff of science fiction. Researchers are already doing this to glean insights into the kind of questions teachers are asking—and which approaches work best to drive student engagement and learning.

They start by capturing high-quality audio, and then run the audio files through several speech-recognition programs, producing a transcript. Then their algorithm goes to work, looking at both the transcript and the audio files (which have markers for intonation, tempo, and more) to match codes provided by human observers.

The computer program has gotten quite good at detecting different types of activities—lectures versus group discussion versus seatwork, for example—and is starting to be able to also differentiate between good and bad questions. Humans are still more reliable coders, especially for ambiguous cases. But the computers are getting better and better, and good enough that, with sufficient data, they can already produce some very reliable findings at a fraction of the cost of a people-powered study.

Connect all of this up to state administrative data and student achievement data and we would finally have an accurate picture of what’s actually going on in U.S. schools. (We’d know, for example, the degree to which schools are narrowing the curriculum and squeezing out science and social studies.) And we’d have vastly more information with which to study the effectiveness of various instructional and curricular approaches. 

Big hurdles remain, to be sure. The biggest aren’t technological, but political: Such an effort must earn the trust of teachers and parents. We must be able to promise that none of the data will be used to evaluate or punish teachers; it must also be protected with the highest level of data security. None of that would be easy, but by allowing schools to opt-in, and by starting with a small pilot, such an initiative might earn the trust of key stakeholders over time.

To be clear, this is a different idea than putting a camera or microphone in every classroom—akin to body cameras or dashboard cameras for police. I’ve written about that notion too, and see potential value in it, but it raises Orwellian questions that are a whole order of magnitude higher.

The goal here is to accelerate the R&D process by improving our ability to learn about instruction. For that purpose, a relatively small number of schools or classrooms—in communities that volunteer to participate—would do the trick.

***

To be clear, collecting better information about classroom practice is just one part of the puzzle. We also need to fund rigorous studies to turn the data into insights about what’s working, and then have to figure out how to get evidence-based practices into the schools. I will tackle all of that in future posts. But without good data, any R&D strategy is destined to fail. It’s a critical foundation, and one that is not nearly sturdy enough today.

Last May, Slate ran an eight-part series exploring the rise in online learning for high school students who had failed a course. One of the articles included a screenshot of this tweet with identifying information removed: “If anyone wants to go online and do my chemistry credit recovery, I’d be more than happy to give you my username and password.”

Ouch. Teenage bravado, perhaps, but it illustrates our worst fears about credit recovery.

For the uninitiated, credit recovery is the practice of enabling high-school students to retrieve credits from courses that they either failed or failed to complete. And it’s at the crossroads of two big trends in education.

The first is the desire to move toward “competency based” education. Rather than make all pupils march through a prescribed curriculum on a one-size-fits-all timeline, this approach allows them to move at their own pace, earning a credential by demonstrating what they know and can do, not because they accumulated a fixed number of hours in their classroom seats.

The second trend is the push to dramatically boost graduation rates. That started with a No Child Left Behind regulation under former U.S. Education Secretary Margaret Spellings, mandating that states measure graduation rates in a standardized way, that they strive toward challenging goals for boosting said rates, and that they hold high schools accountable for not attaining them.

It worked, sort of. On-time high school graduation rates rose from 79 percent in 2011 to 84 percent in 2016—yet there’s little evidence that students are learning more as a result. A key question is whether these rising rates indicate real educational progress—or games and chicanery. Clearly, credit recovery has played a part in the increase, as it was explicitly developed to help credit-deficient high school students complete all their course requirements. What we don’t know is whether the credits thus obtained are the fruit of high-quality alternative courses that expect students to demonstrate the same knowledge and skills of the original courses—or are vacuous imitations, a sort of educational fakery.

Anecdotal evidence offers ample cause for concern. The high-profile 2017 investigation into Washington, D.C.,’s Ballou High School is perhaps the most vivid example. Investigators found that most students graduated despite sky-high absenteeism rates and widespread use of credit-recovery courses. But the problem is not restricted to one school in the nation’s capital. Similar concerns about credit recovery have been raised in multiple states, including California, North Carolina, New York, and Virginia.

Under the Every Student Succeeds Act, states must still factor graduation rates into their high school accountability plans. So the federal incentive to game the statistic remains in place, which underscores the need to keep it from turning into a total end run around actual learning.

Unfortunately, reliable data are non-existent when it comes to how states and districts define credit recovery programs, much less when it comes to gauging their quality. That data drought received a welcome sprinkle of rain when, for the first time, the Office for Civil Rights released data in 2018 on whether individual schools offered credit-recovery programs and how many students were enrolled in them.

These new data, though humble in scope, allow us to report credit recovery participation rates at national, state, and local levels—and provide a baseline by which future participation can be compared. Fordham’s Associate Director of Research Adam Tyner and Research Associate Nicholas Munyan-Penney were keen to delve into the new data and are the co-authors of our newest study, Gotta Give 'Em Credit: State and District Variation in Credit Recovery Participation Rates.

They sought to answer three questions: How many high schools have active credit recovery programs, and are some types of schools more likely than others to have them? How many students are enrolled in credit recovery? And to what extent do schools enroll large shares of their students in credit recovery, and is that more common in certain types of schools?

For each school, Tyner and Munyan-Penney merged the OCR program and student participation data from 2015–16 with same-year enrollment and demographic data from the National Center for Education Statistics (NCES). They included traditional district-operated schools as well as charters and magnet schools.

Their analysis yielded five key findings.

1. Most high schools have credit recovery programs, although these are far less common in charter schools.
2. While the presence of credit recovery programs is generally not related to school poverty levels, schools with many minority students are slightly more likely to have active programs.
3. Credit recovery programs are less common in smaller schools.
4. In high schools with active credit recovery programs, an average of 8 percent of students participate. However, nearly one in ten schools enrolls 20 percent or more of its students.
5. Higher enrollment in credit recovery is more common in large and urban schools, as well as in charter schools and schools with higher proportions of poor and minority students.

We derive four takeaways from these findings, the first of which is to adopt the old adage of trust but verify. Credit recovery programs are so varied and uneven that it’s wise to view them with skepticism. Until we have better data than are presently available, we shouldn’t assume that they’re of high quality, absent closer inspection. Frankly it’s suspicious that higher enrollment in CR is more common in urban schools and in schools with higher proportions of poor and minority students, given that these are often the same schools where graduation rates have risen the fastest.

That said, don’t assume that all forms of CR are fraudulent. It’s premature to make sweeping generalizations about the value and efficacy of these programs. Like most educational interventions, we can find places that are experiencing genuine success as a result of the intervention, but we can also find places that manifest dismal failure.

On that note, the second takeaway is target the outliers. Districts and schools (both traditional and charter) with higher rates of students participating in credit recovery warrant scrutiny. The authors name some of those districts in this report. When they have nearly half or more of their students enrolled in credit recovery, it is past time to wave a cautionary flag. Automatic audits should be triggered by such rates because the potential to abuse this kind of programs is high—and highest when there’s no external evidence.

Third, adopt the recommendations put forth by this report’s authors. That includes collecting additional data about credit recovery programs at the federal and state levels, such as whether courses are taken in person or online and the name of the curriculum or software program used; adopting formal state guidelines for credit recovery programs, especially as they pertain to eligibility and vetting of courses; and using external assessments to hold students and schools accountable. We’re particularly enthusiastic about the latter, which would go a long way toward making credit recovery a form of competency-based education; it’s hard to say that about it today.

Fourth, in the absence of an external check on quality, adopt the “ounce of prevention” rule. Stipulate, as some states already do, that students must have previously failed the course to be eligible for credit recovery. Or do as Alabama and Tennessee have done and demand that students achieve a minimum score in the original course so that they aren’t starting from scratch with new material. If they don’t reach that minimum, require them to retake the traditional course to earn credit.

***

Having more than one way to demonstrate mastery of content in high school is laudable, especially because students’ distinctive needs require greater personalization. But these promising methods must be executed with more than good intentions. That means making certain that credit recovery meets our expectations for quality, including using external assessments to ensure comparability and rigor, and cracking down on schools that appear to be over-relying on it. In other words, erect sufficient safeguards to protect students from misuse.

Otherwise, the incentives to propel unprepared adolescents towards graduation will outweigh those that advance the challenging but noble work of sufficiently educating them.

College-level courses taken while in high school have never been more popular. Chief among these sources is Advanced Placement (AP), whose five million exams were taken by almost three million students in 2018. AP’s popularity has skyrocketed for a host of reasons: parents seeking a more rigorous curriculum for their kids; principals wanting to drive their school up on competitive national rankings; and students hoping to polish their college applications, propel themselves through the ivy-covered gates, and—if all goes well—skip some vast, lecture-style intro courses when they get there. It’s also widely believed (including by some 58 percent of U.S. teachers) that AP’s expansion owes something to students wanting to save money or graduate early with the help of AP credits.

Yet little is known about how many students do actually earn college credit from taking AP exams, and how this actually benefits them. A new analysis from Vanderbilt University assistant professor Brent J. Evans takes a timely look at who gains college credit via AP and what positive outcomes are associated with that credit.

Evans examined data from a nationally representative survey, following 14,830 students who started college in 2004 and tracking them for six years. He looked at their demographics and a laundry list of their subsequent outcomes: whether they attended a four-year institution; the selectivity of their college; first-year college GPA; six-year bachelor’s degree attainment; total non-AP credits earned; whether they double-majored; whether they majored in a STEM subject; the number of months and terms needed to complete their degree; if they earned advanced laboratory science or math credits; and whether they enrolled in graduate school. He also looked at certain labor elements, such as whether a student was employed and for how many hours during their first year of college, as well as their cumulative loan debt. And he controlled for prior academic achievement and demographic differences, as well as differences in student sorting across higher education institutions in colleges’ policies for granting credit on the basis of AP.

One surprising finding is that just 7.6 percent of first year college students enter with a least one credit derived from AP—this in contrast with the (today) nearly one-fourth of high school graduates the College Board says have acquired at least one AP score of 3 or better. Those who started with such credits averaged ten credit hours applied to a 120-hour degree program and reduced their cumulative loan debt by about $1,000 each. They also used their AP credits to help reduce time to degree (especially among lower-income students who qualified for Pell Grants), pursue STEM majors, and to take more advanced math courses rather than increasing their leisure time or hours of paid work. Evans also finds strong positive effects from earning AP credit on academic outcomes during college, including increased first-year GPAs and increased likelihood of attaining a bachelor’s degree within six years, with the strongest outcomes associated with a shorter time to degree completion and increased likelihood of double-majoring. The more credits earned, the larger the effects on most outcomes.

The study stands out for the sheer scale of outcomes it examines. Few previous AP studies have come close, especially with a sample of this size. Also noteworthy is that it examined actual credits earned, not potential credits, or predictive credits, as is most common in prior research.

Still, the study has many of the same limitations as other AP research, beginning with selection effects: AP students are more likely to be better students, which entails unobservable student characteristics (such as intrinsic motivation, prior academic ability, etc.) that may affect their college outcomes more than earning credit from AP. Ultimately, the study demonstrates that earning credit from AP can potentially benefit those who do, but such students are likely to be high achievers anyway. The subject demands greater attention and analysis, as much has changed over the past fourteen years in how AP credits are granted; even while more elite universities are limiting credit-granting to scores of 5 (or are eliminating this practice altogether), some twenty-eight states now have mandatory (system-wide or statewide) credit-in-return for scores of 3 or above.

Nevertheless, Evans’s study adds another important layer to our understanding of how students earn credit from AP in college and the potential benefits for kids who manage to do so.

SOURCE: Brent J. Evans, “How College Students Use Advanced Placement Credit,” American Educational Research Journal (November 2018).

Journalists are told to “follow the money,” and it seems only fair the same adage be applied to education. A new report from the School Choice Demonstration Project at the University of Arkansas does just that, comparing the levels and sources of funding between traditional public schools (TPS) and charter schools in some of our nation’s largest metropolitan areas. Faithfully following the money, the authors finding a whopping $5,835 annual advantage for each TPS student.

The researchers—Corey A. DeAngelis, Patrick J. Wolf, Larry D. Maloney, and Jay F. May—collected revenue data for the 2015–16 school year from what they refer to as “a maze” of district, state, and federal websites. Where states did not gather the same depth of data for charters, information came from annual reports, independent audits, and federal 990 forms from individual schools. The authors grade fourteen metropolitan areas by the width of the funding gap between TPS and charters—cities with less than a 5 percent disparity receive an “A,” and those with more than a 25 percent disparity receive an “F.” All the numbers in the report are per pupil, calculated using fall enrollment data.

Notably, the study analyzes yearly revenues, not yearly expenditures. Some charter skeptics say expenditures are a fairer metric because TPS have more fixed, mandatory costs (like transportation) than charters do. The authors reject this outlook, arguing that innovation—including financial innovation—is one of the central purposes of charter schools. “A revenue-based analysis,” they say, “is grounded in a concept of equal funding for equal purpose.”

They find that, on average, public charter schools received $5,828 less than TPS in per-pupil funding—a gap of about 27 percent. Thirteen of the fourteen cities received a C or lower for funding equity; the outlying A grade is for Houston, Texas, where charter students receive 95 percent of the funding as their peers in TPS. The “worst” F grade goes to Atlanta, Georgia, which gives charter schools 49 percent less money per student.

Of course, many charter schools enroll lower percentages of low-income students, English language learners (ELLs), or students with disabilities than do TPS, which could explain some or all of the funding gaps. However, the researchers found that enrollment of these three subgroups in the fourteen regions was not as disproportionately low as some may claim. Even where charters did enroll fewer disadvantaged students, the small percentage differences did not account for larger per-student funding gaps.

Local funding accounts for most of the disparity between TPS and charter revenues. On average, charter schools received 76 percent less funding from their own cities—about $8,000 less per student. In fact, six cities gave no dollars to their local charter schools. Charters also receive lower funding from states and the federal government. They recoup some of their losses through private donations, which go disproportionately to charter schools. However, donations are a relatively small portion of every school’s annual budget, so the per-pupil advantage is only about $655. This clearly does not go far in covering the overall funding gap.

While most of the findings in this study are not very clearly communicated and the authors’ grading cutoffs seem somewhat arbitrary, they rightly conclude that state and local governments can do more to extend equal opportunities to charter school students. Families should not have to sacrifice thousands of dollars each year for the chance to send their children to the school of their choice. One change the authors recommend is that states should base school funding—for TPS and charters alike—more heavily on student need, so that disadvantaged students can receive the support they need no matter where they go each morning.

SOURCE: Corey A. DeAngelis, Patrick J. Wolf, Larry D. Maloney, and Jay F. May, “Charter School Funding: (More) Inequity in the City.” University of Arkansas School Choice Demonstration Project, November 2018.

COLUMBUS (OH) – The Senate Education Committee today amended House Bill 491 to extend previously-relaxed graduation requirements for the class of 2018 to the classes of 2019 and 2020.

“Despite consistent feedback that too many Ohio high school graduates aren’t ready for credit bearing college courses and don’t possess the skills necessary to enter the workforce, the Senate is again rolling back what’s required to receive a high school diploma,” said Chad L. Aldis, vice president for Ohio policy and advocacy at the Thomas B. Fordham Institute. “The point of raising the bar in the first place was to help students be prepared when they leave high school. While adults in the education system will rejoice if this change becomes law, students taking an easier path and left without an industry credential or grade level math and English skills will be left to pay the ultimate price.”

Rather than earning a diploma by successfully passing end-of-course exams, achieving remediation-free scores on the ACT or SAT, or attaining an industry credential and demonstrating workforce skills, students in the class of 2019 would be able to graduate by completing tasks from a list which includes a 93 percent senior year attendance rate, holding down a part time job or a volunteer position for 120 hours, and earning a 2.5 grade point average in their senior year. Students in the class of 2020 would have similar but slightly modified requirements.

“The modern world economy demands more of its workers than ever before. Unfortunately, instead of rolling up our sleeves and helping all Ohio graduates attain the skills necessary to earn a living wage and support a family, we’ve once again said that simply showing up is enough,” Aldis added. “If we believe that all students can learn, we owe it to them to provide the supports necessary to prepare them for the future.”

The amended legislation still needs to receive the approval of the full Senate and House before going to Governor Kasich for his approval.

Thank you, Chair Lehner, Vice Chair Huffman, Ranking Member Sykes, and Senate Education Committee members for the opportunity to provide written testimony on amendments potentially being offered on House Bill 491 related to softening the graduation requirements for future graduating classes.

In 2014, when the legislature adopted the current graduation requirements and raised the expectations for Ohio students to get a diploma, we applauded your resolve and commitment. It was a powerful acknowledgement that too few Ohio students were graduating high school with the skills necessary to be successful in college or to enter the workforce. Fully one third of Ohio students who did enter an Ohio college required remediation before taking credit-bearing courses. And we routinely heard reports of good paying jobs sitting vacant because young people didn’t have the skills that employers needed.

That’s why this body raised graduation requirements. Last year’s graduating class, the Class of 2018, was supposed to be the first required to earn a diploma by demonstrating readiness in one of three ways: college readiness via the SAT or ACT, successful mastery of high school level academic competencies via objective end-of-course exams, or earning an industry credential and demonstrating work readiness skills.

Hoping to give students another year to adjust to the increased expectations, the legislature extended a set of alternative graduation requirements that were anything but rigorous. It was a huge missed opportunity but understandable if it was temporary. Unfortunately, the same issue is back again this year, and it’s clear that for many it was never meant to be a temporary fix.

Now we’re debating whether to further delay implementation to students who were just completing sixth or seventh grade when the original requirements were passed.

This is a problem. For years, we have been vocal in our criticism of the long-term consequences for lowering expectations for students and giving parents a false sense of security by misrepresenting their children’s readiness for post-high school success. You can pass a law giving students a diploma, but being given a diploma isn’t the same thing as earning it. Without bolstering their academic skills, students are less likely to be ready for college. For those without an interest in going to college, putting in the work to acquire an industry certification could open the door to a job earning a living wage. If they can get a diploma without putting in the effort though, the chances are smaller that they graduate with in-demand skills.

I’m sure that none of this is new so far. Well, here’s something you’re unlikely to have heard before: there is absolutely no need for the legislature to take any action. Why?

1. There is an incredible lack of data and transparency around this issue. Despite repeatedly asking the legislature to change the law, there’s starting to be some indications that the situation isn’t as dire as some would lead you to believe. At the October state board meeting, the Ohio Department of Education released limited graduation rate data for the Class of 2018. It shows that 68 percent of students from the Class of 2018 graduated because they earned the required scores on state end-of-course (EOC) or ACT/SAT exams. While it’s yet to be released by ODE, if you add in historical data related to the number of students typically earning a diploma via the career-technical graduation pathway and the students who can receive a diploma under Ohio law based upon their Individualized Education Plan, you’re likely going to see somewhere between 77 and 80 percent of Ohio students earning a diploma. This is lower than the 84 percent average graduation rate under the Ohio Graduation Test framework, but it’s a far cry from the disaster that many predicted. Moreover, this was the data for year one of significantly increased expectations. What would have happened in year 2? My money is on students and schools adjusting to the higher expectations and improved performance. That means more students graduating ready for college or prepared to directly enter the workforce.

2. If you’re still of the mind that this is a problem that needs to be resolved, there’s a better solution than extending the requirements that had been used for the Class of 2018. This body gave the State Board of Education the authority to set the number of end-of-course exam points needed to earn a diploma. Eighteen points isn’t law. It’s a decision of the state board. If they truly believe that Ohio students need more time to adapt to the higher expectations, they could immediately pass a resolution to lower the points required to the number they see fit. This would give relief to students immediately while giving the legislature time to get more data from ODE and determine its long-term approach to the issue. Despite this power, the state board seems intent on abdicating its statutory responsibility and asking the legislature to be the one to lower expectations for Ohio students.  

Don’t do it.

When you raised the bar for graduation in 2014, you put the needs of students above your own comfort. I urge you maintain high expectations for all students. If you don’t, it’ll be the poorest, most disadvantaged students who suffer in the long run. They won’t get the additional supports they need to increase their likelihood of future success. Moreover, when the legislature weakens its expectations for graduation it communicates one of two things to students: either learning doesn’t matter, or we don’t believe you can reach these expectations. We owe it to our students, their families, and every citizen in Ohio to ensure that our graduates don’t earn participation trophies, but meaningful diplomas that indicate basic mastery of reading, writing, and math.