Were there any shootings at your workplace last year? Want some time to think about it? Better check the files or ask the H.R. department. Maybe you were out that day or forgot. Are you really, completely, hand-to-God, one hundred percent certain you know for a fact whether there was or was not a shooting at your work in the last twelve months?
It’s not a trick question. Of course you’re certain. If someone had fired a shot in anger in your office, factory, or school, it’s something you wouldn’t forget quickly. Or ever.
If you missed the news in the run-up to the Labor Day holiday, Anya Kamenetz of NPR committed a remarkable act of journalism last week. According to U.S. Education Department data for the 2015–16 school year, 240 schools reported at least one incident involving a school-related shooting. “NPR reached out to every one of those schools repeatedly over the course of three months,” Kamenetz reported, “and found that more than two-thirds of these reported incidents never happened.” Of the 240 incidents reported by the U.S. Ed Department’s Civil Rights Data Collection (CRDC), NPR was able confirm only eleven.
Eleven versus 240 is not, to put it blandly, a small discrepancy. Kamenetz cited a separate investigation by the ACLU of Southern California, which could also verify fewer than a dozen of the incidents, while confirming an astonishing 59 percent error rate in the CRDC data, which also informs our views and policy positions on things like chronic absenteeism—over six million students reportedly missed fifteen or more days of school—and corporal punishment. (Aside to Ms. Kamenetz: If NPR wants to continue its valuable service playing MythBusters, track down some of the 110,000 students the CRDC data claims were subjected to corporal punishment. I’ll wager heavily that number is wildly overstated, misreported, or based on an impressionistic definition that most of us wouldn’t recognize or accept as “corporal punishment.”)
If an incident as rare and binary as a shooting—either a gun was fired in a school or it was not—is challenging to recall, record, and report accurately, then how much confidence should we have in OCR statistics or other data on attendance, access to rigorous coursework, special education services, incidents of bullying, and the myriad other data points that inform education policy and practice? Do we even want to talk about charged and volatile issues like school suspensions and disparate impact? If we are relying on 100,000 public schools to accurately capture, code, and self-report a wide spectrum of critical data, it is not unreasonable to wonder what else they get wrong. And how wrong.
The Data Quality Campaign, a thirteen-year old nonprofit that advocates for improved collection, availability, and use of high-quality education data, was sufficiently alarmed by the NPR report to issue a statement noting that good data “takes time to collect and report accurately…Because the information used in the NPR story is from the first year that all schools were required to report information as part of the CRDC, it still needs the benefit of time to ensure that the data is both accurate and reliable.” Most school-level data is self-reported, explains Paige Kowalski, the organization’s executive vice president. Much depends on the person responsible for data entry in a school (seldom a full-time responsibility). What is their job? Do they understand how this information is used and why it might be important to do it accurately? Are there stakes attached to getting it right? “It gets down to what does it take to get quality data, what’s the role of the feds, what’s the role of the state, what’s the role of the district?” she explains. “And finally, that person who sits in the school who has the phone on their ear, typing with one hand, talking to a student across the way, and eating their lunch all at the same time.”
Kowalski recalls being astounded early in her career that "states didn’t have an accurate count of the number of boys and the number of girls enrolled in their schools because it was up to the district to determine data fields and definitions. We couldn’t get gender right,” she tells me. “But once we understood why, it wasn’t complex to fix.” Another frustration was over the wildly different ways four-year high school graduation rates were counted and calculated. The challenges, she says, tend to be a function of how questions are worded, data collected, coded, and reported. Reasonable explanations, but they don’t bolster the public’s confidence that we are on firm ground when we make confident assertions about “what we know” based on data.
The eyebrow-raising NPR report caps off a bad few months for data, research, and evidence-based practice. Researchers failed to replicate landmark studies like the “marshmallow test” of delayed gratification. The validity of the widely cited “30 million word gap” study of home language has come under suspicion. Last month Jay Greene of the University of Arkansas lit a small brush fire that deserved to be bigger calling into question whether political bias affects what research gets published in leading journals. (Spoiler alert: It does). Earlier this year, a feel-good story about graduation rates at a high-poverty Washington, D.C., high school turned out to be mostly bogus. We continue to tout “historic” high school graduation rates having no idea whether kids are graduating college and career ready, or being merely kicked up and out via various credit recovery schemes (I know which way I’m betting).
It’s not surprising when data with stakes attached, like graduation rates, are off; schools have every incentive to report data casting themselves in the most favorable light. It’s harder to explain away getting something like school shootings egregiously wrong, neither noticing nor caring when the data don’t pass the smell test. Credulousness is surely a factor. The lines between education research, advocacy, activism, and agenda-driven media coverage get blurry at times, increasing the likelihood that we will either actively promote or fail to apply appropriate skepticism to “policy-based evidence-making.” If you favor strict gun control measures to combat school shootings, for example, then an epidemic level of incidents reaffirms your sense of crisis and advances your narrative and prescription. If the idea of arming teachers to stop school shooters alarms you, it behooves your policy argument to note how vanishingly rare are such events. Making decisions on behalf of children is muddied even further with wholly invented data and statistics wielded by the nakedly self-interested pushing for changes in curriculum, pedagogy, and practice. You’ve surely heard, for example—and perhaps repeated—that we must be radically transform schools, since 85 percent of the jobs that today’s students will do as adults haven’t been invented yet. Or maybe it’s 65 percent. No, wait. It’s 60 percent. But such imprecision hardly matters when one-third of all jobs are about to be automated. Or one-half. Or whatever.
The crisis of confidence in data, if that’s what this is, ostensibly benefits the testing-and-accountability wing of ed reform, since, as my colleague Mike Petrilli notes, it’s hard to misreport test scores. It also bolsters the arguments of those of us inclined to weigh more heavily parental prerogative, including greater latitude for school choice. If we can’t trust our data to tell us what’s going on in schools, what parents see with their own eyes is less easily dismissed. But the bottom line is this: If your pet reform policy, program, or initiative rests on “what we know” based on school-reported data, this might be a good time to change the subject.
