With little fanfare, the New York City Department of Education (DOE) last month released a draft of its new “School Quality Snapshot”—Chancellor Carmen Fariña’s bid to evaluate each of Gotham’s more than 1,800 schools based on “multiple measures.” The DOE’s website invites public comment on the new reports until May 8. Here’s mine:
I confess I wasn’t the biggest fan of the single-letter-grade school report cards of the Bloomberg-Klein era. But as a signaling device to schools and teachers about what mattered to the higher-ups at the DOE’s Tweed Courthouse headquarters, they were clear and unambiguous: Raise test scores, but most importantly raise everyone’s test scores. With 85 percent of a school’s grade based on test scores—and 60 percent of the total based on test score growth—the report cards, for good or for ill, left little room for doubt that testing was king. Valorizing growth was an earnest attempt to measure schools’ contributions to student learning, not merely demographics or zip code.
Fariña, whose contempt for data I’ve remarked upon previously, values “trust.” You may worry if your child can’t read or do math. So does Chancellor Fariña, sort of. But she deeply cares if “teachers trust the principal at his or her word,” whether “teachers trust each other,” if students say teachers treat them with respect, and if parents say that a school’s staff “works hard to build a trusting relationship with parents.” The signal feature of the new report cards is the warm, enveloping embrace of the circle of trust.
Other elements of the draft Snapshot are equally warm and fuzzy. Sure, “rigorous instruction” is on top, but “student achievement,” like the proverbial pony, is buried beneath a pile of collaborative teachers, supportive environments, and strong family-community ties. In short, this document feels driven more by philosophy than data, relying on qualitative measures of uncertain value while strongly de-emphasizing student achievement in general and student growth specifically.
When she announced her intention to overhaul the school report cards last year, Fariña promised “the first balanced picture of a school’s quality,” one that “reflects our promise to stop judging students and schools based on a single, summative grade.” To be fair, it was widely acknowledged even in the pre-Fariña DOE that the reductive nature of the school report cards was having unintended ill effects. A 2013 white paper noted that the existing progress reports’ overreliance on state tests meant that “some educators have felt pressure to engage in test prep, narrowing the curriculum,” and that “finding the right balance of multiple measures is critical.” Even top-performing schools were anxious about their letter grades.
However, there are a couple of serious problems with this offering. Like a driver overcorrecting and losing control, the new reports go from focusing almost exclusively on student achievement to making it one of seven areas of reporting. And where 85 percent of a school’s value was based on test scores in the past, no weighting is given to any area of evaluation in the new reports. The biggest and potentially most troubling change is the removal of a coherent growth methodology by which schools are measured. The risk in de-emphasizing growth is that it might push schools and teachers back to focusing on how many kids pass the tests, which incentivizes focusing on “bubble kids” at or near the passing line. If a school’s going to be successful, everyone needs to improve.
If test data isn’t driving the new Snapshot, what is? Most of the information comes from the NYC School Survey administered annually to parents, teachers, and students, or else from a school’s “quality review”—ostensibly an extensive school visit in which an experienced educator observes classrooms, interviews school leaders, and evaluates how well the enterprise supports student achievement. Curriculum, teaching, assessments, school culture, and teacher collaboration are among the areas scrutinized during each school’s review.
In theory, it’s a robust and useful lens through which to view a school. At present, though, not every New York City school is reviewed every year (it’s been three years since some schools have been reviewed). Worse, there are reports that what used to be a rigorous three-day inspection is now being performed in as little as half a day—little more than a drive-by. (A DOE spokesperson has not responded to a request for comment on the number or duration of reviews performed or scheduled this year). Meanwhile, charter schools are not subject to quality review at all, which will render the Snapshots virtually meaningless (many growth-focused charters, it should be noted, benefitted mightily from the single-letter-grade report cards).
A former DOE official lamented to me privately that nearly everyone with data expertise was driven out of Tweed when Fariña took over. Thus, perhaps the greatest concern is not even the new Snapshots’ favoring qualitative measures over student achievement, but Tweed’s capacity to gather meaningful, high-quality data even on the things it prizes.
In sum, balancing achievement data with a quality review makes good sense; it was the direction in which the DOE was headed even before Fariña . But devaluing test data—combined with infrequent or cursory reviews—risks making what should be a dynamic and useful report to parents and teachers either static or a mere exercise in confirmation bias. At heart, New York City’s School Quality Snapshot feels like a political text. It is not hard to foresee it casting a warm glow on the kinds of schools Fariña likes, whether or not student achievement is robust.
Having worked in a DOE school for several years, I can attest that schools focus on gestures from Tweed as if they were smoke signals from the Sistine Chapel when a new pope is chosen. Thus, the most important function these reports serve is to indicate what matters to schools and teachers. The draft School Quality Snapshot says clearly and unambiguously that the days of measuring a school by academic performance in New York City are over.
