A push by some charter advocates resulted in a last-minute amendment to House Bill 2 which may introduce the “California Similar Students Measures” (CSSM) into Ohio’s school-accountability system. This is an entirely unnecessary effort, and CSSM should not be implemented in the Buckeye State.
The California Charter Schools Association developed CSSM, a simple regression model that uses school-level data, to approximate a value-added student growth model. The reason: California does not have an official student growth measure. CCSM is an improvement over using only a school’s raw proficiency results to evaluate schools, and the organization deserves credit for implementing it in California. However, a CSSM-like analysis should only be used in the absence of a proper student growth measure—and as such, it has no place in Ohio.
Ohio legislators should read very carefully CCSA’s own caveat emptor (emphasis added):
While CCSA believes these metrics [CSSMs] are an improvement on the existing measures in law for charter renewal, longitudinally linked, individual student growth data is the ideal source for most appropriately assessing a school’s performance. Because the Similar Students Measure is calculated with aggregate school-level data, it is an approximation of value-added modeling. True value-added modeling requires individual student data connected to the schools and educators instructing those students.
Then lawmakers should read the statement above again, with this information in mind:
Ohio, along with other states like Pennsylvania, North Carolina, and Tennessee, uses the SAS/EVAAS value-added model. Indeed, this model uses longitudinally linked, individual-level data to calculate a school’s value-added result.
The clear advantage of using student data is that they allow analysts to distinguish actual student learning gains, tracked over time. This is the crucial information needed to fairly and robustly evaluate a school. The student data allow analysts to answer the fundamental question: Is a school making an impact on students (i.e., contributing to learning gains)? This is why many states, including Ohio, have wisely incorporated a value-added model into their accountability systems.
Worth noting also is that the finest education research uses student-level data, generally avoiding school-level data. Prominent researchers have used rich troves of (non-identifiable) student-level data to evaluate states’ charter schools, the long-run benefit of effective teachers, voucher and tax credit programs, early college high schools, and school closures.
Unfortunately, school-level data do not allow analysts to chart student learning trajectories. As a result, analysts cannot make strong claims about the impact of a school when using only school-level data—which explains the clear note of caution from CCSA.
Moreover, using school-level data could create perverse incentives that may actually harm students. When applied widely, a CCSM-like model could encourage schools to manipulate their student demographics to their own advantage. For example: A school could be tempted to counsel out low-achieving students, since raw achievement, on school level, is the outcome variable used in CCSM’s regression. (The same perverse incentive exists when a proficiency measure alone is used for accountability.) Yet because it examines students’ learning trajectories over time—not a static measure easily known to school administrators—the SAS/EVAAS model is less susceptible to manipulation.
The SAS/EVAAS statistical model isn’t the Holy Grail of student growth measures. But it’s a whole lot better than the California model, both in its technical properties and the incentives that it establishes. If the legislature adopted CSSM for use, it would be like trading in a Ferrari for a horse and buggy.
***
So what’s with the dissatisfaction with the SAS/EVAAS model, and the sudden impulse to introduce the California model? I’ve heard several off-base criticisms of Ohio’s value-added model, which are based on misconceptions. Let’s address two particular issues that may be driving the discussion.
One argument is that the SAS/EVAAS model cannot be valid because it doesn’t yield a “bell-shaped quality curve.” Many, including myself, have observed the odd distribution of the state’s A–F value-added ratings (Figure 1). Something certainly seems amiss.
Figure 1: Distribution of value-added ratings, Ohio schools, 2013–14
[[{"fid":"114450","view_mode":"default","fields":{"format":"default"},"type":"media","link_text":null,"attributes":{"class":"media-element file-default"}}]]
But in reality, the value-added scores yield a fairly normal, bell-shaped curve at the school level (Figure 2).
Figure 2: Distribution of value-added scores, Ohio schools, 2013–14
[[{"fid":"114451","view_mode":"default","fields":{"format":"default"},"type":"media","link_text":null,"attributes":{"class":"media-element file-default"}}]]
Source: Ohio Department of Education. Notes: The value-added index scores range from -24.34 to 22.10; a small number of outlier schools are excluded from the chart for clearer display. The mean score is 0.98, and the number of schools is 2,573. The index score is the average estimated gain/standard error; the index scores are used to determine schools’ A–F rating. For the cut-points in the A–F ratings, see here.
The problem isn’t with the results of the value-added model. Rather, the oddly shaped A–F rating distribution is a function of where the legislature has set the “cut points” for each letter grade (ORC 3302.03). Something is lost when the value-added scores are translated into value-added ratings. (The A–F cut points may need to be set differently, but that’s a topic for another day.)
Second, the SAS/EVAAS model is sometimes characterized as a “black box.” It is true that the statistical modelling is complex and that the student data—as they should be—are not made available to the public. Yet there is also ample documentation of the procedures, business rules, and method that SAS uses to conduct the data analysis. You can read its technical manual and a brief that summarizes the models (and common misconceptions) in everyday language. Meanwhile, there is no clear reason to believe that SAS—a widely respected IT company (think IBM or Oracle)—would be anything less than an independent and credible partner when analyzing Ohio data.
In the end, the precise mechanics behind products are often unfamiliar to the layperson. Do I perfectly understand the inner workings of a car engine? Nope. Could I describe, in exact detail, how a cell phone functions? Or the World Wide Web? I could not. But neither do I disparage car engines, cell phones, or the Internet as “black boxes.”
***
Let’s be frank. Those most apt to criticize the SAS/EVAAS value-added model are probably also those affiliated with the schools that perform poorly along this measure. It’s no secret that in the education world, virtually everyone believes their own school is da bomb—even when there may be trouble brewing. When the results aren’t glowing, it’s easy to criticize the methodology or dismiss the results as “invalid.” Sometimes they’ll even cook up their own methodologies that (surprise!) make their schools look better. All sour grapes, I say.
There is no compelling reason for Ohio to incorporate the California model into the state accountability system. Compared to Ohio’s value-added measure, CSSM is a very crude way of getting at student growth. Heck, don’t even take my advice—just listen to its creators.
