Jie Chen and Thomas Ferguson, University of Massachusetts
February 20, 2002
Speaking of intricate analyses, University of Massachusetts political scientist Thomas Ferguson and statistician Jie Chen recently unveiled this hundred-page look at the performance of Bay State school districts on the state's much-discussed Massachusetts Comprehensive Assessment System (MCAS). It's caused something of a stir because of its assertion that some of the state's wealthier and more prominent school districts haven't lived up to expectations on MCAS, and its claim that "disadvantaged districts are progressing at rates which are not systematically different from those of richer districts." We're skeptical. Setting aside the senior author's association with such publications as The Nation and Mother Jones, this analysis seems to come from the kitchen-sink school of social science. Using various econometric techniques to try to isolate the effects of a bunch of different variables on districts' MCAS scores, it makes such odd assertions as that "athletic budgets have substantial impacts on district test scores" and "districts with competitive Senate races...have higher MCAS scores." We found ourselves wondering why they hadn't looked to see if the superintendent is left-handed or if the dogcatcher election also bears on MCAS scores. Do sunspots affect them? Ambient air quality? The incidence of tattoos on the school nurse? It looks as if the authors assembled whatever they could get their hands on by way of district-level data and played around to see what might show a relationship to test scores. Note, too, that the latest year of test scores they examined was 2000, before the impressive gains that many Massachusetts districts (and students) racked up in 2001. In fact, the score changes between 1998 and 2000 were quite small. In top-scoring Harvard (the town, not the university), the district average went from 245.47 to 246.37. In middle-scoring Plymouth, it went from 233.72 to 234.95. And in bottom-scoring Holyoke it declined fractionally from 217.49 to 217.32. It seems to us that only a headline-seeking social scientist with little else to do would expend this much effort trying to account for those wee changes. If you want to see for yourself, a PDF version of the report can be found at http://www.mccormack.umb.edu/Publications/docs/MCAS022602.pdf.
