The left-leaning Think Tank Review Project reviews virtually every analytic report that Fordham publishes—and they have yet to find one that they like. So it is completely unsurprising that they issued an unfavorable review last week of our report Do High Flyers Maintain Their Altitude? Typically, we don't engage with them since it is clear that their ideology is at least as important to how they view Fordham's work as is our work itself. But after having read their review, we and the Kingsbury Center, who we contracted to conduct the study, wanted to clarify three issues that were off base in Dr. Jaekyung Lee's review.
1) First, ours was a descriptive study, seeking to catalog the degree to which early high achievers are losing their way. Mr. Lee described our method as a “black-box approach that assumes a link between its findings and NCLB-related policies”. Yet nowhere in the report (including the Fordham-penned Foreword) do we claim that the growth patterns observed among high achievers were the direct result of NCLB. Rather, we acknowledged that many factors could partially explain these disturbing numbers.
2) Second, Mr. Lee took issue with particular aspects of our methodology, some of which appeared to stem from his own misunderstanding. For example, he remarked that our use of percentile rank metrics must create “winners” and “losers.” But it is common practice to group students by percentile ranking, and it's the nature of the study that when one refers to high achievers, there are also non-high achievers. That said, the percentile ranks used in our study were based on external norms, meaning that the numbers of high achievers need not remain static—and there is no such thing as a “zero sum game”. In fact, the numbers of high achievers increased over time for all four of our cohorts. Additionally, we addressed the issue of measurement by calculating the likely rates of attrition due to measurement error, finding that the numbers of Descenders were still larger than expected. We did not consider regression to the mean as a separate issue from measurement error, since the high correlation over time between the MAP assessments rendered the issue relatively trivial (see here).
3) Third, and finally, we were perplexed by Dr. Lee's proposed “alternative” analyses, which he offered as a validity check to our study. He followed completely different samples of students longitudinally over a different age/time span than in our study, using different assessments. Then he uses his findings as evidence that our findings are “biased” (p. 10), simply because they were not consistent with his own. Why would one expect two different studies using entirely different methods and measures to produce the same findings?
We designed a study to describe achievement trends among high performers and concluded that the number of students “descending” was unacceptable. We believe these findings have serious consequences for our society, and in particular, for children and their families. We've made no attempt to use the evidence of this study as an argument against the need for improving the prospects of low achievers, especially students from disadvantaged backgrounds. But we do believe that all students, high achieving or not, deserve equal consideration under state and federal policy.
--Fordham and the Kingsbury Center at NWEA