photo credit: ChalkbeatNY via photopin cc
With the release last month of the latest round test scores, Success Academy founder and CEO Eva Moskowitz is now a bona fide national-education-reform celebrity. She is also the latest in a line of educator-activists—like Michelle Rhee or Diane Ravitch—who embody, for supporters and opponents alike, one “side” of the education-reform debate. As a result, discussions about the stellar results posted by her schools have generated much more heat than light. Allies are eager to elevate her to exemplar status and critics desperate to prevent her from receiving even a modicum of credit for her students’ success.
The argument breaks down along predictable lines. For advocates of charter schools and education reform more generally, Success Academies demonstrate what can be accomplished when a strong leader with a laser-like focus on student achievement can do when she is free to hire the best teachers for the neediest students.
Critics of charter schools and reform are equally confident that Moskowitz is, in effect, running a scam: skimming the easiest-to-teach students, pushing out the ones who prove difficult later, and claiming all the while to be doing better work than the city’s other educators who are forced to operate without her considerable advantages and educational slight-of-hand.
What neither side is willing or able to admit, however, is how little we really know about why Moskowitz’s students seem to be doing so much better than so many of the other children in New York City—far better even than her charter school peers. And make no mistake; students at Success Academies were substantially more likely to be rated academically proficient than their peers across New York City. This was true in every grade level tested and for both math and English language arts.
Rather than try to explain those results in partisan or ideologically convenient terms, I want to try to lay out what we know and do not know about Success Academy. This, in turn, will hopefully help illuminate what we still need to learn if we want to explain—or hope to reproduce—the impressive test scores of Eva Moskowitz’s students.
Creaming
What we know
The Success Academies are charter schools to which students—or their families—must apply. It is therefore possible that SA students are not perfectly representative of students in NYC overall, or of students in the schools’ surrounding neighborhoods.
Indeed, we have some measures of student characteristics that demonstrate that Success Academies are not perfectly representative. Below are the demographics of the seven oldest Success Academies (those for which school-progress report data are available) compared to the city as a whole.
Seven oldest Success Academies | Citywide | |
Black or Hispanic | 95.9% | 68.9% |
English learners | 6.6% | 13.6% |
SPED | 14.0% | 17.5% |
What we do not know
Even the demographic information that is most commonly collected from and reported about students offers only a crude picture of the academic advantages or disadvantages that they have independent of their school. Whether students are classified as having special-education needs, for example, tells you very little about the severity of their disabilities. We also do not collect reliable, accessible data about student characteristics—like conscientiousness or IQ—that may have substantial, direct implications for academic success, and which may not be distributed equally among students who do or do not apply to charter schools.
Measured differences in student characteristics between schools may also reflect school practices as much as they indicate student characteristics. So, for example, we do not know if Success Academies have fewer English learners in part because they help students reach English proficiency more rapidly, as Moskowitz has claimed. There is also evidence that charter schools are less likely than district schools to enroll students with disabilities and less likely to place students in special education or keep them there.
Attrition
What we know
Cohorts at Success Academies often shrink substantially over the years. The first SA school—Harlem 1—opened with seventy-three first-grade students for the 2006–07 school year. This year, however, Harlem 1 tested only thirty-two eighth-graders. In other words, the inaugural cohort of first-graders at Harlem 1 shrank by more than half—56 percent—before the end of eighth grade.
Cohort shrinkage at Success Academies is the norm, though it is not always so extreme. Harlem 4, for example, opened with seventy-eight first-graders in 2008 but tested only fifty-four in 2014, a decline of 30 percent. In 2011, the four Harlem Success Academies collectively tested 323 third-grade students in English language arts. In 2014, those four schools together tested only 259 sixth-graders, a decline of 20 percent in three years. Success Academies tested 753 third-graders in 2013, but only 681 fourth-graders in 2014, a drop of almost 10 percent.
Net drops in enrollment like this may be uncommon and suggest that Success Academies are not replacing students as quickly as they leave. However, it is not unusual for students in New York City to leave their schools; slightly more than 10 percent of elementary-age children do so every year, with black, Hispanic, and poorer students doing so more frequently.
What we do not know
Which students are leaving Success Academies and why? There have been claims that Success Academy schools are especially likely to push out students with special needs or with behavior problems, but we do not have good evidence on overall patterns of attrition—how are Success Academy students replaced, if at all—or how those patterns are different from those at other schools in the city.
Peer effects
What we know
If Success Academy is successful in attracting and retaining high-achieving students, it could have an impact on scores of even lower-achievers. In at least some situations, students appear to benefit from being surrounded by stronger or more-advantaged students, while weaker students have been found to be detrimental to the learning of their classmates by reducing the quality of classroom time. Other authors have concluded that girls are particularly advantaged by the presence of stronger students.
Notably, however, other studies—including studies of charter schools—find that peer effects do not seem to matter. Some researchers have found, for example, that “peer effects…play no role in explaining charter effectiveness.” Instead, they suggest that whether a charter school is effective depends almost entirely on its approach to instruction and student behavior.
What we do not know
The exact magnitude of peer effects are difficult to measure, and probably not consistent across contexts, so it is difficult to say with certainty how much students in Success Academies are benefitting from the composition of their student body—or whether Moskowitz’s schools attract higher-potential kids in the first place. Are students motivated to work harder when their peers are more hard-working and conscientious? These mechanisms are not well understood in general and have not been carefully studied at Success Academies in particular.
Test prep
What we know
“Test prep,” a term of some ambiguity, generally refers to teaching practices that prepare students to score well on tests without developing their knowledge or understanding of the skills or concepts being tested.
There is some evidence that test prep is relatively ineffective at boosting students’ test scores. The Measures of Effective Teaching Project, for example, found that “test-prep” practices were only weakly (though positively) correlated with value-added measures of teacher effectiveness.
What we do not know
We do not know how Success Academies prepare their students for state testing, or the extent to which their teachers rely on “test prep” as opposed to instruction aligned with state standards and tests. We also do not know whether more class time is devoted to test prep at Success Academies than at other schools in New York City (or elsewhere). More generally, existing evidence on the effectiveness of test prep is far from definitive.
School culture and instructional effectiveness
What we know
Success Academy self-identifies as a “no-excuses” school, an approach generally associated with increased instructional time, strict rules for student behavior, selective teacher hiring, “data-driven” instruction, “high expectations,” and a focus on traditional math and reading skills.
There is evidence that “no-excuses” schools effectively boost student achievement in at least some contexts. The most effective charter schools are often “no-excuses” schools, and the approach seems to be effective in raising test scores when implemented in district (i.e., non-charter) middle schools.
Moskowitz also describes her schools’ approaches to reading instruction as drawing heavily from “balanced literacy.” That approach has a controversial history in New York, and there is some evidence that it is relatively ineffective, so it is also notable that Moskowitz emphasizes that her schools have modified balanced literacy in significant ways.
What we do not know
We know relatively little about Success Academy’s school culture and instructional practices and how they differ from those in other schools in the city. It would also be useful to know more about the schools’ instructional approach to math, and the extent to which math and reading instruction at Success Academies is supplemented by—or crowds out—instruction in other content areas like science and history.
Conclusion
The various factors potentially contributing to Success Academies’ remarkably high test scores are not mutually exclusive; it may well be they are all at work to one degree or another.
Nor are these factors necessarily independent of one another. Strict “no-excuses” policies toward behavior may affect the type of students who are attracted to—or leave from—Success Academy schools, for example, or the charter application process may contribute to a student body that is somehow better suited to the schools’ instructional approaches.
What may be most important to note is that these issues are not, for the most part, unique to Success Academies. Charter schools of all kinds routinely boast—or are accused of exploiting—these very same factors. Any credible account of Moskowitz’s apparent success needs to explain not only why her students test so well, but also why they test so much better than even students at other, seemingly-similar charter schools.
Given how little we know about Success Academies, we should be very cautious about leaping to conclusions—positive or negative—about the factors contributing to their students’ success.
Paul Bruno has taught middle school science at district and charter schools in Oakland and Los Angeles, California.
