The Ohio Department of Education is expected to release report cards for the 2016-17 school year by the end of this week. Like an annual checkup with a physician, these report cards offer valuable information on the academic health of Buckeye schools and students.
The Ohio Department of Education is expected to release report cards for the 2016-17 school year by the end of this week. Like an annual checkup with a physician, these report cards offer valuable information on the academic health of Buckeye schools and students.
As many Ohioans know, state leaders have overhauled the assessment and report card system in recent years. To their credit, they’ve implemented more demanding state exams that now offer a clearer picture of student proficiency than under former assessments. The report cards themselves are much different from those in years past; they now include various A-F components that consider not only traditional measures like proficiency and graduation rates, but also pupils’ growth over time and their readiness for college or career. While Ohio legislators still need to do considerable work to help report cards function properly—we’ll be releasing several recommendations for tweaking them next month—the stability in state assessment policies and on key pieces of the school grading system is praiseworthy.
What are we keeping an eye out for when report cards drop? Here are three things:
Will the use of multi-year averages help to stabilize value-added ratings?
In recent years, one of the more concerning developments has been the instability of the value-added ratings—a critical gauge of student growth over time. Last year, a fair number of districts experienced dramatic swings in their value-added ratings relative to the prior year; for example, Dayton Public Schools went from an F to an A on this measure in the course of a single academic year. Some of the volatility we saw was likely due to the recent use single-year value-added data, rather than multi-year averages, which help to smooth random fluctuations. (The one-year results were used as Ohio transitioned state exams.) Now that Ohio has implemented two years of the same assessment, value-added ratings based on multi-year averages should return. Multi-year averaging is a good practice that should yield clearer views of school effectiveness and hopefully bolster public confidence in this important accountability measure.
Will Ohio’s charter schools again outperform district schools in the Big Eight?
Last year, brick-and-mortar charters fared modestly better than comparable urban district schools on value-added results. My analysis found that 29 percent of Buckeye charters received an A or B on value- added compared to 19 percent of Big Eight district schools. Conversely, a smaller proportion of charters were rated D or F compared to district run schools (52 to 68 percent). While one year of data cannot be deemed conclusive evidence that Ohio’s charter sector is on the upswing after the enactment of major reforms in fall 2015, we took it as a hopeful sign of improvement. We’ll be on the lookout to see whether the charter advantage persists with the addition of a second year of value-added data. As usual, we’ll again dissect the performance of Ohio’s urban charters in comparison to their urban district counterparts. And yes, this is our friendly reminder that it is misleading to compare urban charter performance to schools statewide—especially on proficiency-based measures! Which leads us to…
Will nearly all high-poverty schools get slammed on most report card components?
There used to be a TV show called Early Edition where the main character received the next day’s newspaper a day early. This meant that he knew for certain things like the winning lottery numbers and where and when tragedies would strike. Alas, for high-poverty schools, they don’t need to get early editions to predict their results on the vast majority of Ohio’s report card components. In all likelihood, they won’t be pretty. For as long as Ohio has issued report cards, high-poverty schools have struggled mightily when judged on the basis of “status” measures such as proficiency rates, performance indexes, graduation rates, and now college admissions test results. Research consistently indicates that disadvantaged pupils tend to start out behind their peers, which places high-poverty schools at a greater risk of receiving of Ds and Fs on measures that don’t control for prior achievement—not necessarily due to ineffective instruction but because their students have such big gaps to close.
A quick quiz: What is the only component that isn’t as closely correlated with pupil background characteristics? Answer: The state’s value-added measure, which accounts for students’ prior achievement levels, and hence more evenhandedly evaluates school performance by looking at student growth over time. However, this report card element is not given enough emphasis on report cards, leaving the public with the distorted view that virtually all high-poverty schools are failures—even those that are making an extraordinary impact on pupils’ academic trajectories.
* * *
Stay tuned to the Ohio Gadfly Daily as report cards are released. We’ll be looking into these—and other questions—and we hope you’ll turn to us as a source for objective, fair analyses of the report card data. And remember: You can also follow us on Twitter (@ohiogadfly) and Facebook for commentary and quick facts as report cards come out.
In response to widespread fears that too many students would fail to pass the state’s seven high school End Of Course (EOC) tests, Ohio lawmakers recently created additional graduation pathways for the class of 2018. The pathway generating the most discussion allows students to receive a diploma by completing two of nine alternative measures, one of which is earning at least a 2.5 grade point average (GPA) during their senior year.
State Superintendent Paolo DeMaria has defended the inclusion of GPAs as one of the options, saying that “a GPA increasingly both in research and in practice has been shown to be a far better indicator of a student’s readiness for college success and frankly for workforce success than any standardized test.”
DeMaria is partly right. Several analyses have found a link between students’ high school grade average and their college success. For instance, this study from the Institute of Education Sciences (IES) found not only that high school GPAs are an “extremely good and consistent predictor of college performance,” but also that they “encapsulate all the predictive power of a full high school transcript in explaining college outcomes.” These other two studies found that a student’s GPA and her composite ACT score were more predictive than either option alone. (There is little research evidence linking GPAs and workforce success.)
Yet GPAs pose several puzzles. No one denies that a student’s work ethic and level of investment matter—and influence achievement. GPAs can give an indication of the non-cognitive skills that a student will need to be successful at the next level. But before anyone declares the superior predictive power of GPAs and the wisdom of using this to determine whether a student should receive a high school diploma, some important nuances in the research need to be addressed. Consider:
Along with these important nuances, one more issue should make policymakers wary of putting too much emphasis on GPAs: their susceptibility to gaming, even when they are a low-stakes measure. One way this occurs is through grade inflation—giving students higher grades than their performance honestly warrants. I taught plenty of students who came to my class with high GPAs, worked very hard, and turned in all their assignments—yet still could not read and write proficiently. If their readiness for college and career were based entirely on their GPAs, they would have been considered well-prepared. In reality, they were years behind.
Kids who are academically behind aren’t the only ones who see inflated grades. A recent article in The Atlantic found that students enrolled in private and suburban public high schools are awarded higher grades than their urban peers despite similar levels of talent and potential. In fact, between 1998 and 2016, the GPAs of students at private and suburban public high schools went up even as scores on the SAT went down.
When it comes to relying on GPAs alone as indicators of student readiness and downplaying the role of standardized tests, policymakers should tread cautiously. Research certainly indicates that high school grades are an important predictor of college success, but many of these studies overlook the students who will enter the military or the workforce and also need to have a strong academic foundation. Furthermore, without research that specifically links GPAs to workforce readiness and performance, we can’t accurately argue that GPAs predict better than tests like WorkKeys, i.e. tests specifically designed to measure career preparedness.
In an era of rampant grade inflation, it’s irresponsible for state leaders to push aside objective measures that can gauge a student’s academic preparedness and that are not entirely within the control of that student’s teachers. Standardized tests don’t tell us everything we need to know about student potential, achievement, or ability, but they do shine an objective beam on what students know and are able to do. As the old saying goes, it’s important to trust but verify—and good assessments are a key part of verifying that students’ grades align with what they’ve actually learned and are able to do.
The impacts of school closures on student achievement have been studied in various locales, including Louisiana, Michigan, New York City, and our home state of Ohio. But a new report from Stanford University’s Center for Research on Education Outcomes (CREDO) is the most expansive analysis on this topic to date. Analysts examine 1,204 district and 318 charter school closures across twenty-six states occurring between 2006–07 and 2012–13. The study focuses on low-achieving schools—those whose students’ math and reading scores were in the bottom quintile in their respective states prior to closure. Most of the closures took place in urban areas—69 percent across both district and charter sectors.
CREDO offers a comprehensive view of closures in these states, including the patterns of closures and their academic impacts on students. Though not an experimental analysis, by matching students on observable characteristics and baseline achievement, CREDO does offer solid evidence on the impacts of closure. There were a few striking findings.
First, the schools that closed were on downward trajectories prior to shutting. The analysis found that average math and reading achievement were deteriorating across these schools, as was average student enrollment. The enrollment data suggest that market-oriented pressures—less demand for these schools—played a role in these closures. Districts and charter authorizers then took the step of permanently shuttering these schools.
Second, a majority of pupils displaced by closure landed in similar or better performing schools. CREDO found that 48 and 45 percent of students in closed charter and district schools, respectively, went on to higher-performing schools, while another one in three transferred to schools of similar quality. Although some media coverage focused only on students landing in higher-quality schools—perhaps portraying closures in a more negative light—just a small proportion of displaced students landed in a lower performing school (about 15 percent). To gauge whether closures induced transfers to higher or lower quality schools, CREDO compared schools’ percentile rankings on math and reading scores. That method departs somewhat from prior studies in Ohio and Louisiana, which examined the performance of students’ closed versus new schools based on value-added scores, a measure of student growth. In Ohio, we considered both changes in school-level achievement and growth, finding similar results across both methods.
Third, the quality of the school students attended after closure made a big difference in their academic growth. CREDO estimates that displaced students attending higher-performing schools enjoyed gains of about ten to forty days of learning, post-closure. However, students who transferred to inferior quality schools lost considerable ground—losses that ranged from twenty to eighty days of learning. While this group represented just a fraction of displaced pupils, they experienced a double-whammy: Their education was not only disrupted but they also attended even weaker schools. This is also what we found in Ohio—and aligns with common sense. School closures can work if displaced students land in better schools. But that’s a big “if.”
The analysts conclude that “The quality of the alternatives matters greatly for the students affected by school closures.” ‘Tis true—whether closing dysfunctional schools benefits students hinges on the quality and capacity of the existing school supply. The key is to grow more high-quality schools, especially in urban communities, so that students stuck in chronically poor-performing schools can move to better ones. That means focusing as much on school openings as school closures.
Chunping Han, et. al., Lights Off: Practice and Impact of Closing Low-Performing Schools, Center for Research on Education Outcomes (2017).
As part of the most recent state budget, Ohio lawmakers created alternative graduation pathways for the class of 2018 in response to widespread fears on the part of district administrators that too many students would fail to pass the seven End Of Course (EOC) tests that are administered during high school in the four core subjects.
We at Fordham strongly opposed this move because we believe it will hurt students in the long run. We weren’t the only ones who questioned it. Nevertheless, the alternative pathways became law. Recently, State Superintendent Paolo DeMaria indicated that objections such as ours were not well founded. Specifically, he told reporters:
The students who aren’t going to do well in college and in the workforce are those who don’t take their education seriously and a GPA increasingly both in research and in practice has been shown to be a far better indicator of a student’s readiness for college success and frankly for workforce success than any standardized test.
Whether, when, and how GPAs may be a better indicator of readiness than standardized tests is a subject for a different day. Let’s focus instead on the Superintendent’s assertion that students who aren’t doing well in college are those who “don’t take their education seriously.”
Yes, there’s a grain of truth here. Those who don’t work hard or flat out don’t care about their education are likely to struggle in school, in college, and well beyond. The greatest teacher on the planet cannot force an unwilling student to learn. But it’s wrong to go from that reality to the suggestion that every Ohio kid who struggles academically or in a job just didn’t take education seriously. Dr. DeMaria is a smart and caring fellow who surely understands that it’s not always lack of will on the kid’s part.
I taught high schoolers. I can still recite the names of those who looked me in the eye and told me they didn’t care about school or about what was happening in my class. But that list is far, far shorter than the number of kids I taught who cared deeply about their education but still failed my class, the state test, or both. They didn’t fail because they didn’t care. They failed because they read at the fourth-grade level in ninth grade, and while they may have made two years of growth in my class, that still didn’t bring them up to grade level. They failed because the school system had failed to educate them before they got to my class. Many were kids who worked hard, participated with enthusiasm, and didn’t cause trouble. Their previous teachers felt bad about failing them (and heaven forbid schools retain them) even though they hadn’t learned what they needed to learn in order to be ready for the next grade. Good kids. Well-meaning teachers. Ineffectual instruction.
Some of the students who failed my class or the state EOC arrived with straight As. Many turned in their homework every day and hung on every word I said in class. Some came in for tutoring after school, before school, and on Saturday. Do you know how hard it is to get teenagers to come to school if they don’t have to be there? They don’t come unless they care. I taught hundreds of students during my teaching career, and I can say one thing with complete and utter certainty: The vast majority of my students cared profoundly about their education and their future.
I didn’t teach in Ohio, but I doubt that Buckeye kids are much different from those I taught. I’m willing to wager that there are thousands of students across this state who care deeply about their educational success, even if they haven’t experienced much of it. But just like my students, these students have been passed from grade to grade because they show up and behave well and because they participate and turn in their homework. They get good grades, but they fail their state tests. Many of these students have probably come to the conclusion that state tests don’t actually matter. Why should they care about a score report they get once a year when the report cards they get four times a year say they’re doing okay?
But then they get to college and get pushed into remediation instead of credit-bearing courses. Or they get out into the workforce and aren’t sure how to write an email or read a technical manual or calculate percentages. Their boss is unhappy, or their professors are unimpressed, and they feel both overwhelmed and betrayed. They probably end up asking a version of the question that I was repeatedly asked by students during my teaching career: How did I perform this poorly when I got all As?
I taught in high-poverty urban schools. But those aren’t the only places where students get high grades and GPAs that don’t necessarily match their achievement. A recent article in The Atlantic examined research on grade inflation and found that students enrolled in private and suburban public high schools are awarded higher grades than their urban peers despite similar levels of talent and potential. In fact, between 1998 and 2016, the grade point averages of students at private high schools and suburban public high schools went up even as scores on the SAT went down.
Yes, kids have a responsibility to take learning seriously. But so do the adults who are charged with making sure that they learn and are held to a high standard of achievement. Those adults include legislators who recently punted on the future of some of those kids. Those adults also include the State Superintendent.
Three years into his first gig as a recruiter/trainer at a job skills program in San Francisco, Mauricio Lim Miller recognized a striking contradiction that changed the trajectory of his life and work. As a person whose family had overcome great personal hardship and who was now trying to help others do the same, he could not reconcile the way he ran his own life with the way he was expected to run a social service program. The proscriptive, top-down structure of so-called benefits programs like his emphasized the “deficits” of their clients and often sought to substitute narrow program rules for individual options. Those rules were sometimes contradictory (as when multiple programs were involved) and sometimes self-defeating (e.g. child-care subsidies that lapsed if a program participant earned a little too much money from work). Even worse, he became convinced that such service programs were conferring greater benefit on their employees than on their clients. When he was invited to attend President Clinton’s State of the Union address as recognition for his work, Miller says he nearly declined out of guilt. As soon as he was given a chance by California Governor Jerry Brown to reshape the assistance available to poor families in his native Oakland, Miller began formulating what he calls The Alternative—a new approach to supporting families experiencing poverty by focusing on strengths and supporting individual efforts toward upward mobility. This approach characterizes a fast-growing effort called The Family Independence Initiative (FII). Miller’s new book chronicles his journey and the expansion of FII from California into Michigan and now Ohio.
The book’s premise is deceptively simple: Miller believes that both parts of the old “give a man a fish” aphorism are wrong. People aren’t fishing not necessarily due to a lack of knowledge; rather they lack the money for a pole or transportation to the water. Or they might not have the time to wait for the fish to bite because of family commitments. But assuming people “don’t know how to fish” and that large scale programs must exist to teach everyone has long been the default version of social service in the United States. Despite good intentions, this has led to paternalistic, expensive, and hidebound service-providing agencies that force poor families into the service pipeline and hold them there. They have been designed to address perceived “deficits” in people’s lives and they ignore–or even quash–families’ and communities’ own efforts to move upward. The alternative approach, the one that animates FII, assumes nothing of families except that they are the “experts of their own lives” and provides nothing to families except support for what they are already working toward on their own. The key missing ingredients in traditional social services are client agency and choice, the very things that often define middle- and upper-class families. “Taking control away from people, or trying to induce them with carrots or sticks,” Miller writes, “is…the opposite of what most of us with privilege would accept.” While he doesn’t talk much about education except as a key recurring goal for the children of families he works with, those of us who support school choice should recognize this mindset and echo its aims.
The alternative approach is so revolutionary as to defy easy description–hence the need for a book to explain it and hence Miller’s well-deserved MacArthur Genius Award. It is an effort to empower families and communities to be their own change agents, mainly by telling them they can be—a new and different message for most FII families. I won’t rob readers of the pleasure of hearing Miller’s own account of the evolution, structure, and successes of FII, but other key features include:
Miller’s own mother would not take a “handout” while struggling to support her family, believing that action to be an admission of helplessness. But despite working three jobs at a time and having the talent to design and make clothing, investment in her strengths and abilities was nowhere to be found. So she struggled. Middle- and upper-class entrepreneurs can regularly access angel investors and seed capital to “bet” on their eventual success, while low-income entrepreneurs can perhaps access a loan that must be paid back, however low the interest might be. FII invests in the efforts of individuals and groups to provide for their own needs, working to nurture those positive deviants and early adopters (entrepreneurs, college graduates, community builders, etc.) to Gladwell’s tipping point. FII also tells the stories of positive change to show that, far from being “needy,” families experiencing poverty are resourceful and capable.
“Our experts and service providers have a very shallow, often siloed view of the complexity of life in and around the poverty level,” writes Miller. “If the various programs want to be helpful, as well as protective of their funding, they need to be more effective and efficient. They need to understand the issues from the perspective of those they want to help and not just the silo of help they represent. To truly assist in solving any problem they need to take direction from the very families they seek to help since they are the experts of their own lives.”
This novel perspective should resonate with the work of education reformers at least as much as it does with social service providers. Stagnant learning environments, low expectations, social promotion, education systems that are more rewarding for the adults working in them than the students supposedly learning in them, and college access without ensuring college readiness are all forms of well-intentioned “handouts” that end up doing more harm than good for the most part. Mauricio Miller’s efforts to change social service work into the type of supports his family truly needed–rather than a rigid and self-perpetuating pipeline of dubious “help”–could be a new pattern for education reform efforts going forward.
SOURCE: Mauricio L. Miller, The Alternative, Lulu Publishing Services (August, 2017).
It’s no secret that high-quality early childhood education can lead to significant and positive short-term impacts for children, particularly those from disadvantaged circumstances. Unfortunately, much of the current research also points to a troubling “fade out” trend—the gains that students make in preschool gradually decrease until they disappear completely.
A recent study from Mathematica seeks to add to this discussion by investigating whether the pre-K programs offered by some KIPP charter schools produce more lasting impacts. Researchers selected KIPP for several reasons, including the fact that it employs several practices that are considered high quality (such as well-educated teachers and low teacher-child ratios). Most significant, though, is that many KIPP pre-K students continue their education in a KIPP elementary school—increasing the probability that their elementary school experience will align with their pre-K experiences, and thereby potentially lead to longer-lasting impacts.
The study explored three research questions and used slightly different methods to examine each. The samples were relatively small, but the analysts were able to employ experimental methods that allow us to draw stronger conclusions about the effects of KIPP pre-K. A series of standardized tests (like the Woodcock-Johnson III Tests of Achievement) were used to measure both academic achievement and executive function (skills that help students do critical things, like focus their attention and multi-task). Interviews of KIPP staff were also conducted.
First, the study examined the effects of attending a KIPP pre-K and elementary school on student achievement. Specifically, it compared outcomes for students who won KIPP admission via lottery in pre-K and remained in a KIPP elementary school through second grade to the outcomes for students who were not offered admission. Researchers looked at three oversubscribed KIPP elementary schools that offered pre-K in two cities. There were two key findings: First, students who attended both KIPP pre-K and elementary schools made significant gains on reading and math achievement by the end of second grade relative to students who did not win the pre-K lottery. Second, KIPP pre-K and elementary schools may also have had a positive impact on student executive function, though the findings were not statistically significant.
Second, researchers aimed to isolate the effects of KIPP pre-K by comparing outcomes for students who enrolled in a KIPP school starting in pre-K to those who enrolled in KIPP starting in kindergarten. The kindergarten cohort was made up a sample of students from five oversubscribed KIPP schools that did not offer pre-K in four cities. Researchers found that KIPP pre-K may provide an additional benefit for reaching achievement above and beyond just attending KIPP elementary schools. Although the results did not reach conventional levels of statistical significance due to small sample sizes, the researchers found that the magnitude of reading impacts was larger for KIPP schools that offered pre-K than for those that did not, which suggests that “earlier and longer exposure to KIPP improves reading outcomes.”
Third, the study sought to determine how the size of KIPP’s impact changed over time for students who attended KIPP pre-K. To answer this question, researchers examined the impacts for the pre-K cohort in kindergarten and then again in second grade using the same academic tests. The sample was restricted to students who had test outcome data for both kindergarten and second grade. Researchers found that the impact on reading skills persisted over time, but impacts on reading comprehension dissipated by second grade.
Overall, it’s clear that the cumulative impacts of KIPP pre-K and elementary school are positive and substantial—the earlier students are enrolled in a KIPP school, the better. But the biggest takeaway extends beyond the performance of a single charter network. High-quality pre-K is good for kids, but high-quality pre-K plus a great elementary school experience could be the key to avoiding the fade out trend—and that’s why it’s so important that more charter schools be permitted to provide pre-K.
SOURCE: Virginia Knechtel, Thomas Coen, Pia Caronongan, Nickie Fung, Lisbeth Goble, “Pre-Kindergarten Impacts Over Time: An Analysis of KIPP Charter Schools,” Mathematica Policy Research (August 2017).
