Subscribers to our podcast, the Education Gadfly Show, are treated weekly to the Research Minute, a segment where I highlight a recent rigorous, useful, or intriguing education-related study (ideally all three).
I cannot pretend to have read every study that fits that description. But here is my countdown of the top five studies (well, six) that I reviewed this past year.
Honorable Mention: Mahsi’choo for the Info! Molly of Denali teaches children about informational text, by Ready To Learn Research (reviewed on the show in July).
A cool little study by the Educational Development Center and SRI International sheds light on how we can use public media to improve childhood literacy. Specifically, the research team conducted two randomized control trials of the PBS Kids show Molly of Denali, an award-winning series designed to teach children ages three to eight how to use informational text. The show is distributed nationally and features an Alaska Native female lead character who, along with her friends, models the use and creation of informational text to solve authentic problems in their everyday life in our largest state. They rely on sources like field guides, instruction manuals, maps, weather reports, and even homemade recipes (including one for mosquito repellant).
The key finding is that the Molly of Denali resources improved children’s ability to use informational text to solve real-world problems. This impact was more pronounced for the youngest kids and those who used the resources for longer periods. The overall effect size was small, but analysts say the impacts are notable for a nine-week intervention in which children interacted with the materials roughly an hour a week, while “other education interventions rely on intensive year-long curricula to achieve similar impacts.” I liked that intentionally-developed digital content was made to strengthen particular skills and that this high-quality public TV show was available to all kids, even those without broadband or computers.
Number Five: Accountability-Driven School Reform: Are there unintended effects on younger children in untested grades?, by Gary T. Henry, Shelby M. McNeill, and Erica Harbatkin (reviewed on the show in February).
February brought a study that examined the impact of test-based accountability on young children. It looked at an initiative conducted from 2015 to 2017 in North Carolina to turn around its lowest performing schools. Did this effort impact the outcomes of young children, as well as the assignment of teachers into untested grades?
Teachers in K–2, which are untested grades in North Carolina, have effectiveness scores calculated by their students’ text reading comprehension (TRC) scores. Researchers leveraged these scores in a regression continuity design, estimating the effect of being just below the cut-off score for assignment to the turnaround initiative.
Initial results saw negative effects on early literacy and reading in the first year of the intervention, but a rebound soon followed, showing positive effects in the second year in the range of 0.08 to 0.09 standard deviations for both subjects. Analysts found that low-effectiveness teachers in tested grades across the full sample were more likely to be reassigned to untested early grades in 2016 but no more often than in comparison schools. The bottom line for me was that accountability-driven reform may not be leading to huge gains, but it also isn’t doing youngsters harm.
Number Four: The evolution of gender gaps in numeracy and literacy between childhood and young adulthood, by Francesca Borgonovi, Alvaro Choi, and Marco Paccagnella (reviewed on the show in April).
Research indicates that gaps in numeracy and literacy skew along gender lines, with boys tending to outperform girls in numeracy and girls tending to outperform boys in literacy. But we know little about how these gaps evolve as children grow up. An international study in the Economics of Education Review aimed to chart their evolution between childhood and young adulthood.
Analysts combined data from existing cross-sectional large-scale assessments that contain representative samples of the same birth cohort at different points in time in up to eleven countries. Thus they could track the evolution of gender gaps for a single cohort of students that participated in different waves of different assessments from ages ten to twenty-seven. They pull data from TIMSS, PIRLS, PISA, and PIACC test administrations.
The key finding is that, in a large majority of countries, the gender gap in numeracy in favor of boys tends to be linear, growing as students age. It’s particularly pronounced after they exit compulsory schooling and enter post-secondary education or the labor market. By contrast, the gender gap in literacy is widest during the teenage years. Specifically, girls have a large advantage at ages nine and ten, which grows larger by ages fifteen and sixteen. At ages twenty-six and twenty-seven, however, the advantage shrinks to essentially zero, as young men then have a surprising (but non-statistically-significant) advantage of 13 percent of a standard deviation. Most of these findings also hold true in the United States, although numeracy gaps in favor of boys are even larger.
Interestingly, the analysts posit that literacy skills are more universal, so perhaps men are able to develop them on the job and catch up with women after formal education has ended. Their study raises the question of how to increase the math competency of girls in the elementary years, so that we prevent gaps from opening in the first place. But there was good news, too, in that adult males can make up for lost ground on the literacy front.
Number Three: Why Does Education Increase Voting? Evidence from Boston’s Charter Schools, by Sarah Cohodes and James Feigenbaum (reviewed on the show in September).
This NBER paper leverages the natural experiment generated by school choice lotteries to analyze the impacts of charter school attendance on academic outcomes, which has been examined before, but this study adds the newer wrinkle of citizenship effects (a.k.a voting outcomes).
It includes data from applicants to all twelve Boston charters with lottery records from the classes of 2006 to 2017 and matches student data to voter files in Massachusetts. The voting data include party registration and election turnout in elections from 2008 to 2018. Analysts measure voter participation as “ever voted,” as well as participation in the first presidential election after a student turns eighteen.
Results show that Boston charters continue to show large gains in scores on state exams, AP test taking and scores, SAT scores, and college enrollment. Next, analysts find that charter school attendance makes little difference in the likelihood of registering to vote, with many students—around 45 percent—registering to vote by their nineteenth birthday regardless of their lottery status. But charter attendance does boost the rate at which students ever vote, from about 44 to 50 percent, which is a 6-percentage-point increase—mainly driven by presidential election participation.
Charter attendance does not impact voting in non-presidential general elections or in primaries. Importantly, the researchers find that the voting gains come almost entirely from a boost in girls voting in their first possible presidential election. This all sounds positive for girls, but we can’t afford to neglect the plight of our young boys.
Number Two: To What Extent Does In-Person Schooling Contribute to the Spread of COVID-19? Evidence from Michigan and Washington, by Dan Goldhaber, Scott A. Imberman, Katharine O. Strunk, Bryant Hopkins, Nate Brown, Erica Harbatkin, and Tara Kilbride (reviewed on the show in February).
A septet of superstar analysts brought us this timely and actionable research related to the ongoing pandemic, wherein they examined the extent to which in-person schooling contributed to the spread of Covid-19.
They analyzed data on reported Covid cases in Michigan and Washington, as reported by the CDC and the respective state health agencies, as well as data on district-level instructional modality—in-person, hybrid, or remote—collected via monthly surveys administered to school district leaders. Basically, they knew the approximate share of students receiving education in each of these different modes. They also included controls designed to account for non-schooling risk factors for Covid spread—including the share of individuals who mask, the share who stay at home, the county unemployment rate, the share of the population that attend county public schools, the share who are age sixty-five or older, the share that live in a nursing home, and so on.
The results show that in-person schooling is not associated with increased spread of Covid—at least in counties reporting low levels of pre-existing Covid cases overall. But increased Covid-19 spread does occur relative to in-person learning in areas with moderate to high pre-existing Covid rates.
Still, the estimated effect of in-person schooling on Covid spread depends greatly on what students and staff in the counterfactual situation (no in-person schooling) are doing at home. In other words, we can’t assume fully-remote learning is safer, since some remote families are participating in learning pods, others may be hiring babysitters and nannies who watch several kids, or families may be involving kids in other social activities because they are missing school. These non-school activities could serve to expose kids and adults to Covid as a direct result of imposing a remote learning modality meant to keep them safe.
At the time, their finding that in-person schooling has a significant impact only at higher pre-existing levels of community spread was commonsensical and in line with other guidance. Their study added to our growing knowledge of how to approach schooling in highly uncertain times.
Number One: The Revealed Preferences for School Reopening: Evidence from public-school disenrollment, By Thomas S. Dee, Elizabeth Huffaker, Cheryl Philips, and Eric Sagara (reviewed on the show in August).
The most consequential report I reviewed this year was also pandemic-related. It was published in August and received quite a bit of attention. The authors began by noting that, during the first full school year after the onset of the Covid-19 pandemic, enrollment in U.S. public schools fell by about 1.1 million students, or 2 percent. They examined how school reopening decisions affected those declines.
They used federal and state data to gather district enrollment counts from 2015–16 through 2020–21. They also collected district opening plans as aggregated by Burbio and tracked instructional mode status for roughly 1,200 districts at various points in time by school and grade level. The sample encompassed 35 percent of all public school students. Analysts tracked Covid-19 prevalence at the county level and state decisions around Covid-related restrictions, such as stay-at-home orders and public transportation limitations, which could influence schools’ instructional mode and parents’ perceptions of the Covid risk.
The headline finding: The decision to offer remote-only instruction in fall 2020 contributed materially to disenrollment from public schools. Specifically, the study found that offering remote-only instruction exacerbated disenrollment by 42 percent relative to in-person instruction, while hybrid instruction had small and statistically insignificant effects. Yet roughly 57 percent of students faced remote-only instruction as of November 2020.
That decline implies that public schools lost roughly 300,000 K–12 students as a result of those decisions. Which suggests that widespread adoption of remote-only instruction explains roughly a quarter of the disenrollment from public schools. The researchers also found that the effects of remote-only instruction on the decline in enrollment were particularly concentrated in the kindergarten and elementary grades. Remote instruction did not appear to influence middle or high school enrollment—or contribute to dropout behavior—nor did hybrid instruction have an impact either way.
Most of us had concerns early on that remote instruction would portend adverse effects for kids. This study quantified those unfortunate impacts for young learners, though we don’t know the extent to which they transferred to private or home schools (or delayed kindergarten entry). Thankfully, the enrollment repercussions of remote-only schooling appear to have been minimal for the older kids. While we’re grateful for timely pandemic-related research like these latter two studies, here’s hoping that we have less need for them in 2022.
There you have the top
5 6 research publications I reviewed this year. Merry Christmas, Happy Holidays, and Happy New Year to all of our listeners out there! The Research Minute will be back again on January 5 with more of the best studies. Don’t miss it!