This new study, the product of a partnership between District of Columbia Public Schools (DCPS) and researchers at New York University and the University of Maryland (including Dr. June Ahn, author of our recent report Enrollment and Achievement in Ohio's Virtual Charter Schools), examines how students’ use of educational software affects their achievement.
In 2012, DCPS began to implement a web-based mathematics program called “First in Math” (FIM) for students in grades K–8. The initiative consisted of games centered on basic computational skills and concepts like fractions or decimals. The authors examine student-level usage data, including how much time students spent on the FIM system, which modules they completed, and what achievements (like points, collecting “badges,” or unlocking bonus games) they earned at various points in the school year. That information was combined with student-level data, such as gender, English language learner status, special education status, race, grade level, and achievement on the mathematics component of the DC-Comprehensive Assessment System (DC-CAS). The final sample included approximately 9,200 students in Grades 4–8 during the 2012–13 school year.
The analysis reveals some intriguing findings. Time spent using FIM had a small but significant positive relationship with performance on standardized mathematics assessments, even controlling for factors such as prior academic achievement, English language learner status, and special education designation.
Students in lower-achieving schools used FIM approximately twenty-five minutes more per year than students in high-achieving schools, suggesting that they were more likely to benefit from the program. Yet the lower-achieving the student was, the less time they spent using FIM.
Usage was related to other student characteristics, too. Students of color spent more time using FIM than white students—approximately three hours more per year for black and Hispanic students, and four hours more for Asian students. And compared to their peers, female and special education students used the program about thirty-five minutes less, and English language learners about thirty-five minutes more.
The authors do, however, admit limitations to both their methodology and the program itself. FIM consists of largely rote practice tasks but does not teach deeper conceptual understanding. This may mask the effects of teachers or peers on achievement, or the impacts of additional programs that students may be participating in alongside FIM (authors were limited in what they could control at the school level). Authors concede that there is merely correlation between program usage and student achievement—not causation. While they show a linear relationship between time-on-task and the amount of stickers/badges earned (indicating progress), detailed information about how students spend their time-on-task is not available.
Nevertheless, the results suggest that even a modest use of 10–20 hours over the course of the school year (15–30 minutes per week for forty weeks) may have a modest but significant impact on math achievement. The study is also a testament to the benefits of researcher-practitioner partnerships to effectively share data and evaluate programs. Education software developers and teachers must continue to think of innovative ways to make their offerings appealing to reluctant students who could most benefit from the supplemental tools.
SOURCE: June Ahn, Austin Beck, John Rice, and Michelle Foster, “Exploring Issues of Implementation, Equity, and Student Achievement With Educational Software in the DC Public Schools,” AERA Open (October – December 2016).
