The use of technology in education—in place before the pandemic but increased in magnitude and ubiquity since 2020—is drawing increasing scrutiny from many sides. The villagers are lighting up their torches and coming en masse for cellphones, online learning platforms, Chromebook-based assignments, Smartboards, and more. A trio of researchers from the University of Michigan suggest another log to stack on the pyre: electronic grade books. Their new research shows, among other things, how even the bedrock of the English alphabet can be weaponized when brought in contact with the white heat of technology.
Specifically, the researchers look at more than 31 million grading records submitted to a learning management system called Canvas by graders from an anonymous large public university in the United States between 2014 and 2022. Canvas is the most popular system of its type, in use at 32 percent of all higher education institutions in the United States and Canada in 2020. To avoid biasing the analysis, the researchers include only human-graded assignments, removing assignments and courses that have either massive numbers of grades or an extremely small number of grades, as well as assignments graded offline and only submitted electronically. The final data set still contains a whopping 31,048,479 electronic grades covering 851,582 assignments, 139,425 students, and 21,119 graders. Data include both numerical values and textual values (comments from the grader). Timestamps allow the researchers to determine both the order in which assignments are submitted and in which they are graded. Additionally, the Canvas platform includes any comments entered by students in response to the grades after they are made available.
The bedrock finding is that electronic grading of student work appears to be a slog for most humans cursed with the task, and that using a system like Canvas—designed to ease the burden—only compounds the problems. This analysis indicates that negative impacts of electronic grading can accrue to specific individuals as a matter of course. Students whose assignments were graded later in the process—however that process was sequenced—received lower marks and more negative comments than their peers. Surname alphabetical order grading, the default setting for Canvas and comprising over 40 percent of the submissions analyzed, tells the tale clearly. Students whose surnames started with A, B, C, D, or E received a 0.3-point higher grade (out of 100) than did students with surnames later in the alphabet. Likewise, students with surnames W through Z received a 0.3-point lower grade than their earlier-in-the-alphabet peers—creating a 0.6-point gap between the Abdullahis and the Zimmermans of the world. Robustness checks among different graders, including a small group (about 5 percent) who happened to have graded in reverse alphabetical order and exhibited the same gap, confirmed this pattern. That might seem a small difference, but if it happens to the same students on every assignment in multiple classes over multiple years, the small gap could grow into a much larger one.
Grader comments on assignments evaluated later in the sequence were found to be more negative and less polite (the handful of examples included in the report would be truly disheartening for their unfortunate recipients) than those given earlier in the sequence. Grading quality seemed to deteriorate down the sequence, as well. Students with assignments graded later were significantly more likely to log questions and challenges about their marks in Canvas. Specifically, students graded between fiftieth and sixtieth in order are five times more likely to submit a regrade request compared with the first ten students graded, no matter in what order the grading is done.
While the surname alphabetical grading order illustrates sequential bias most clearly and sensationally, the same pattern was also observed across assignments graded in quasi-random order. The more assignments that must be graded, the more likely students at the end of the sequence are to be impacted. And in the case of learning management technology, that largely means students whose surnames are late in the alphabet. As noted, Canvas (and its three closest competitors) defaults to alphabetical order and must be manually changed by graders or institutions to avoid this. Together, these four systems accounted for over 90 percent of the U.S. and Canadian market at the end of 2020, so the potential negative impact to the Whites, the Williamses, and the Youngs is huge.
The researchers end with three commonsense recommendations. First, creators of learning management systems should switch their products’ default from alphabetical to random order (with educational institutions doing so manually until that time), although that would only diffuse the sequential bias effect across more students. Second, graders should be trained on the nature of sequential bias and strategies to avoid it in their work, which still leaves the problem of the slog. Finally, to address the problem fully, institutions should limit the number of assignments evaluated by any one grader. Eliminating electronic submission and grading entirely—bound to be popular with many stakeholders looking to limit technology in education—may be the most obvious option not suggested here.
SOURCE: Zhihan (Helen) Wang, Jiaxin Pei, and Jun Li, “30 Million Canvas Grading Records Reveal Widespread Sequential Bias and System-Induced Surname Initial Disparity,” Management Science (March 2024).
