At about the same time last year that the College Board was letting go of its previously-announced plans to add an “adversity index score” to its test result reporting to colleges, the journal Educational Policy published a research paper describing the results of an experiment which simulated a similar idea. Evidence indicates that backing down from the original idea was probably wise.
That research, conducted by a group of professors from the University of Michigan and the University of Iowa, utilizes a randomized-controlled trial using four simulated admissions packets to see if showing how students perform relative to their school and neighborhood peers influences colleges’ decisions to admit students, as well as whether including a personal essay that conveys “grit” influences admission decisions. The researchers were mainly interested in whether that additional information would increase the likelihood that applicants of low socioeconomic status (SES) would be admitted.
The sample was drawn from a pool of admission officers working at selective colleges and universities, defined as the top three tiers in Barron’s competitive rankings. Recruitment emails were sent to 960 admissions officers who met this criteria; 321 clicked on the online invitation to participate. Two experiments were tested—one where treatment participants’ applications all included median SAT test scores for the applicant’s high school and zip code (very similar to the College Board’s previously stated goal to “place students’ SAT scores in the context of their socioeconomic advantages or disadvantages.”), and the control condition where admission officers saw applicants’ own SAT test scores without the additional context. Applicant characteristics were kept constant (i.e., a white female majoring in the social sciences from the northeast). Officers were supplied with applications tailored to their university tier (1–3). The simulated files for low-SES students had more modest SAT scores assigned relative to high-SES applicants, as they tend to have lower scores on average, and also had raw SAT scores that exceeded their median high school and zip code test scores.
The second experiment tested the impact of a personal essay that portrayed grit. Each participant reviewed two applications (one high SES and one low SES) with “grit essays” and two applications (high and low SES again) with a standard personal statement. One grit essay described what it was like growing up in a home without a father and with a mother who worked long hours to support her family; the second was about growing up as the daughter of an alcoholic. Both conveyed how the student had overcome adversity, highlighting the skills she had developed because of her experiences. The researchers used a variety of analytic methods, including multilevel logistic regression with acceptance recommendations nested within participants.
Their key finding was that admissions officers in the treatment condition were significantly more likely to accept both the low SES and high SES applicants than were those officers not provided with the additional context around SAT scores. In fact, the magnitude of the marginal effects (ranging from 9–11 percentage points) was roughly the same across applicants, such that the low-SES students did not differentially benefit. So even though the intervention may have helped low-SES students be admitted, it did not level the playing field.
Moreover, admissions officers were not more likely to accept the applicants who conveyed grit in their essays. Overall, those essays were not significantly related to the probability of acceptance. The research team suggests that perhaps first-person essays simply aren’t the best way to convey non-cognitive traits such as perseverance; they posit that letters of recommendation or a record of extracurricular activities may be more informative in that regard.
Efforts to identify high school graduates coming from poverty who could succeed and thrive in selective institutions are laudable. But providing comparative peer data alone won’t do it. The means to make more of these youngsters stand out stem largely from a high-quality K–12 education: high expectations, terrific instruction, and rigorous content delivered from the earliest grades possible.
SOURCE: Michael N. Bastedo, et. al., “Contextualizing the SAT: Experimental Evidence on College Admission Recommendations for Low-SES Applicants,” Educational Policy (September 2019).
