Whose ChatGPT? Unveiling Real-World Educational Inequalities Introduced by Large Language Models

Linguistically disadvantaged students benefit more from using ChatGPT

Yu, R., Xu, Z., CH-Wang, S., & Arum, R. (2024). Whose ChatGPT? Unveiling Real-World Educational Inequalities Introduced by Large Language Models (No. arXiv:2410.22282). arXiv. https://doi.org/10.48550/arXiv.2410.22282

This was a great paper that was really poorly named. Many of us are searching for tools and techniques that can close gaps between groups of learners who have historically performed lower than other groups of learners. While most of the things people try in order to close this gap have no impact at all, the few things that do make a difference make a difference for all learners, sliding the gap to the right rather than narrowing it. This research shows ChatGPT having a differential benefit for lower performing learners, narrowing the gap.

The research looks at a large body of student writing from the “pre-LLM” era through the “post-LLM” era, “analyz[ing] 1,140,328 academic writing submissions from 16,791 college students across 2,391 courses between 2021 and 2024 at a public, minority-serving institution in the US,” and finds:

“Based on large-scale writing records from authentic learning environments, we uncover how the emergence of LLM tools altered writing quality and its disparities across students from different linguistic and socioeconomic backgrounds. We find that as the public became more familiar with LLM tools, the writing gaps between linguistically disadvantaged and advantaged students were increasingly narrower.”

“While all student groups had somewhat more proficient writing in the LLM era, linguistically disadvantaged students (from non-English-speaking countries or families, or with lower entering writing scores) had significantly larger improvement than their peers across all indicators of writing language proficiency.”

“This aligns with the emerging research on the potential of LLMs to empower disadvantaged learners especially in language-related tasks such as second language writing (Warschauer et al., 2023).”

Researchers also looked for differential gains among only the linguistically disadvantaged group, and found that the improvements in writing quality were “somewhat more concentrated on high-SES students” within the group which I suppose lead to the name of the paper. But “somewhat more concentrated,” really? Students from low-income families improved less than three one hundredths of a standard deviation less (0.025 standard deviations on average) than their higher-income peers improved. This difference was statistically significant, but was it practically significant? And was it really worth focusing the name of the paper on, overshadowing the other findings?