AI on the Brain

A recent MIT paper, "Your Brain on ChatGPT: Accumulation of Cognitive Debt when Using an AI Assistant for Essay Writing Task" observed that subjects using Large Language Models (LLMs) to write an essay in a lab engaged their brains less than those writing without AI assistance. The authors predict a "likely decrease in learning skills, as LLMs reduce the cognitive effort required for writing, and as with any task, less effort often translates to poorer long-term knowledge retention. Media outlets quickly sensationalized the findings with headlines such as "AI will make you a dumber writer, says science" and "ChatGPT could be silently rewiring your brain as experts urge caution for long-term use. Additionally, LLMs often use templates in their answer structure and vocabulary, which predictably leads to more homogenous outputs than without AI assistance. While the experimental methods were not entirely rigorous, which limits definitive conclusions, the research suggests that using an LLM can reduce the cognitive load of writing, albeit with expected trade-offs in information retention.

The study involved 54 participants aged 18-39, from MIT, Wellesley, Harvard, Tufts, and Northeastern. Over three sessions spanning three months, participants were given 20 minutes to write an essay using a text editor on an SAT topic of their choice. They were randomly assigned to one of three groups. The "LLM" group exclusively used OpenAI's GPT-4o for writing, with no access to web browsers. The "Search Engine" group could use browsers but not LLMs. The "Brain-only" group had no access to either LLMs or browsers. One-third of the participants completed a fourth session in which the "LLM" and "Brain-only" groups switched roles, allowing them to use an LLM, becoming the "Brain-to-LLM" and "LLM-to-Brain" groups, respectively. Throughout these tasks, participants' cognitive engagement and load were monitored using EEG. The LLM-to-Brain group showed less brain activity, and the Brain-to-LLM group showed more recall and brain activity. The conclusions about brain atrophy are based on this single finding of eighteen people (nine in each group).

In all measures, including algorithmic and human scoring, vocabulary variety, satisfaction with their work, a sense of ownership over their essays, and the ability to reflect on their work, the Brain-only group outperformed the others. The LLM group performed the worst, with the Search Engine group falling somewhere in the middle. The LLM group exhibited the least neural activity, while the Brain-only group showed the most, mirroring the findings. The essays produced by the LLM group were also the most structurally homogenous and were heavily influenced by typical LLM vocabulary words. Participants in the LLM group also found it more challenging to recall specific quotes from their essays, especially in the initial session when they didn’t anticipate the task. Furthermore, their understanding of the essay topics was shallower compared to the other groups.

However, the paper faces several methodological limitations, weakening the conclusions and generalizability. It hasn't been peer-reviewed and relies on a small, biased sample of highly educated individuals. It involved only a single task type (SAT-style essays) and a small participant pool that had been writing similar essays for a large portion of their lives, presumably, very well, as students at elite private schools.  Additionally, the lab setting makes the situation artificial, particularly with the EEG constantly reminding people they were being monitored. Given the anonymity and lack of reward for creativity and deep knowledge, subjects were probably more inclined to take shortcuts and use the LLM as a crutch. The 20-minute time limit, which differs considerably from the more in-depth essays typically required in academia, further encouraged these shortcuts.

Furthermore, the paper lacked detailed instructions on how participants should use the LLM. Because there was no guidance around LLM usage (perhaps deliberately), post-assessment interviews revealed wide variability in LLM usage, ranging from basic ideation to generating a large portion of the essays. Preventing LLM users from using a browser meant they couldn't follow best practices for LLM usage in writing, such as cross-referencing information, checking sources, and seeking additional information from the sources the LLM used. This inconsistency in LLM usage likely introduced significant variability into the LLM group's results, making it difficult to pinpoint the exact impact of different types of LLM usage. However, even if future studies corroborate these findings with more rigorous methods, they don't necessarily signal brain atrophy. Instead, they indicate a shift in how people approach writing opinion papers for academic assignments, and may not generalize to other cognitive skills.

The article asserts that “When individuals fail to critically engage with a subject, their writing might become biased and superficial. This pattern reflects the accumulation of cognitive debt, a condition in which repeated reliance on external systems like LLMs replaces the effortful cognitive processes required for independent thinking.“ This outcome is expected because when writers rely heavily on other sources for content creation, they reflect the biases of the material they have used and cannot delve deeper into a topic. This result aligns with the Levels of Processing Theory, which states that the more someone engages with content, understanding its meaning, context, and connections, the more effectively it's committed to memory. Consequently, intensive processing involves greater brain engagement. Therefore, it's hardly surprising that relying solely on an LLM for writing requires less cognitive effort than writing from scratch or using a web search for research (unless the latter is plagiarizing from those sources). Thus, it's logical that the Brain-only group could more easily quote their own work because they constructed every sentence themselves. The other Search Engine and Brain-only groups likely engaged in a more editorial process, synthesizing and integrating external inspiration. The Brain-only group's higher satisfaction with their work is also consistent with the “IKEA Effect,” a phenomenon where individuals develop a greater attachment to things they've invested more effort into.

It's also unsurprising that the LLM group's essays lacked variety when compared to the other groups. For a given essay topic, the prompts fed to the LLM were likely quite similar. LLMs inherently tend towards homogeneity in their answer structures, sentence patterns, and vocabulary, essentially providing a template. Humans are highly suggestible and will tend to reflect back structure and vocabulary subconsciously. With only 20 minutes to complete the essay, there simply wasn't enough time to significantly enrich or diversify the initial LLM-generated content, leading to a homogeneity in both ontology and semantics.

While the results are ambiguous due to the small sample size of nine participants in the LLM-to-Brain group, there is some evidence that prior experience with the LLM helped the subjects complete a “Brain-only” essay. "While these LLM-to-Brain participants demonstrated substantial improvements over 'initial' performance (Session 1) of the Brain-only group, achieving significantly higher connectivity across frequency bands, they consistently underperformed relative to Session 2 of the Brain-only group, and failed to develop the consolidation networks present in Session 3 of the Brain-only group." While it's ambiguous whether their performance should be benchmarked against the first or a later session of the "Brain-only" group, it's clear they initially performed better than those without previous LLM exposure.  It's also possible that the LLM group, if allowed to use a search engine to verify information and delve deeper, might have engaged their brains more than the "Brain-only" group. LLMs can spark ideas that can then be more thoroughly researched using web searches. 

This mirrors the difference between providing just an encyclopedia versus providing an encyclopedia along with access to all its source material. The study's conclusions, however, are strictly limited to the scenario of exclusive LLM use, which is not representative of how LLMs are typically integrated into most essay writing processes. There certainly is no overwhelming evidence of cognitive atrophy.

As expected, LLMs simplify writing, and easier tasks generally lead to less retention of information. Consequently, students who rely solely on LLMs for writing likely develop a shallower understanding of the material. Teachers who use essays to impart content must adapt their assignments. They may need to explicitly prohibit LLM usage or modify assignments to ensure that students actively understand and process the material. For instance, requiring deeper, more analytical essays could encourage greater cognitive engagement. 

When new technologies emerge, we usually welcome their ability to reduce effort. The results in this paper are not surprising; rather, they are how many technologies work, receiving some of the cognitive load necessary to perform a task. Yet, we might also lament the decline of skills that the technology replaces. For instance, we appreciate the widespread availability, portability, and variety of music that are available today. However, we may also overlook the societal value of performing music, as fewer individuals develop the associated skills. 

Whether this shift is a net positive depends on how LLMs are utilized. For example, the compass replaced the complex skill of celestial and landmark navigation, and then GPS largely replaced the compass. Each replacement didn't just simplify tasks; they also enabled humanity to achieve far more, much faster. Navigators, no longer "hugging the coasts," could venture further. It remains to be seen if LLMs will unlock similar advancements in human potential, leaving “brain space” for even more exciting endeavors.

Rachel Lomasky is Chief Data Scientist at Flux, a company that helps organizations do responsible AI.