Study finds that LLM use leads to ‘likely decrease’ in learning skills

Scientists from the Massachusetts Institute of Technology have determined that using large language models leads to a “likely decrease” in learning skills.

Time reported the research team’s findings on Tuesday. In a preprint paper, the scientists detailed that the new data is the fruit of a months-long study. 

The project saw the MIT team ask 54 individuals from the Boston area to write a short essay in 20 minutes. The participants were divided into three groups. The first group wrote the essay without any outside assistance, the second had access to search engines and the third used ChatGPT.

The researchers repeated the exercise four times. On the fourth occasion, the participants who had access to ChatGPT switched roles with the ones who wrote the essays on their own. The fourth test was conducted four months after the first.

“While the benefits were initially apparent, as we demonstrated over the course of 4 months, the LLM group’s participants performed worse than their counterparts in the Brain-only group at all levels: neural, linguistic, scoring,” the researchers wrote in the paper.

The MIT team collected data from the tests with the help of electroencephalography, or EEG, headsets worn by the participants. Such devices use electrodes to measure the wearer’s brain activity. The researchers also posed a set of questions to the participants in order to round out the data gathered by the headsets.

EGG devices use a metric called dDTF, or Dynamic Directed Transfer Function, connectivity to measure the wearer’s cognitive load. This metric describes the intensity with which different brain regions interact with one another. According to the researchers, the participants who used LLMs exhibited up to 55% lower dDTF connectivity while writing essays than the ones who didn’t have access to ChatGPT.

The LLM-equipped group also exhibited lower frontal-midline theta activity during the writing exercise. Frontal-midline theta brain waves are associated with cognitive activities that involve focused attention. “Theta connections that were prominent in Brain-only group were relatively weak or absent in the LLM group,” the researchers found.

In a subsequent phase of the project, the MIT team asked the study participants to quote from their essays. The group that used LLMs didn’t perform the task as well as the other two. Furthermore, the participants in that group reported lower “perceived ownership” of the essays they produced.

“We demonstrate the pressing matter of a likely decrease in learning skills based on the
results of our study,” the researchers wrote.  “These findings support an educational model that delays AI integration until learners have engaged in sufficient self-driven cognitive effort. Such an approach may promote both immediate tool efficacy and lasting cognitive autonomy.”

Photo: Focal Foto/Flickr