Are we closing the filter? Scientists tested how good AI is at copywriting
ChatGPT – the writer’s right hand or a serious competitor?
According to the new research artificial intelligence, in particular the chatbot ChatGPT, improves writing skills and significantly speeds up writing texts. AI is especially effective for authors whose initial skills are not too high. Smart machines bring their results closer to those of experienced users.
“This could mean that in the long term we will see a decrease in social inequality, as the results are equalized between different groups in terms of skill levels,” says Whitney Zhang, an MIT economics graduate student and co-author of the study.
After the launch of ChatGPT in November 2022, a lot of speculation arose about the impact of this tool on the labor market. A chatbot that quickly generates text of any complexity has become the subject of lively discussions. In response to these speculations, Zhang, along with colleague Shakked Noi, developed an experiment that helped to study the impact of a chatbot on employee productivity. The scientific study was published in the journal Science.
As part of the study, 453 professionals with higher education were involved, including marketers, grant writers and managers. They were asked to write press releases, brief reports, analysis plans, and other texts. After completing the first task, half of the participants signed up for ChatGPT and used it for another task.
The results were surprising: the group that used the chatbot completed the second task significantly faster, taking an average of 16 minutes. For those who did not use ChatGPT, it took about 27 minutes. Texts written using AI received 18% higher ratings from professionals compared to works written without the help of a chatbot.
The most impressive results were for the participants who received the lowest scores on the first task. Their scores increased by 1-2 points after using the chatbot.
“It creates a democratizing effect,” explains economist Robert Seamans of New York University’s Stern School of Business. “The most significant progress from the use of AI is observed among workers with less experience.”
However, according to Anton Korinek, an economist at the Brookings Institution think tank, ChatGPT could negate the need for certain skills and personnel. “Workers with analytical skills and excellent writing skills that a language model can perform will find their labor devalued.”
The participants in the experiment who used ChatGPT barely corrected the text generated by the AI. Small changes they made did not lead to better grades.
“Most jobs involve many different tasks, and writing lyrics is just one of them. This aspect can be improved with the help of language models, but all other tasks still require human participation, at least for today,” emphasizes Korinek.
Robert Seamans concludes: “Artificial intelligence is being used more and more like the Internet. Those who understand the technology and know how to work with it will win.”