A team of researchers at the University of Pennsylvania School of Engineering and Applied Science are developing ways to mitigate the risks of artificial intelligence’s role in reinforcing social biases, committing fraud and identity theft, generating fake news, spreading misinformation, and more. In a peer-reviewed paper presented at the February 2023 Association for the Advancement of Artificial Intelligence meeting, the researchers demonstrate that people can learn to spot the difference between machine-generated and human-written text.
Through the study, the researchers prove that machines make distinctive common sense, relevance, reasoning, and logical errors, which humans can learn to detect. The study uses data collected from Real or Fake Text?, a web-based training game.
The game transforms the standard experimental method for detection studies into a more accurate recreation of how people use AI to generate text. The Penn model begins by showing the player human-written text. Each example then transitions into generated text, asking participants to mark where they believe this transition begins. Trainees identify and describe the features of the text that indicate an error and receive a score. The study results show that participants scored significantly better than random chance, providing evidence that AI-created text is, to some extent, detectable.
However, the study also outlines a promising and exciting future for our relationship with this technology. The researchers believe that once we can harness our optimism about AI text generators, we can devote attention to these tools’ capacity to help us write more imaginative and exciting texts.
