SINGAPORE - Students love it, but it is the nightmare of teachers. ChatGPT, an artificial intelligence (AI) chatbot, has been thrust into the spotlight since its latest version was released in November 2022.
The chatbot, developed by well-funded United States start-up OpenAI, is able to generate coherent essays and solve word-based maths problems.
The race to beat AI-assisted plagiarism and cheating is on. The Straits Times looks at five plagiarism detection tools.
1. GPTZero
Mr Edward Tien, a senior at Princeton University, spent his winter break in 2022 and New Year’s Eve developing GPTZero, an app to quickly tell if an essay was written by a human or ChatGPT.
The 22-year-old, who is pursuing a double major in computer science and journalism, uploaded a beta version of the tool on app-hosting platform Streamlit in the beginning of January to help educators detect “aIgiarism” (AI-assisted plagiarism).
“We’ve tested the beta with a dataset of BBC news articles, and GPT-generated articles (with) the same headlines. The false positive rate is less than 2 per cent,” he told The Straits Times.
This means up to two out of every 100 articles are incorrectly classified as ChatGPT-generated when they are not.
“We’re working on several model improvements to make GPTZero highly accurate and easy to use for teachers,” he said, pointing to a sign-up list for the upcoming launch of GPTZero.me
GPTZero uses two different metrics to assess whether content is written by a bot: perplexity and burstiness. Perplexity is a measurement of randomness in a sentence, while burstiness measures overall randomness for all the sentences in a text, factoring in length.
Text placed into the app will be assigned a number for both metrics. The lower the number, the more likely the content is created by a bot. Human writing tends to have sentences that vary in complexity, whereas bots tend to be more consistent. Texts with higher perplexities are more chaotic and not likely to be generated by language models.
Mr Tien said he has since been contacted by multiple venture capital firms, including American VCs Sequoia Capital and Andreessen Horowitz.
2. GLTR (Giant Language Model Test Room)
Researchers from the MIT-IBM Watson AI Lab and the Harvard Natural Language Processing Group came together to create GLTR (Giant Language Model Test Room), an algorithm to detect if text was written by a bot when OpenAI released GPT-2 in 2019.
The researchers were concerned about the potential abuse of language models that have the potential to generate textual output that is indistinguishable from human-written text such as fake reviews, comments or news articles to influence public opinion.
The algorithm uses the “it takes one to know one” method to predict if a body of text is written by a bot. Typically, bots – unlike human writers – are less likely to select unpredictable words. GLTR is limited in that it has not been tested on a large scale.
3. GPT-2 Output Detector
Since OpenAI released ChatGPT’s predecessor GPT-2 in 2019, the start-up has also demonstrated that it can create a bot that is capable of detecting machine-generated text with the release of GPT-2 Output Detector.
Users simply paste text into a box to immediately see on a scale how likely the text was written by AI.
According to research from OpenAI, the tool has a detection rate of about 95 per cent for machine-generated text using GPT-2. Even so, the accuracy rate is not high enough for standalone detection, according to OpenAI, but needs to be paired with “metadata-based approaches, human judgment and public education to be more effective”.
4. Watermarking (coming soon)
OpenAI is also working on watermarking all ChatGPT text to tackle potential abuses, including plagiarism. In a blog post in November 2022, OpenAI guest researcher Scott Aaronson wrote: “We want it to be much harder to take a GPT output and pass it off as if it came from a human.”
This could be helpful for preventing academic plagiarism as well as mass generation of propaganda. “You know, spamming every blog with seemingly on-topic comments supporting Russia’s invasion of Ukraine, without even a building full of trolls in Moscow, or impersonating someone’s writing style in order to incriminate them,” he wrote.
The start-up has a working prototype of the watermarking scheme, built by OpenAI engineer Hendrik Kirchner. Inserting or deleting a few words, or rearranging the order of some sentences, will not remove the watermarking signal.
However, people have found other ways to beat it. On TikTok, some users are already sharing workarounds by pasting AI-written text into another AI-powered paraphrasing tool, QuillBot. OpenAI is working on improving the watermarking prototype to address such workarounds.
5. Turnitin Originality for ChatGPT (coming soon)
Professors and educators currently use Turnitin Originality, a tool created by privately held American firm Turnitin, to ensure students submit original work. Like a Web crawler, Turnitin finds similarities in sentences and paragraphs against an online or offline database of content, and provides a similarity score. A returned percentage of below 15 per cent would typically indicate that copying has not occurred.
The company, which works with thousands of universities and high schools, plans to roll out a service in 2023 that can accurately tell whether ChatGPT has done a student’s assignment for them, its chief executive officer Chris Caren wrote in a blog post in December 2022. The tool is based on the understanding that AI models pick the most probable word in the most probable place. Humans, on the other hand, are idiosyncratic.
