Operation and Reliability of AI Detectors

AI detectors, also known as ChatGPT checkers, are tools that provide a score indicating how likely it is that a text was written by an AI. These detectors analyze texts based on predictability and patterns characteristic of language models. They pay attention to the consistency of sentences and the predictable choice of words, but the scores they assign are not absolute proof of AI generation.

The operation of an AI detector is based on concepts such as perplexity and burstiness. Perplexity refers to how predictable the text is; a lower value indicates that the text is more likely to have been produced by AI. The concept of burstiness describes the variation in sentence length and structure: human writing styles are often more varied than those of AI. It is important to mention that these detectors do not make comparisons with a database as in plagiarism checks, but rather provide a probability percentage that the text was written by AI.

However, the reliability of AI detectors leaves much to be desired. OpenAI discontinued its own AI detector in July 2023, as it was correct in only 26 percent of cases. Research by Stanford showed that as much as 61 percent of authentic, human-written essays were incorrectly identified as AI-generated. In particular, texts that are simpler or show less variation in word choice are at greater risk of being categorized as AI text.

Regarding performance on Dutch texts, the situation is less favorable. Most AI detectors are primarily trained on English texts. Tools like Scribbr offer an AI checker, but the free version is only available for English texts. When evaluating texts with these tools, length is important: short texts often produce unreliable results. For individuals facing a high AI score for original texts, it is recommended to keep evidence of the writing process, such as notes and earlier versions, and not to consider the scores as definitive proof.

For more information on how AI-written texts are recognized and the applicability of these tools, you can consult the various sources.

Read the full article from AI Insider.