AI Detectors Fail to Accurately Identify Human-Written Text, Raising Concerns About Reliability

Wait, What? The Bible, Bhagavad Gita, and Preamble are All AI Generated?

So, the question is which AI detector is accurate in today's age. As writers, our early fears revolved around plagiarism checkers, making sure that every word we wrote was authentically ours. But now, with AI in the picture, the challenge has shifted. Companies and clients aren't just worried about the content being original anymore; they're extremely wary of AI-generated writing. Fair enough, there's always a demand for human-crafted stories. But how reliable are these AI detectors in proving the same? Lately, many freelance writers have taken to LinkedIn, frustrated by the inaccurate results of these tools. One writer shared his experience on the platform and said, "ZeroGPT and Copyleaks are scamming people. No AI detector is accurate." It's a conversation that's only getting started. If we were to trust these tools, we'd believe that even our Holy Books are AI-generated! A LinkedIn user posted that the Bible is "97% AI-generated", but considering that the earliest writings in the Bible date back to around 1400 BC, written by communities of scribes, the tools' accuracy is a question. Curious to know how the tools work, we added the Bhagavad Gita content, and here's the result. So, the Bhagavad Gita, part of the Mahabharata, written at some point between 400 BCE and 200 CE by the sage Ved Vyasa is also credited to AI. There's more, even the Preamble of the Indian Constitution is supposedly AI-generated, according to these flawed AI detectors. While AI detectors can be considered a helpful starting point for assessing content authenticity, they can't guarantee 100% accuracy. This is because these tools, still in their infancy, are evolving to match the sophistication of modern AI writing. Each detector varies in effectiveness because they rely on different training datasets. Also, the line between AI-generated and human-written content is becoming increasingly blurred. Advanced AI writing tools now produce content that closely resembles human writing, making it a task for detectors to distinguish between the two. So, to truly understand the reliability of these tools, it's important to know how they operate. The detectors are typically trained on datasets containing both human and AI-generated text. By analysing this data, they identify specific characteristics that are more likely to appear in AI-written content. Experts noted that the two major traits these tools focus on are perplexity and burstiness. Perplexity measures the unpredictability of the content. AI-generated text usually has lower perplexity, making it more predictable, while human writing tends to be more varied and less predictable. Burstiness refers to the variation in sentence length and structure. AI content tends to be more uniform and steady while human writing often exhibits a mix of long and short sentences. Other qualities that detectors flag as AI include the repetition of words, repetitive sentence structures, a generic tone, and more. There are a few reasons why clients and companies are very particular about their content. Data suggests that the entire process - from writing to publication - can be completed in as little as 16 minutes with AI, compared to an average of 69 minutes for humans. Despite this efficiency, the question of trust remains significant for organisations and customers. According to the Hootsuite Social Media Consumer 2024 Survey, 62% of consumers are less likely to engage with or trust content if they know it is AI-generated content. It further reflects on the traffic or engagement of the content too. Neil Patel, the co-founder at Neil Patel Digital, noted, "If everyone uses AI to create content, and if AI uses data already available to develop these projects, we won't have anything new and authentic anymore." It appears that only marketing professionals are confident in AI, while consumers remain sceptical. Freelance writers have a few suggestions for the clients. According to them, there must be an experienced content editor or manager to run through the content and decide if it reads 'AI-written', suggest the changes, tweak, if needed, and then push it through the AI detectors (preferably multiple, to compare the results and conclude whether it's human or AI-written). There is a worry that AI-generated content won't rank on Google. It's important to note that mobile disparities, website quality issues, accidental content alterations, and technical SEO challenges can also negatively impact traffic. So, it's not just AI-generated content that's sending your views crashing. As the entire AI system is new and evolving every day, the clients and companies need to exercise a bit of patience before rushing to conclusions and rejecting writers. The technology is still finding its feet, and quick judgments could mean overlooking real human talent in the process.

AI Detectors Claim the Declaration of Independence Was 98% AI-Generated - Decrypt

Watch out, cheaters -- AI detectors are here to catch you and your chatbot red-handed. Or, at least, that's what AI developers use as a selling point and want us to believe. When ChatGPT entered the cultural zeitgeist in 2022, teachers and professors balked at the surge in AI-generated research papers and homework. To curb the use of AI in the classroom, educators have been using AI detectors that claim to distinguish AI-written text from human-written text. But how accurate are these tools? According to Christopher Penn, Chief Data Scientist at Boston-based marketing analytics firm Trust Insights, "AI detectors are a joke." One AI detector he tested claimed that 97.75% of the preamble to the U.S. Declaration of Independence was AI-generated. "What led me to the testing of AI detectors was seeing colleagues battling back and forth, arguing about whether a piece of content was AI-generated," Penn told Decrypt. "I saw this on LinkedIn; some people were lobbing accusations against each other that so-and-so was being a lazy marketer, taking the easy way out, and just using AI." Fighting words? Perhaps. Said Penn: "We should probably test that to understand whether or not this is actually true." Penn decided to test several AI detectors using the Declaration of Independence, and was dismayed by what he found: "I think they're dangerous," he said of such detectors. "They are unsophisticated and harmful." "These tools are being used to do things like disqualify students, putting them on academic probation or suspension," he said. That's "a very high-risk application when, in the United States, a college education is tens of thousands of dollars a year." We decided to do a test of our own to see how these sites did. In the first, we used the same excerpt Penn used from the Declaration of Independence to determine which detectors erroneously believed the text was AI-generated. For the second test, we took an excerpt from E.M. Forrester's 1909 science fiction short story "The Machine Stops" and had ChatGPT rewrite it to see which detector identified the passage as AI written. Here are our results: Taking the same text Penn used, we compared several AI detectors: Grammarly, GPTZero, QuillBot, and ZeroGPT, the AI detector Penn showed in his LinkedIn post. "Grammarly continues to deepen its expertise in evaluating text originality and responsible AI use," a Grammarly spokesperson told Decrypt, pointing to a company post about its AI detection software. "We're adding AI detection to our originality features as part of our commitment to responsible AI use," the company said. "We're prioritizing giving our users, particularly students, as much transparent information as possible, even though the technology has inherent limitations." The Grammarly spokesperson also highlighted the company's latest update, Grammarly Authorship, a Google Chrome extension that enables users to demonstrate which parts of a document were human-created, AI-generated, or AI-edited. "We would recommend against using AI detection results to directly discipline students," GPTZero CTO Alex Cui told Decrypt. "I think it's useful as a diagnostic tool, but requires our authorship tools for a real solution." Like Grammarly, GPTZero features an "authorship" tool that Cui recommends be used to verify that future content submissions are written by humans. "Our writing reports in Google Docs and our own editor analyze the typing patterns on a document to see if the document was human written, and massively reduce the risk of incorrect conclusions," he said. Cui emphasized the importance of continually training the AI model on a diverse dataset. "We use large natural language processing (NLP) and machine learning models trained on a dataset of millions of AI and human-generated documents, and are tested to have low error before release," he said. "We tuned our detector to have less than 1% false-positive rates before launching in general to lower the risk of false positives." Penn pointed out that blindly relying on AI detectors to find plagiarism and cheating is just as dangerous as relying on AI to write a fact-based report. "My caution to anyone thinking about using these tools is that they have unacceptably high false-positive rates for any mission-critical or high-risk application," Penn said. "The false-positive rate -- if you're going to kick someone out of college or revoke their doctoral degree -- has to be zero. Period. End of story. If institutions did that rigorous testing, they would quickly find out there's not a single tool on the market they could buy. But that's what needs to happen." Thankfully, only 5% of this article came back as AI-generated. ZeroGPT, and Quillbot did not immediately respond to requests for comment.