How AI content detectors work

Is the third bot the charm?

At the start of 2025, I experienced a first in my 16+ year career as a writer. I was told that the blog I had submitted was revealed by an AI detector to be 36% AI.

Imagine my surprise, as I became a writer to write actual content, not to become a prompt engineer. The idea that I’d turn over my job to a bot (who, it just so happens, I’m afraid is coming after said job anyway) was an anathema to me, especially as I still believe there’s a beauty to 100% human writing that even the best AI tools can’t replicate.

And yet, here we were. The machines, allegedly, don’t lie -- so how is it possible that my content could have been both 100% human written and 36% AI written at the very same time?

I wasn’t alone in facing this mystery, either. This is a question many of my fellow writers have grappled with. In fact, a quick Google search shows many complaints from frustrated writers who want to know why AI detectors are flagging their original work.

So, how is this happening? Unfortunately, it has to do with how AI content detectors operate. So, let’s take a look at the process these detectors use and where the problems are coming from.

How do AI detectors work?

While I don’t outsource my writing to AI, I do outsource some basic research questions, so I’m happy to tell you that ChatGPT says there are around 15 to 30 widely-known tools used to detect AI content (as well as 10 to 20 academic and enterprise systems).

These AI detectors, of course, can’t read and understand writing, as LLMs can’t engage in human-like reasoning (at least not yet!). Instead, the AI detectors all employ a similar process. They use machine learning models to try to recognize the patterns that set AI-content apart from human content.

Typically, the creators of AI detectors train the models based on huge volumes of both AI writing and human writing, to enable the models to recognize major differences in sentence structure, style, and predictability.

New writing is analyzed with these patterns in mind, with detectors looking for the classic hallmarks of AI-written work versus human prose.

What do AI detectors look for when analyzing work?

So, what are some of the key patterns that tip off an AI detector that a fellow AI agent was the creator of the work it’s analyzing?

Let’s look at GPTZero, one of the more popular detectors, to find out.

GPTZero explains that it “was one of the first AI detectors to pioneer the idea of using ‘perplexity’ and ‘burstiness’ to evaluate writing.” So, what does that mean exactly?

• Burstiness. Burstiness (yes, it’s a real word) refers to variations in sentence structure, length, and complexity throughout the text. Humans tend to vary their sentences more, with a mix of long, short, simple, and complex sentences. AI-written works tend to have sentences that are more uniform in length, structure, and style.

• Perplexity: Perplexity measures how predictable your word choice is. AI models use words that are statistically likely to follow other words, which means their writing has lower perplexity. Humans tend to be less predictable. The fewer surprises there are in word choice, the more likely it is that the text is AI.

While GPTZero’s model is continually evolving and now incorporates a multi-layered approach with additional components, perplexity and burstiness continue to play a fundamental role in AI-content detection, not just for GPTZero but for other detectors as well.

Essentially, if a detector goes word-to-word and finds that the statistically probable word was used 90% or 100% of the time, it will flag that part of the text as being likely AI because a real human writer is very unlikely to always choose the exact word that’s most likely to follow the previous one.

Perfection can also be a factor, as AI doesn’t tend to make the types of mistakes people do in writing. For example, while it may hallucinate entire legal cases and get lawyers sanctioned, it is far less likely than I am to type “layer” when it means “lawyer.” When work is perfectly free of typos and grammar mistakes, this is a red flag.

This is a source of frustration for both myself and my fellow writers, who feel like sometimes being “too good” at self-editing can now cause a false positive.

How often are AI detectors wrong?

AI detectors can be wrong in two ways:

• The detector may miss the presence of AI text.

• It could falsely identify a text as being written by AI when it is not.

A false positive is what happened to me, and it happens a lot. In fact, research from the University of Pennsylvania showed “dangerously high” rates of false positives, and research published by Cornell found “the available detection tools are neither accurate nor reliable.” Inside Higher Ed provided a long list of similar studies showing problems with detection tools.

This isn’t a surprise because, after all, sometimes humans do write in predictable ways -- especially if they’re writing content about technical subjects where there are only so many ways to phrase things or if they’re writing SEO content that usually follows a somewhat standard format.

There are also other issues at play as well.

For one thing, AI models were trained to write, in part, by learning from content that’s on the web.

Since I’ve written over 5,000 blogs and SEO pages published on the web -- and my fellow writers dealing with false positives have written hundreds of thousands more pages -- it’s not a huge surprise that some of our content sounds a bit like AI wrote it. We may very well have contributed to teaching AI how to write in the first place.

There’s also the fact that some of the detectors that alert you to the presence of AI content conveniently just happen to have a solution: You can pay them to “humanize” the AI text for you. Of course, it’s probably just a coincidence that they happen to find so much of the text they analyze in need of that service.

For example, I put the first few paragraphs of this article, which was 0% AI-written, through Sidekicker, which found that 97% of my text showed signs of AI generation… and which offered me the chance to “Remove AI content.”

Unfortunately, when I went through the “humanization” process, I would have had to pay $1.95 for a seven-day trial to unlock my newly-machine-written “human content,” unless of course I wanted to just go ahead and subscribe to a monthly plan!

Fortunately for Grok, it wouldn’t necessarily have to pay this fee, as I asked Grok to write me 500 words on how AI detectors work and input that text (unchanged) into Sidekicker, which found Grok’s work was only 70% AI.

GPTZero did much better, finding my text was very likely “entirely human-written” while Grok’s content was 100% AI-Generated.

QuillBot, on the other hand, found my content to have 0% AI, and Grok’s to be 92%, so still fairly good.

And myPerfect Words thought my writing was 12% AI, including, oddly, my opening sentence, which included a personal story, while Grok’s text was 91% AI generated.

These results show just how important it is to find the right detector if you’re going to rely on these programs to help you evaluate a content’s source.

What happens if work is falsely flagged -- and are there solutions?

Unfortunately, given the nature of the world right now, what we end up with here is an AI tool, detecting AI-written content, which will presumably be rewritten to sound more human by a third bot -- because I guess the third bot is the charm.

And the consequences of this situation can be pretty dire for all the humans involved in the process.

If AI detectors make a mistake in an academic setting, students face disciplinary action. In the professional world, companies feel taken advantage of if they think human writers are trying to pass off AI content as original. Writers could also lose their jobs or spend a very angry hour or two revising falsely flagged text to keep the peace and keep the paycheck.

So, what’s the solution? There may not be a great one.

Research has also shown that people are pretty bad at detecting AI, especially with more advanced LLM models writing text, so editors can’t necessarily count on their own abilities to identify whether something is human-written or not.

Companies can require draft notes, review Google Docs histories, or use tools like Grammarly Authorship to see if text was pasted in or written. They can also work only with writers they trust who have a large portfolio of pre-AI work, which would be great for established writers but not so good for those trying to break into the business.

For their part, writers should make sure not to be formulaic, to vary their word choices and sentence lengths, and to incorporate personal anecdotes when they can.

I’ve even heard of some fellow writers throwing in a typo or two on purpose when they turn in their drafts, which, of course, is not something I’d ever consider in my own werk.