How Accurate Are AI Detectors, Really? The Published Numbers

Every AI detector advertises an accuracy figure. Almost none of them publish the two things that would make the figure usable: the false-positive rate, and the corpus the number was measured on. This article is about what those numbers can and cannot tell you before you act on a score.

For the short version, see how accurate are AI detectors. What follows is the evidence behind that answer.

What a single accuracy number hides

“Accuracy” is the share of documents a detector classified correctly on one particular test set. It merges two errors that have completely different costs. Missing generated text is an inconvenience. Labelling a person’s own writing as AI can end a grade, an application or a reputation.

Those two errors also move in opposite directions. Lower the threshold and you catch more generated text while flagging more humans. Raise it and you protect writers while missing more machine output. A vendor can pick whichever point makes its headline look best, so a single percentage says nothing about where the tool actually sits.

A number without a false-positive rate is a marketing claim. A false-positive rate without a described corpus is a claim about one test set.

What published numbers look like

The most instructive figure came from OpenAI. Its own AI Text Classifier correctly identified 26% of AI-written text as likely AI-written and incorrectly flagged human writing 9% of the time. OpenAI withdrew the tool on 20 July 2023, citing its low rate of accuracy, and noted it was unreliable below about 1,000 characters.

The lab with the most direct knowledge of how its models write shipped a detector, measured it, and took it down. That is the honest baseline against which every 99% claim should be read.

Institutions have run the same arithmetic in the other direction. Vanderbilt University disabled its Turnitin AI detector on 16 August 2023, pointing out that with the 1% false-positive rate quoted at launch, the roughly 75,000 papers it submitted in 2022 would have produced around 750 incorrectly flagged student papers. A rate that sounds negligible per document is not negligible at institutional volume.

Why a benchmark number does not transfer

Detection performance is measured under conditions. Change the conditions and the number changes with it. The variables that move results most are the ones you cannot control when a real document lands on your desk:

Generator. A detector tuned on one model family degrades on the next release.
Domain. News-like prose, academic essays, CVs and support emails behave differently.
Length. Short samples carry too little signal for any method to separate reliably.
Language. Most published evaluation is English-heavy; per-language calibration is rarely reported.
Post-editing. Paraphrase, translation and grammar passes all move a document away from the distribution the detector was measured on.

Large evaluations make this concrete. The RAID benchmark spans millions of generations across multiple models, domains, decoding strategies and adversarial attacks, and it fixes the false-positive rate before comparing detectors — precisely because comparing raw accuracy across different operating points is meaningless. An out-of-distribution evaluation of several zero-shot and supervised detectors reports true positive rates at a 1% false-positive rate collapsing toward 0% in some scenarios.

None of this makes detection useless. It makes a decontextualised accuracy claim useless.

Who pays for the errors

Error rates are not distributed evenly across writers. Liang and colleagues evaluated commercial GPT detectors on two human corpora and found that more than half of TOEFL essays written by non-native English speakers were misclassified as AI-generated, while US eighth-grade essays were classified almost perfectly.

An aggregate accuracy figure averages that away. The people most likely to be wrongly flagged are the least likely to appear in the number a vendor quotes. We cover this separately in why AI detectors flag non-native English writers.

What a credible disclosure contains

You do not need to run your own benchmark to evaluate a vendor claim. Ask for six things, and treat their absence as the answer:

The false-positive rate, stated next to the accuracy figure.
The corpus: size, languages, genres, and the date it was built.
Which generators produced the AI samples, and at which versions.
How short documents are handled — a floor, or an abstention state.
Whether the threshold was chosen at a fixed false-positive rate.
Whether non-native writing was audited separately.

IA Checker publishes a dated regression run on the AI detector page with its corpus size and limits in the same block as the figures. It is deliberately small: a check between model versions, not a global accuracy claim. We would rather show a modest measured number with its limits than a large one without them.

How to use a score once you have one

Treat the percentage as a pointer to passages, not as a verdict. Read the highlighted spans. Ask whether they are stock introductions or the analytical middle of the argument. Compare against earlier writing from the same person where you can. Expect two reputable tools to disagree on the same document, because they are measuring different things on different corpora.

Then keep a person accountable for the decision. A detector narrows where to look; it does not establish who wrote the text. See how to read an AI detection score for the reading protocol, and why AI detectors flag human writing for the failure modes.

How accurate are AI detectors, really?