A detector that catches most AI text but wrongly flags 10% of human work can be worse for your team than one that misses more AI but rarely flags a person incorrectly. The first tool looks impressive in a demo. The second may be far more useful in an actual queue.
The problem is the base rate. Say you review 1,000 drafts and only 50 contain the kind of undisclosed AI use you care about. A detector catches 45 of those but flags 95 legitimate drafts too. Your reviewers now have 140 “suspicious” submissions, and most are innocent. Congratulations, the detector has successfully detected extra meetings.
This is why I’d push back slightly on testing with an evenly balanced set. A 50/50 collection of human and AI passages is fine for comparing raw behavior, but it does not represent most real submission streams. After the initial blind test, recalculate the results using realistic rates such as 1%, 5%, and 20% questionable content. Look at how many manual reviews each tool would generate and what fraction of those reviews would be wasted. That tells you much more than an overall accuracy number.
You should define the policy violation before choosing the detector too. Are you trying to find completely generated drafts, undisclosed AI assistance, copied factual errors, or simply low-quality writing? Those are different problems. A person can use AI for an outline and write every sentence themselves. Another person can submit fully human work that is generic, repetitive, and poorly sourced. The detector may confidently dislike the first person’s sentence rhythm while missing the second problem that actually matters.
I’d run any candidate in shadow mode first. Record the scores, but do not show them to the editors making the initial quality decision. Later, compare the flags with revision history, source use, factual problems, and whatever your team considers unacceptable assistance. This avoids training reviewers to see guilt whenever a colorful percentage appears on screen.
The winner, if there is one, should be the tool that creates a manageable review queue without turning normal writers into suspects. If none can do that on your real submission mix, skipping detectors is a perfectly valid result. Buying software is not mandatory just because the software has a confidence gauge.