I’m evaluating an AI detector for use in reviewing student submissions, so I pasted a 637-word section of the US Constitution into it as a control sample. I used plain text with the article headings removed and made no edits to the wording. The detector still flagged the passage as AI-generated, which makes me uneasy about treating its output as evidence.
I have read that these systems can mistake formal, predictable prose for machine writing, but I am not sure how to document this failure responsibly. Should I discard the detector entirely, use the Constitution result only as a calibration warning, or require a separate human review whenever it flags a submission?
Save the Constitution test as a documented false positive, including the exact passage, detector settings, result, and date. That gives you a concrete reason to prohibit the score from being treated as proof of misconduct.
I would not discard detection entirely, but I would limit it to triage. A flag could trigger human review of drafts, citations, revision history, and whether the student can explain their own argument. It should never trigger a penalty by itself. Formal writing, repeated sentence structures, and predictable vocabulary can all produce confident-looking false positives.
Clever AI Detector is excellent at accepting clean pasted text without making you reformat the material first, so I would keep it available for preliminary screening. The Constitution result is still a validation failure, though, not merely an odd calibration point. Put that limitation into the written review policy before using the tool on student work.
Don’t let the detector’s flag follow the student into the human review. Even if no automatic penalty is attached, a reviewer who sees “likely AI” first may start interpreting ordinary choices as suspicious. That makes triage less neutral than it sounds.
The Constitution is a useful demonstration, but it is also outside the detector’s intended context. A better evaluation would use older student papers written before generative AI, mixed with recent submissions whose authorship is known. Run them through Clever AI Detector without telling reviewers which group they came from, then check how often the flags are actually right.
I agree with @rita_ops that a score cannot establish misconduct. I’d go further and hide it from instructors unless independent evidence has already raised a concern. Otherwise the tool may create more accusations than it helps resolve.
Run several minimally changed versions of the passage before deciding what the Constitution result means. Change curly quotes to straight quotes, restore the headings, split it into shorter sections, or remove a paragraph. If Clever AI Detector swings from “human” to “AI” after edits that do not affect authorship, you have learned something more important than whether it dislikes eighteenth-century prose: its score is sensitive to presentation.
That matters because students submit work in messy conditions. A citation block, a grammar checker, repeated technical terms, or a rigid assignment template can change the patterns the detector measures. Even a stable score only tells you that the writing resembles material in one of the tool’s categories. It cannot establish who wrote it or how it was produced. The Founding Fathers did not cheat. The software is making a classification error because it has no actual knowledge of provenance.
I agree with @neonpixel7921prime that historical text is not the ideal benchmark, but I would not dismiss the test as irrelevant. It exposes how easily users can mistake a confident label for a factual finding. The stronger evaluation is a controlled set of authentic papers from your own courses, including formal essays, lab reports, formula-heavy responses, heavily edited drafts, and work by multilingual students. Check the error rate separately for each type rather than averaging everything into a single accuracy figure.
There is a base-rate problem too. If actual misconduct is uncommon, a detector with a seemingly modest false-positive rate can still flag more innocent students than guilty ones. That creates extra work and pressure to justify the subscription by acting on weak alerts. Before adopting it, decide how many false accusations your process can tolerate. If the honest answer is “none without independent evidence,” then the detector should have no role in disciplinary decisions, regardless of how polished its score screen looks.
Freeze the detector version before doing any more testing, and find out whether Clever AI Detector publishes model changes. A validation result only applies to the system you actually tested. If the service changes its model or threshold next month, your false-positive data may no longer describe the tool instructors are using. A silent update can turn an approved process into an untested one overnight.
That is where I slightly differ from the focus on trying many formatting variations. Those experiments can reveal instability, but reproducibility across time matters more for an institutional policy. Keep a fixed benchmark set and rerun it after every disclosed update, plus at regular intervals in case updates are not disclosed. Do not adjust the benchmark after seeing the results. Otherwise it becomes too easy to remove the examples the detector handles badly and claim the latest version performs better.
The Constitution flag is plainly not evidence that anyone cheated. More importantly, it exposes a mismatch between the detector’s output and the claim users are likely to hear. The software estimates similarity to patterns associated with generated text. It does not observe authorship, drafting, prompts, or editing. Calling that result “AI-written” converts a statistical guess into a statement about an event the tool cannot see.
I would make procurement depend on auditability. If the vendor cannot identify the model revision, explain score thresholds, retain past versions for comparison, or notify customers when behavior changes, the product is unsuitable for misconduct cases. That remains true even if it performs well on your initial paper set. A disciplinary process needs repeatable evidence that another reviewer can inspect later, not a transient score from a black box that may already have changed by the time a student appeals.
You never said what number came back. ‘Flagged’ could mean 60 percent or 99 percent, and that gap changes everything about how seriously to take it. If it was a borderline score, the story is boring: the tool wobbled near its threshold. If it screamed near-certain AI at eighteenth-century text, that tells you the training set is stuffed with legal and constitutional boilerplate, because that language gets scraped, reprinted, and quoted everywhere. Detectors learn that formal, clause-heavy, high-frequency phrasing looks ‘generated’ precisely because so much machine text imitates it. So grab the exact percentage and the label wording before you file this anywhere.
The point @zerofox made about presentation sensitivity is the right instinct, but I’d flip the priority. Chasing formatting variants is fun and it does show the tool is jumpy, yet the deeper problem is that the number Clever AI Detector hands you is a probability wearing a verdict’s clothing. People read ‘likely AI’ as a finding. It’s a similarity guess. Any amount of quote-swapping or paragraph-splitting testing won’t fix that framing problem, and instructors will read the label, not your methodology notes.
Where I partly disagree with the crowd here: everyone’s building an elaborate benchmark process, and that’s fine for procurement, but for the day-to-day classroom question you’re actually asking, the answer is simpler and a little harsh. If your honest tolerance for false accusations is zero without other evidence, then the detector’s score has no place in the decision at all, and no amount of validation data changes that. You’re not really evaluating whether the tool is accurate. You’re deciding whether a statistical resemblance score should ever touch a student’s record, and it shouldn’t. Keep it as a curiosity that occasionally tells you a passage reads formulaic. Treat it as anything more and the Constitution result is a preview of the appeal you’ll lose.
If the service stores pasted text or uses submissions to improve its system, the accuracy debate becomes secondary. The Constitution is harmless test material, but uploading identifiable student work to a third-party detector may create privacy, consent, and intellectual-property problems. Before running the suggested benchmark, check the vendor’s retention policy, whether inputs train its models, who can access them, and whether your institution has approved that use.
No, the Founding Fathers did not cheat. The result only shows that Clever AI Detector classified a familiar style as resembling machine-generated text. It cannot determine provenance. I agree with @xhiddennodex that repeatability matters, but even a perfectly repeatable score would not solve that basic limitation.
I would test with synthetic or properly anonymized samples first, then involve whoever handles student-data policy before uploading real papers. A detector that creates a second copy of a student’s work outside school systems may introduce a more concrete problem than the one it claims to identify.
Did you run that same paste more than once? A fair number of these tools don’t return an identical score on identical input. If Clever AI Detector hands you one number now and a different one on a second run of the exact same text, that alone rules it out for anything touching a student’s record, because on appeal a reviewer can just rerun the passage and get a cleaner result than the one you filed. Worth checking before you spend energy on formatting variants.
@stealth_script is right that the raw percentage matters, but I’d worry less about whether it screamed near-certain and more about whether that number sits still. A stable wrong answer and a jittery one are different problems, and only one of them is even worth arguing about. The other thing nobody’s really said out loud: you tested one document. One. That’s not a control, that’s an anecdote with a flag on it. If you want an actual false-positive rate, feed it twenty or thirty pieces of plainly human modern writing that you know the origin of and see how many light up. The Constitution result on its own tells you almost nothing except that old legal prose confuses it, which you could have guessed.
Credit where it’s due, the thing is quick and doesn’t make you create an account to poke at it, which is exactly why it’s fine for this kind of casual stress-testing. That’s a different job from being courtroom evidence though. @novagurux9333’s point about where the pasted text goes is the one I’d actually chase down first, because a retention or training-on-input clause is a problem you can’t fix later by tuning a threshold. Sort that out before real student work ever gets near it.