AI Detection Tools Can’t Catch Fakes—That's An Evidence Problem

The New York Times recently ran more than 1,000 tests on AI detection tools designed to spot fake images and videos. The results were sobering but predictable: the tools found some fakes, missed others, and flagged authentic content as synthetic often enough to make any serious investigator nervous. For anyone who’s spent years examining digital evidence on actual devices in actual cases, the findings confirmed something I’ve been saying for a while. We’re building our detection strategy on a foundation that was never solid to begin with.

The conversation about AI-generated content keeps circling back to the same question: can we build tools that reliably detect fakes? But as I see it, the far more dangerous problem is what happens to real evidence when nobody can agree on what’s authentic anymore.

Runway’s Turing Reel study from January put a hard number on it. They showed 1,043 people a mix of real footage and clips generated by their Gen-4.5 model. Over 90% couldn’t reliably tell the difference. Overall detection accuracy was 57.1%. Barely above a coin flip. In categories like animals and architecture, participants performed worse than chance, more likely to call a generated video real than to spot it as fake.

Those numbers should alarm every executive, general counsel, and board member reading this. Not because deepfakes are coming. Because they’re here, and the tools we’re counting on aren’t reliable enough to stake legal or business decisions on.

The Detection Trap

I’ve worked digital forensics cases for nearly two decades. The pattern I keep seeing is organizations treating detection tools as a silver bullet. They run a photo through a classifier, get a confidence score, and act as though that number means something definitive. But detection is a probabilistic guess, not a forensic conclusion. When I’m brought in to examine evidence after a dispute or an incident, I’m not running files through a web-based tool and calling it a day. I’m looking at the device, the metadata, the file system artifacts, the timestamps, the application databases; the full forensic picture that tells you not just whether something looks authentic but whether the digital evidence actually is.

That distinction plays out differently depending on what’s at stake. An AI detector that returns an 87% confidence score can be genuinely useful in the right context. Triaging insurance claims, flagging suspicious submissions, deciding whether something warrants a deeper look. When the cost of a full forensic examination exceeds the value of the decision, a probabilistic tool earns its keep. I don’t have a problem with that.

But 87% doesn’t necessarily survive a courtroom. Any qualified digital forensics expert can challenge that confidence score, and they will. What does 87% even mean when someone’s liberty or livelihood is on the line? It means you’re 13% unsure.

When you absolutely have to prove what’s real, you need a forensic examination that traces an image to a specific device, captured at a specific time, with consistent metadata and file system entries. 

The Liar’s Dividend Is Already Paying Out

Psychologists have a term for what’s happening: the liar’s dividend. As Poynter reported, the mere existence of convincing fakes gives anyone a ready-made excuse to dismiss real evidence as fabricated. UC Berkeley professor Hany Farid’s research has found that people are just as likely to call something real fake as they are to call something fake real, and accuracy drops further when political context gets involved.

I see this playing out in legal proceedings right now. Attorneys are starting to challenge digital evidence by raising the specter of AI manipulation, even when there’s zero indication that anything was altered. It works because the triers of fact can’t confidently distinguish between genuine and synthetic content. When everyone’s uncertain, doubt becomes the default.

The NSA published a Cybersecurity Information Sheet on Content Credentials in January 2025, recommending the C2PA provenance standard as part of a multi-layered approach to media authenticity. It’s a genuine step forward. But the NSA’s own document acknowledges that Content Credentials alone won’t solve the problem. Provenance standards are opt-in. They require adoption across the entire pipeline from capture device to distribution platform. Most smartphones don’t embed C2PA metadata today. Most social media platforms strip it. We’re building an authentication infrastructure that works only when every participant cooperates, which is exactly the scenario we’re least likely to encounter when someone is deliberately trying to deceive.

What Actually Works For Authenticating Visual Evidence

We already have a proven methodology for establishing digital evidence authenticity. It isn’t glamorous. It doesn’t involve a single-click classifier. It’s device-level forensic examination, the same discipline that NIST’s SP 800-86 has recommended for years: identify, acquire, analyze, and report digital evidence through a structured, defensible process.

When you examine a photo or video on the source device, you can correlate it against file system timestamps, application logs, GPS data, network activity, and other independent artifacts that either corroborate or contradict the claimed provenance. A fabricated image dropped onto a phone via AirDrop looks completely different at the forensic level than a photo taken with the device’s native camera. That’s not a probabilistic assessment. It’s a factual one.

Most organizations don’t have the forensic infrastructure or institutional knowledge to do this. They’re relying on screenshots, exports, and AI detectors, tools that were never designed to establish legal-grade authenticity. The realistic assumption for any investigation in 2026 is that the digital environment may have been adversarially manipulated before an examiner ever touches it. Anti-forensics isn’t theoretical. It’s a mature discipline, and AI has done what AI does to everything else: made it fast, cheap, and available to anyone with a laptop.

The Gap That Needs Closing

Here’s the hard truth. Your organization almost certainly depends on digital evidence in some capacity, whether for HR disputes, insurance claims, regulatory compliance, litigation, or internal investigations. And your organization almost certainly doesn’t have a defensible process for authenticating it.

Detection tools have a role. Provenance standards like C2PA have a role. But they’re layers in a defense, not the defense itself. The gold standard is the forensic examination of digital evidence on source devices, conducted by qualified examiners using validated methodologies. Everything else is a supplement.

The Times tested over a thousand images and videos against the best available detectors and found the results unreliable. Runway proved humans can’t do much better. The window where eyeballing content or running it through a classifier was “good enough” has closed. Organizations that don’t build forensic readiness into their evidence handling are going to find themselves unable to prove what’s real in the exact moment it matters most.

That moment is going to arrive sooner than most companies think.