Building an AI Video Detector: What Ten Hours of

Matt Wolfe has a problem: his friends and family keep sending him AI-generated videos, believing they are real. His response was to try to build a fix. The result, documented in Trying To Solve The Biggest AI Problem on his YouTube channel, is a 20-minute chronicle of ten-plus hours of prompting, an overnight automated testing marathon, and a tool that sort of works but cannot be shared. The failure modes are more instructive than the modest success.

The Idea, and the First Faceplant

The plan was simple: a website where you paste a link from Instagram, TikTok, YouTube, or X, and it tells you whether the video is AI. Wolfe leaned on Google's SynthID for watermark detection and Gemini's video-understanding models, reasoning that Gemini is among the few models that can watch footage and describe it. He used ChatGPT's coding agent to build the thing.

The first test came fast. He fed the detector a clip he calls "FartZuka": a woman in an inflatable jetpack suit skimming across water, obviously synthetic to any human eye. The detector rated it "probably not AI" with high confidence, reasoning that a particle spray looked like traditional CGI rather than generative AI.

When he asked the newer coding model to review the verdict logic, its own diagnosis was blunt: "I would not trust the current high confidence labels." High confidence was being assigned automatically from Gemini's raw score, and the app could override an inconclusive assessment entirely. In other words, the confidence numbers were decoration. This is a mistake worth naming because it recurs everywhere: raw model scores are not calibrated probabilities, and slapping the word "confidence" on an uncalibrated number is how detectors earn trust they have not earned.

Eight Hours of a Robot Saying No

Wolfe then told the agent to keep testing itself until the detector worked. It ran for over eight hours, produced four documents, and ended with the status "goal stalled." Along the way it ran a benchmark: the detector caught 23 of 39 AI clips and falsely flagged 2 of 36 real clips. Roughly 59 percent sensitivity against real-world slop, plus some false positives. Not a foundation for a public service.

The agent's own conclusion was to reach for a third-party API, a service called Site Engine that offers AI detection as a paid feature. Note what happened here: the general-purpose coding agent, given a goal and hours to work, independently converged on the conclusion that the frontier multimodal models cannot do this task, and a specialized computer-vision pipeline can do it better. Site Engine flagged AI indicators in 19 of 23 sampled frames of one test reel while Gemini found "no clear AI indicators" in the same footage.

The irony is structural. Gemini can describe what happens in a video, but it cannot reliably say whether the video was generated. The model with the best video comprehension in the consumer market is among the worst at judging provenance.

Why Detection Keeps Failing

This is not just one YouTuber's bad luck. Research on generative media forensics has documented the pattern for years. A CVPR workshop paper describes how detectors degrade badly on "out-of-domain" material, footage unlike whatever they were trained on, which is exactly the situation a public-facing detector faces every day. Separately, work on frame-consistency signals tested whether "video detectors" generalize across different generation models and found that performance drops when the generator changes. Detection accuracy is brittle by default, not by accident.

Add the cat-and-mouse dynamic Wolfe acknowledges: every improvement in detection is an invitation for generators to improve evasion. Readers who want the human-expert version of this problem, where trained eyes catch what automated tools miss, can find it in Spotting AI Scams: A VFX Perspective.

There is also a disagreement problem baked into multi-detector designs. When Site Engine says AI and Gemini says clean, Wolfe's app surfaced "inconclusive" rather than picking a winner. That is the right engineering call, and it produced his most frustrated moment: "I can still give it an AI video that I, as a human, can clearly tell is AI, and it still tells me it's inconclusive." Honest inconclusiveness feels worse than wrong confidence, but it is more useful. A tool that says "maybe" teaches users to check elsewhere. A tool that confidently says "real" teaches users to stop checking.

The Economics Nobody Prices In

The detector eventually reached decent accuracy on Wolfe's small test set: clean verdicts on human-made videos from friends, AI flags on the FartZuka clip and a physics-breaking crane video. Then came the bill. Running six videos consumed 12,150 API operations; a single video cost 440. The $100 monthly tier of the detection service bought 40 operations. A free public tool was financially impossible on its face.

Wolfe released the code on GitHub instead, bring-your-own-key. That outcome is itself a data point: detection is not hard because the algorithms are secret, it is hard because per-frame analysis at scale is expensive, and nobody has figured out who pays for it. The platforms generating the most synthetic content are not the ones billing for its verification.

The AGI Footnote

Wolfe closes with an argument against the "AGI has arrived" claims from figures like Jensen Huang and OpenAI: if these systems match humans at nearly everything, why can a human spot obvious AI slop in seconds while Gemini, the best video-understanding model available, cannot?

It is a fair rhetorical jab with a real limitation. Human visual judgment of synthetic media draws on physical intuition, contextual knowledge, and suspicion that no current model has been trained to replicate, and a single task does not settle whether any system meets a definition as contested as AGI. But the episode does illustrate something concrete about the gap between benchmark performance and real-world competence. Models ace curated benchmarks and then miss a woman jetpacking across a lake in an inflatable suit.

For now, the practical guidance from Wolfe's experiment holds whether or not the AGI debate is settled: treat AI-content detectors as probabilistic evidence, never verdicts; benchmark any detector on real samples before trusting its confidence labels; expect disagreement between systems and surface it; and price the API bill before promising the world a free tool. The slop is not going away. Neither is the gap between what detection vendors promise and what twelve thousand operations buys you.

Bob Reynolds is Senior Technology Correspondent at Buzzrag.