Ahrefs Tried Replacing Marketers With AI. It Didn't Work

Here's what happens when you actually test the AI-will-replace-marketers thesis instead of just tweeting about it: Ahrefs, the SEO software company, spent a month trying to automate their product marketing with ChatGPT. They fed it a year of blog posts, built custom instructions, iterated on tone and style, and tested it on three real deliverables—a product update blog post, a YouTube script, and a newsletter going to tens of thousands of people.

The goal was 80% publish-ready with minimal human intervention. They got speed. They didn't get quality.

This isn't another "AI is terrible" story. It's more interesting than that. The experiment reveals something specific about where AI content breaks—and it's not where most people think.

The Setup: When Your Boss Hates AI Content

The Ahrefs team started with what should have been the easiest task: generating product update blog posts. They downloaded 12 months of their marketing team's posts, created a ChatGPT project, and wrote instructions. When the first draft came back with errors, they refined the instructions with more details about writing style.

Then they brought in Andre from the product marketing team for honest feedback. His notes were precise: "It's a bit technical. Probably tone down the language. Explain me like I'm five. The benefits could be more specific. Use examples for features that are hard to explain or hard to digest."

They implemented Andre's advice, regenerated the content, and felt confident. At this point, the creator noted: "I was confident Tim would see the value in how AI would save our team tons of time without sacrificing quality."

Tim is the CMO. Tim hates AI content.

The Test: What a Skeptic Actually Sees

When Tim read the AI-generated blog post, his reaction was immediate: "Right away, it feels AI." He pointed to a specific phrase—"The AI citations widget in overview now maps cleanly to brand radar"—and said, "If a human person wrote this, we're going to need to talk with that human."

Then they showed him the human-written version. His response: "Immediately for me, it is better. This feels like our regular update. It delivers the point across more clearly without using any extra words. Halfway through reading the paragraph, I already see what the person is trying to tell me."

The video script test produced something unexpected: Tim couldn't immediately tell which was which. He had to watch both twice. But once he identified the AI version, the pattern was clear again—more convoluted, harder to follow, and crucially: "It doesn't feel like the person knows the product."

That last observation is the entire game.

The Real Problem Isn't Voice, It's Context

The Ahrefs team correctly diagnosed what went wrong: "Our AI bot didn't actually know the product. It only knew what I told it. It didn't have the real-world context of how Ahrefs is actually used."

This is different from the usual complaints about AI-generated content being generic or lacking personality. Those are real problems, but they're solvable with better prompts and style guides. The Ahrefs experiment suggests something more fundamental: AI can mimic patterns, but it can't understand what matters.

Tim explained it this way: "It's not about them being able to tell, it's about us being able to communicate what we want to communicate. And I don't feel that AI has a goal to communicate something."

That's the gap. A human marketer who understands the product knows which details to emphasize, which features connect to which pain points, and which language will land with users at different skill levels. AI, even when trained on a year of your best writing, is fundamentally guessing based on patterns.

The newsletter test crystallized this. Tim's immediate reaction: "Right away, it says, 'In this quick 5 minute video, we walk you through blah blah blah.' I don't expect our newsletter to pitch the video. I expect our newsletter to pitch the product."

That's not a mistake you can fix with better instructions. That's a misunderstanding of purpose.

The Numbers Tell Two Stories

Here's where it gets interesting. According to Andre, doing all three tasks manually takes about a week. With AI? Three minutes.

But Tim rated the AI output at 60-70% quality—not the 80% target. His verdict: "I think I would agree that we would send something like that. If we can push it to like 80-90, then I would be comfortable shipping it."

So yes, AI saved massive amounts of time. And no, it didn't maintain quality standards. Both things are true. The question is which one matters more to your business.

For commodity content—the stuff you're producing because you need to produce something—60% quality at 3 minutes might be a winning trade. For content that actually needs to communicate product value to people who have other options? The math is less obvious.

What This Actually Tells Us

The Ahrefs experiment is useful precisely because it wasn't designed to prove AI is bad. They genuinely tried to make it work. They iterated. They refined prompts. They built custom instructions. They had subject matter experts review and provide feedback.

And they got to 60-70% quality in 3 minutes instead of one week.

That's not a failure. It's just information. The productivity gain is real. The quality gap is also real. Whether that trade works depends entirely on what you're trying to accomplish and who you're trying to reach.

The pattern I've seen repeated across 25 years of tech hype cycles: New tools don't eliminate expertise. They change what expertise looks like. The Ahrefs marketers aren't going to be replaced by AI. They're going to spend less time on first drafts and more time on the parts that actually require product knowledge and strategic judgment.

Which is probably what should have been happening all along.

Mike Sullivan is Buzzrag's Technology Correspondent and has been watching automation promises since people thought Java applets would replace desktop applications.