AI Knowledge Gaps Are a Governance Problem

The EU AI Act, which entered full application for high-risk systems in August 2024, requires that organizations deploying consequential AI maintain "human oversight" and ensure systems behave according to their intended purpose. Article 9 mandates a risk management system covering "all risks that may be reasonably foreseeable." Article 13 requires transparency sufficient for users to understand and correctly interpret outputs.

None of it says a word about what happens when the AI's foundational context — the institutional knowledge encoded into it by the humans who built it — is wrong, stale, or incomplete from the start.

That regulatory silence is, for my money, where the interesting story sits. And a recent video from AI automation creator Nate Herk, however unintentionally, illustrates exactly why.

The Problem Herk Is Actually Describing

Herk's video presents a workflow technique he calls "grill me" — a Claude Code skill that systematically interviews a user about a process, checkpoints their answers after each exchange, and writes everything to a persistent knowledge document. The premise is simple and the diagnosis behind it is accurate: most people who try to build AI systems around their work do a five-minute brain dump, get mediocre results, and then spend weeks iterating toward something functional. Front-load the extraction, Herk argues, and you compress that timeline dramatically.

"The real challenge is still the extraction," Herk says, "getting everything from your head into the AI system so that your skills can use it and that your context is better."

He's describing something knowledge management researchers have called tacit knowledge externalization — the notoriously difficult process of converting expertise that lives in human judgment into explicit, transferable form. What Nonaka and Takeuchi documented in organizational learning theory in the 1990s, Herk is attempting to solve with a prompt that asks Claude to interrogate him until there are no gaps. The problem is not new. The tooling is.

The original "grill me" prompt, which Herk credits to a creator identified in the video only as "Matt PCO" — a citation I can't independently verify beyond Herk's attribution — runs to four or five sentences: interview me relentlessly, walk down each decision branch, ask questions one at a time. Herk extended it to force checkpointing after every answer, worried that in sessions running an hour or more, context window limitations might cause the model to misremember early answers. The result is a folder of markdown files — "brainstorms" — capturing the Q&A log, key decisions, open flags, and summary.

It is, as Herk demonstrates, genuinely useful. His self-reported efficiency numbers — starting at roughly 70% reliability on a naive first-pass versus 90% with front-loaded extraction, reaching a ceiling around 95% after anywhere from 10 to 30 iterations — carry no external validation and should be understood as illustrative of his own experience rather than anything empirically established. But the underlying logic is sound: structured elicitation produces better inputs than unstructured brain dumps, and better inputs produce better outputs.

Where the Workflow Ends and the Accountability Question Begins

Here is where I need to depart from treating this as a workflow story, because it isn't only that once you move it out of the solo-practitioner context Herk is demonstrating and into an enterprise setting.

Consider what these knowledge documents actually are in organizational use. They are not just productivity artifacts. They are the encoded institutional logic that AI agents act on when making or recommending decisions. When a company deploys an AI system to handle procurement, customer service, compliance checking, or content moderation — all plausibly "high-risk" categories under the EU AI Act's Annex III — the knowledge layer Herk is describing becomes load-bearing infrastructure.

Herk gestures at this when he notes that some flagged gaps in his brainstorm sessions required him to go find "the actual stakeholder or operator that does that process" and bring their knowledge back into the document. That's a reasonable acknowledgment that one person rarely holds all the relevant context. But in regulated industries, the gaps in that knowledge document aren't just inefficiencies. They're potential liability triggers.

If an AI system trained on an outdated or incomplete extraction recommends a financial product incorrectly, denies an insurance claim based on stale underwriting logic, or generates compliance guidance that reflects policy as it existed eighteen months ago — who bears responsibility under GDPR's accountability principle (Article 5(2)), which requires controllers to demonstrate that processing is lawful and accurate? Who answers to the AI Act's human oversight requirements when the "oversight" was actually delegated to a knowledge document that nobody has updated since the initial extraction session?

The "grill me" technique, as Herk presents it, has no formal versioning, no update obligation schedule, no audit trail beyond the markdown files themselves, and no mechanism to flag when organizational reality has diverged from what the document records. Herk acknowledges this informally — "as your business evolves and as you evolve, the skill keeps evolving" — and demonstrates returning to a brainstorm file to run a new extraction when something changes. But "return when you remember to" is not a governance framework.

What Regulation Has and Hasn't Addressed

The EU AI Act's human oversight provisions focus on the moment of deployment and operation — can a human monitor and intervene in system outputs? They are largely silent on the quality assurance of the knowledge layer that precedes deployment. GDPR's accuracy principle (Article 5(1)(d)) requires that personal data be "kept up to date" and that "every reasonable step must be taken to ensure that personal data that are inaccurate...are erased or rectified without delay" — but this applies to personal data, not to the institutional knowledge and decision logic encoded in AI context documents.

There is no regulation, to my knowledge, that currently treats the knowledge base of an AI system — the thing that tells it how the organization thinks, what its processes are, and what decisions should look like — as a regulated artifact with maintenance obligations. The AI Act's Annex IV documentation requirements for high-risk systems cover training data, validation, and monitoring. They don't cover the prompt-level knowledge context that, in agentic systems built on large language models, increasingly substitutes for traditional fine-tuning.

Herk's technique, scaled to an enterprise, produces exactly the kind of document that should probably have formal governance requirements: version history, sign-off from relevant stakeholders, a defined review cadence, and some mechanism to invalidate or quarantine knowledge that has become stale. The practitioner community is building these knowledge management practices informally, through trial and error, because regulation hasn't yet recognized this layer exists.

That gap will close eventually, probably after a consequential incident makes it undeniable. The EU AI Act's review mechanism — the Commission is required to evaluate the regulation by August 2026 — is the earliest plausible point at which this could be addressed. Until then, organizations deploying agentic AI systems are making informal decisions about knowledge governance that carry potential legal exposure they may not have mapped.

The Axe-Sharpening Problem

Herk invokes the familiar aphorism — "if I had six hours to chop down a tree, I would spend the first four sharpening the axe" — often misattributed to Abraham Lincoln, for whom no documented evidence of the quote exists. The sentiment is sound regardless of its provenance: front-loaded preparation compounds downstream. His self-deprecating acknowledgment that the numbers he draws are illustrative rather than data-driven is, to his credit, transparent.

The honest version of the axe-sharpening logic applied to enterprise AI governance is this: the organizations spending time now building formal knowledge management practices for their AI systems — defining who owns a knowledge document, when it must be reviewed, what triggers an update, and how outdated context gets flagged — will be far better positioned when regulators eventually catch up to this layer of the stack. The ones who treat it as a workflow convenience will find out later that it was a compliance obligation in everything but name.

Herk is solving a real and underappreciated problem. The technique is worth attention. But the more important question it surfaces — about accountability structures for the knowledge that governs AI behavior in high-stakes settings — is one that practitioners, platform developers, and regulators are all, at the moment, carefully not answering.

---

Samira Barnes covers technology policy and regulation for Buzzrag.