Chunkless RAG: Navigate Documents by Structure

You've done it. You paste a 40-page PDF into an AI tool, ask it a completely reasonable question about something specific on page 28, and watch it hallucinate a confident summary that addresses literally everything except what you asked. The document was right there. The answer was right there. And still: vibes-based garbage.

This isn't bad luck. It's architecture. And a new approach from IBM's Docling project — called Chunkless RAG — is a direct argument that the dominant retrieval method of the last two years has been destroying the thing it needed most.

The map was always there

Standard RAG (retrieval augmented generation — the technique where you feed an AI relevant document snippets to ground its answers in real source material) works like this: take your document, slice it into chunks of maybe 500 words each, convert those chunks into vectors (numerical representations of meaning), store them, and when a question arrives, pull back whichever chunks are mathematically most similar to the question. Hand those to the model. Get an answer.

For a lot of use cases, this is genuinely fine. Fuzzy search across millions of documents? Similarity search wins. Nobody's complaining about it in those scenarios.

The problem is what happens when you hand a long, structured document — a legal contract, an annual report, a technical specification — to a pipeline built this way. In IBM Technology's latest explainer, Ming Zhao walks through why structured documents and chunk-based retrieval are a mismatch that's been hiding in plain sight.

Zhao describes the core issue: when you slice a document into chunks, you've discarded how it was put together. A heading gets separated from the paragraphs it was introducing. A table gets split from the sentences that explain what the table means. If the answer to your question spans multiple sections, the similarity search has no mechanism to know those sections are related — it's only ever comparing small blobs of text against your query.

But here's the part that actually got me: the document already had a structure. The author built a tree — title, headers, sections, subsections, tables, all of it organized hierarchically. Zhao argues that chunking takes that tree and flattens it to run similarity search over the flat pieces, destroying the structure in the process, and then the system has to spend energy trying to recover relationships that were present the whole time. We bulldozed the map and then acted surprised that navigation was hard.

How an agent reads like a person

Chunkless RAG flips the retrieval logic. Instead of flattening the document and pattern-matching, it keeps the tree and lets an AI agent navigate the structure.

The analogy Zhao uses is the one that stuck with me: how would you actually find the answer to a question in a 200-page annual report? You wouldn't read all 200 pages. You'd open the table of contents, locate the relevant section, flip there, read it, and follow any references. You'd navigate.

The agent does exactly this. It starts with a structured outline where each section has a short summary — enough to understand the document's shape without reading the body. It reasons about which section is most likely to hold the answer, opens just that one, and reads. If it's sufficient, it answers. If not, it continues navigating, section by section, until it has what it needs.

Two things fall out of this that are structurally difficult to get with flat chunks. First, context is preserved by default — when the agent reads a paragraph, it already knows which section and subsection that paragraph lives in, because it walked the tree to get there. The heading hierarchy is part of the path. A chunk retrieved by similarity search has no provenance at all; it's a fragment adrift. Second, multi-section questions become tractable. If a policy is defined in one section and its rationale is explained three sections later, the agent can hold its place, branch to the other section, and return. It has a map. Similarity search just hands you whatever looked similar to your query and hopes.

This is the framing Zhao uses to introduce Docling: if chunkless RAG depends on having a document tree, you need something that can actually produce that tree from the formats documents actually arrive in.

The hard part is the PDF problem

PDFs are, functionally, render instructions. They tell a viewer where to place characters and figures on a page. There's no inherent semantic structure in there — no machine-readable hierarchy of sections, no table metadata, just coordinates. The structure the author intended is buried in the visual layout.

Docling — an open-source IBM project — is what bridges that gap. You give it a PDF and it returns a structured document object: real sections with headings, preserved reading order, tables that are still tables. The Docling deep dive we've covered previously goes into how that transformation works under the hood, including how it handles the messy real-world formats that make enterprise document processing genuinely hard. Once you have that object, the Docling agent can navigate it, extract fields, and run the chunkless retrieval loop Zhao describes.

Zhao is clear-eyed about the cost: parsing real-world documents into clean structure is hard, and it's where most of the engineering lives. Walking the document tree also means multiple model calls rather than a single vector lookup, which means more latency. This isn't a strict upgrade over chunk-based retrieval — it's a different tool with a different performance profile.

The hybrid reality

So where does this leave practitioners? Zhao's framing is that both approaches are retrieval — the question is which one you reach for and when.

Similarity search still dominates when you need breadth: scanning a large corpus for anything relevant to a fuzzy query is exactly what vector search was designed for. The structured navigation approach earns its cost on long, organized documents where you need precision and the connections between parts are the actual substance of the answer.

In practice, Zhao suggests, many real-world systems will use both in sequence: similarity search to identify which document to look at, structure-aware navigation to find the answer inside it. A relay, not a replacement.

That framing is honest, but it surfaces a question Zhao doesn't fully answer: at what granularity do you make the handoff? If you're building a pipeline over a corpus of mixed document types — some structured contracts, some unstructured emails, some semi-structured reports — the decision about when to engage chunkless RAG and when to stay in vector-land isn't trivial. It requires a classifier upstream that can reliably identify document type and complexity before routing. That's a solved problem in some cases and a real engineering lift in others. The practitioners asking "should I try Docling's chunkless RAG" probably need to answer "what does my document intake actually look like" first — specifically whether the documents in question have enough structural consistency that Docling's parsing will return clean trees rather than ambiguous hierarchies that confuse the agent more than chunks would.

That's the actual design question. Not "chunks vs. structure" as a philosophical preference, but: given this document distribution and this query type, where does precision matter enough to justify the latency and parsing complexity?

The author already drew the map. Docling reconstructs it. Whether your agent needs a map or just a keyword search depends entirely on the territory.

---

Yuki Okonkwo is Buzzrag's AI & Machine Learning Correspondent.