Hand someone a 200-page annual report and ask, “what changed in the revenue recognition policy this year, and where does the report explain why?” — and a person who knows the document can answer in about two minutes. They flip to the right section and read it. Watch what happens when a model tries to answer the same question, though, and you’ll notice the usual approach throws away the exact thing that made the question easy in the first place.
The usual approach
The standard way to do this is retrieval-augmented generation, or RAG. Cut the document into chunks — every 500 words, or by paragraph. Turn each chunk into a vector and store it. When a question comes in, embed the question too, and pull back the handful of chunks that look most similar. Feed those to the model and let it answer from them.
This works, it’s cheap, and for a lot of questions it’s genuinely all you need.
The strain shows up with big, structured documents. The moment you cut one into chunks, you throw away how it was put together. A heading gets separated from the paragraphs it introduces. A table gets split from the sentences explaining what it means. And if the answer spans multiple sections, similarity search has no way of knowing those sections belong together — it was only ever comparing small blobs of text against each other.
You get fragments back, and the model has to guess how they relate.
Documents were never a pile of text
That document was never a pile of text to begin with. Someone wrote it as a structure: a title, headers, sections and subsections, paragraphs, tables, images — all sitting under a tree. The author already organized the information. Chunking takes that tree, flattens it into a list of blobs so similarity search has something to compare, and in the process throws away relationships that were sitting right there the whole time. Then a lot of engineering effort goes into trying to reconstruct what chunking just destroyed.
So consider the alternative: don’t flatten the document. Keep the tree, and let the model reason its way to the right part instead of matching by similarity.
Think about how you’d actually answer the revenue question yourself. You wouldn’t read all 200 pages. You’d open the table of contents, find the section on accounting policies, and flip to it. If it pointed to a footnote, you’d follow the reference. You’d navigate the document — and that’s exactly what an agent can do too.
It starts with an outline of the tree, where each section comes with a short summary, so it can see the shape of the document without reading the body yet. It reasons about which section most likely holds the answer, opens just that one, and reads it. If that’s enough, it answers. If not, it picks the next section and keeps going, a few steps at a time, until it has what it needs.
What you get back
Two things fall out of this that are hard to get from flat chunks.
Context comes along for free. When the agent is reading a paragraph, it still knows which section and subsection that paragraph lives in, because it walked the tree to get there. The headings above it are part of the path. A chunk pulled by similarity search has no idea where it came from.
You can answer questions that span the whole document. If a policy is defined in one section and the reasoning behind it sits three sections later, the agent can hold its place, go read the other branch, and come back. It’s moving around a map. Similarity search just hands you whatever happens to look similar to the query — it can’t do that.
The agent isn’t doing less total work — walking the tree means several passes, and that adds up. What changes is what ends up in front of the model right before it answers: not the entire document, and not a handful of chunks that have lost their place in it, but one relevant section with its heading still attached. The model reasons over the right material in its right shape, which tends to mean cleaner answers with fewer moments of inventing connections between fragments that were never actually connected.
The hard part: you need the tree first
Everything above depends on actually having a tree to walk. That’s the hard part, because most documents show up as PDFs — which are basically just instructions for placing characters and figures on a page. There’s no clean structure sitting in there waiting to be read.
This is the problem Docling is built to solve. Give it a PDF and you get back a structured document object: real sections and headings, reading order preserved, tables that are still tables. The PDF format buries the hierarchy the author put in, and Docling reconstructs it. Once you have that object, a Docling agent can work on the structure directly — writing, editing, extracting fields, enriching sections — including an agent that does exactly the navigation described above.
The project calls this chunkless RAG, which is a fair description. Same goal as ordinary retrieval — find the right material and ground the answer in it — but it gets there by reasoning over the document’s structure instead of chopping it up and matching by similarity. It’s still retrieval. It just keeps the document whole while doing it.
Where it’s worth the cost
It isn’t free. You need a good tree before any of this works, and parsing real-world documents into clean structure is a genuinely hard problem — that’s most of the engineering effort. Reasoning through the structure also takes more back-and-forth with the model than a single vector lookup, so there’s more latency and more calls involved.
Chunk-based retrieval is still the right tool a lot of the time. With millions of documents and a fuzzy “find me anything about X” query, similarity search is hard to beat. The structure-aware approach earns its keep on long, organized documents where precision matters and the connections between parts are what you’re actually after.
In a lot of real systems, you’d want both: similarity search to find the right document, structure to navigate inside it once you’re there.
That’s the real choice — not whether to do retrieval, but whether you chop a document into pieces and match by similarity, or keep it whole and reason through its structure. The author already drew the map. Structure-preserving tools just hand it back to you, and an agent can finally follow it.