Retrieval-augmented generation answers from a handful of passages it retrieves, which works for questions whose answer sits in one document. It fails on corpus questions — questions whose answer requires examining every document in a set, such as which contracts auto-renew this quarter. Whole-corpus knowledge intelligence closes that gap by ranking, compressing and holding the material a question actually needs in a rolling working set, and by reporting how much of the corpus informed the answer.
The question RAG cannot see
Consider the difference between two questions an in-house legal team might ask:
"What does the indemnity clause in the Acme agreement say?" — This is a retrieval question. The answer lives in one passage of one document. RAG handles it well — and so does document-level intelligence, which reads a file end to end rather than searching across many (IDP vs OCR covers that layer).
"Which of our agreements auto-renew this quarter, and which of those lack a termination-for-convenience clause?" — This is a corpus question. Answering it requires examining every agreement, not the top-k most similar ones. A retrieval step that returns 5, 20 or even 100 chunks has silently discarded the rest of the archive — and with it, the correctness of the answer.
The failure mode is dangerous precisely because it is invisible: the answer arrives fluent and cited, and nothing signals that 4,000 relevant documents were never consulted. In compliance, audit and portfolio-review contexts, a confidently incomplete answer is worse than no answer.
Why "just increase top-k" doesn't fix it
Three structural limits stand between conventional RAG and corpus-level questions:
Retrieval decides what the model is allowed to know. Similarity search retrieves what resembles the question, not what matters to it. A question about missing clauses is answered by documents that don't contain the phrase being searched for — the exact documents similarity ranking pushes to the bottom.
Context windows bound the evidence. Even the largest model context holds a fraction of an enterprise archive. Somewhere, a cut is made; the question is only whether it is made deliberately or by accident.
Coverage is unmeasured. Standard RAG has no notion of how much of the relevant record an answer accounts for. Without a coverage signal, there is no way to distinguish "the archive says X" from "the three chunks I saw say X."
What whole-corpus knowledge intelligence requires
Closing the gap is an architecture problem, not a prompt problem. Five requirements define it:
- Index in place. Connect to the file servers and shares where documents already live and index them there — no migration project, no second copy of sensitive material, and deletions drop out of the index at the next sync.
- Sustained working memory over the corpus. Instead of a single retrieve-then-generate step, the system iterates: retrieval, compression and a rolling working memory that lets reasoning span millions of pages — the effect of unlimited context, engineered rather than assumed.
- Answers cited to document, page and passage. Every claim traceable to its source, so verification replaces trust.
- Coverage and confidence, reported. The answer states how much of the relevant record it examined. This single number is what turns a chat toy into an instrument an operations or compliance team can rely on.
- Permission-aware retrieval. The system answers only from what the person asking is entitled to see, with a full audit trail of who asked what and what was consulted. An AI layer that flattens access controls is a data breach with a search box.
| Retrieval-augmented generation | Whole-corpus knowledge intelligence | |
|---|---|---|
| Suits | Questions whose answer sits in one passage of one document | Questions whose answer spans the whole set and exists in no single document |
| Evidence considered | The top-k passages that fit the context window | Ranked, re-ranked and compressed material held in a rolling working set |
| Behaviour when the set is too large | Answers from what was retrieved, without saying what was missed | Reports how much of the corpus informed the answer and what was excluded |
| Coverage | Unmeasured | Reported alongside the answer |
| Permissions | Whatever the index was built with | Read from the source systems and enforced at query time |
Build vs. buy: the honest comparison
A capable engineering team can assemble a RAG pipeline in weeks — vector database, embedding model, orchestration. What that pipeline lacks is precisely what the five requirements above describe: file-server connectors that respect permissions, difference-only re-indexing at archive scale, OCR and layout analysis for the scanned half of the estate, coverage reporting, and audit logging that satisfies a regulator. Those are not features to add later; they are the product. The build-vs-buy question for enterprise knowledge intelligence is really a question about who maintains that unglamorous 80%.
Where DSX IQ fits
DSX IQ is DSX Digital's enterprise knowledge intelligence platform, built on this architecture: it connects to the file servers your teams already use, indexes the whole document estate in place, keeps the corpus in working memory through retrieval, compression and rolling context, and returns cited answers with confidence and coverage — under permission-aware retrieval and a full audit trail. Ingestion and questions are never metered; licensing is per user with a per-indexed-document rate.
And because knowledge intelligence still makes model calls, IQ registers as a governed agent in DSX Guard, so the archive layer itself passes the same policy controls as every other path to AI in the organisation. The DSX IQ use cases show corpus-level questions answered against a real document estate, and the retrieval, deployment and audit controls underneath them are set out on the security page.