Retrieval augmented generation is a simple idea with a lot of places to go wrong. Find the relevant passages, put them in the prompt, ask the model to answer from them. The quality of the answer is mostly decided before the model is involved.
Chunking is a structure problem
The default approach splits documents into fixed length windows with some overlap. It is easy and it is usually wrong.
A contract clause split across two chunks loses its meaning in both. A table separated from its heading becomes a grid of numbers with no context. A procedure cut in the middle returns half the steps, which is more dangerous than returning none.
Chunk on the document structure instead. Headings, clauses, sections, list items, table rows with their header. A chunk should be a thing a person would quote, because that is what the model is going to quote.
Where a chunk is small, carry its context with it: the document title, the section path, the effective date. A clause that reads "this does not apply in the circumstances described above" is useless without knowing what was above it.
Metadata is what makes retrieval usable
Every chunk should carry the identifier of its source document, the section it came from, the document type, the effective date, the version, and whatever access classification applies.
That metadata does four jobs. It lets you filter before searching, which is faster and more accurate than searching everything and hoping. It lets the answer cite a source the reader can open. It lets you show how current the information is, which matters enormously in a document set where policies get superseded. And it is what permission filtering operates on.
A retrieval system without metadata can find text. It cannot tell you where it came from or whether it is still true.
Permissions before retrieval, always
Filtering after generation is too late. If a restricted passage reaches the generation prompt, it can influence the answer even if the citation is removed. A search engine may have internal candidate stages, but restricted text must not cross into the model context, user response or an unauthorised cache.
Apply the filter as part of the query. The user identity determines which documents are in scope, and the search runs only over those. This is straightforward when the metadata carries the classification and the source system exposes the access rules, and it is the reason both of those matter.
The failure to avoid is a system where an employee can ask a question and receive a paraphrase of a document they could not open directly. It is the same disclosure, laundered through a model.
Hybrid search beats either method alone
Semantic search finds passages that mean something similar. It handles the case where the document says termination and the user asked about cancellation.
Keyword search finds exact terms. It handles part numbers, error codes, proper nouns, invoice references and anything else where the exact string is the point and a similar embedding is useless.
Evaluate both against the actual question set and combine them where that improves retrieval. Do not assume an equal split between exact and semantic questions, or that combining rankings always improves every query.
Freshness is a design decision
Every source needs an update path. Event driven where the system emits changes, scheduled where it does not, with the interval chosen from how often the content actually changes.
Superseded documents are the subtle problem. If a policy is replaced and the old version remains in the index, the system will confidently cite last year rules. Either remove superseded versions or mark them and filter them out by default, and show the effective date on every citation so a reader can see what they are being told.
Evaluate with real questions
Collect the questions people actually ask, with the passage that should be retrieved and the answer that should result. Thirty to fifty is enough to be useful.
Measure retrieval separately from generation. If the right passage was not retrieved, no prompt change will fix the answer, and teams waste weeks tuning prompts for a retrieval problem. Checking whether the correct chunk appears in the top results tells you immediately which half of the system to work on.
The order of work
Inventory the documents and their access rules. Decide chunking from the structure. Attach metadata. Build permission filtered hybrid search. Assemble an evaluation set. Only then worry about which model generates the answer.
Choosing the generator first leaves those information management decisions unresolved.
Test a revoked document, not just a useful answer
Create a test account that can read one procedure but not a restricted appendix. Ask a question whose answer is available only in the appendix. The system should acknowledge the information gap without quoting or paraphrasing the restricted material. Repeat the test after a user loses access, and after a document changes its classification.
This exposes a separate design issue: cached answers may outlive the permissions used to create them. A cache key needs enough scope to prevent sharing an answer between users with different access. In some applications that means caching retrieved identifiers and checking access again before loading text. In others, sensitive answers should not be cached at all. Choose from the data classification rather than treating caching as a universal improvement.
Next replace a procedure with a revised version. Verify that the update job removes obsolete chunks, that citations open the replacement, and that the answer explains any date boundary relevant to the question. A successful indexing job count does not demonstrate any of those outcomes.
Finally test the absence of a source. A question that nobody has documented should produce a clear limit, not a polished guess. Track these unanswered questions separately. They can tell the operations team where a missing procedure is the real problem.