The short answer
Pick one question whose answer you can locate in the document. Inspect the extracted text and retrieved passages, then paste the correct passage directly into a fresh chat. If that works, improve the document or retrieval pipeline before switching to a larger chat model.
Read the essentials below. Open Technical detail whenever you want to go deeper.Separate the four jobs
Retrieval-augmented generation, or RAG, typically extracts text, divides it into searchable passages, retrieves likely matches and supplies those matches to the chat model. An embedding model helps search; the chat model writes the answer. They have different jobs.
For a short document that comfortably fits the available context, direct inclusion is also worth testing. A large library usually requires selecting evidence. Neither approach guarantees that every relevant fact will be used.
| Checkpoint | A useful question |
|---|---|
| Extracted text | Are the correct words, numbers and table headers present? |
| Stored passage | Is the answer still connected to its heading, units and qualifiers? |
| Retrieved results | Does the correct passage appear for the actual question? |
| Final prompt and answer | Was that passage included, and does the answer accurately cite it? |
Technical detailDebug a table without reprocessing the whole library
Take two or three representative pages first. Compare the original with the extracted text line by line around the failed answer. Check column order, repeated headers, decimal points, units, footnotes and text split across page boundaries.
A parser such as Docling supports layout, table structure and OCR, but support is not proof that a particular page converted correctly. Use OCR for image-only content when needed and inspect its output. Preserve page references so you can return to the original.
For a made-up equipment table, the chunk “X7 | 40 | 8” is hard to interpret. Keeping the title and headers—“X7 pump | flow: 40 L/min | pressure: 8 bar”—preserves the relationships. Do not invent missing labels or silently rewrite ambiguous source data.
Use the correct passage as a control test
Paste the answer-bearing passage into a new chat and ask the same question. Request an answer based only on that passage, with the supporting sentence or page reference. If this succeeds while the library query fails, you have evidence to investigate retrieval.
If it still fails, inspect ambiguity, the model’s reading ability, instructions and output limits. Also verify the expected answer: sometimes the document has several revisions or the question assumes information that is absent.
- Collect ten questions with manually checked source passages and page references.
- Include exact identifiers, paraphrased questions, a table value and a question needing two passages.
- Add a few questions the documents cannot answer; expect an explicit statement that the evidence is missing.
- Inspect retrieved results before scoring generated answers. Record which stage fails.
Technical detailMeasure retrieval separately from answer quality
For a simple diagnostic, track whether a sufficient answer-bearing passage appears in the top k retrieved results. If it appears for 7 of 10 answerable questions, your hit rate at that k is 70%. This is a small local check, not a public benchmark or a guarantee of correctness.
Track multi-passage questions separately: finding one of two necessary passages is incomplete. Then score whether the final answer is correct, supported and cites the right source. A retrieval hit and a grounded answer are distinct outcomes.
Keep the same questions while changing one setting. Record retrieval time and total answer time too. More candidates or a reranker may improve coverage while making the workflow less pleasant; measure the tradeoff.
Tune the stage that actually failed
If extraction fails, fix the parser or the source. If chunks lose meaning, preserve headings and table relationships and try a different chunk size with modest overlap. There is no chunk size that suits every document.
Check the embedding model’s supported language, input length and query instructions. Rebuild the document embeddings when switching embedding models; vectors from different models do not automatically share a meaningful space.
If exact product codes or names are missed, test keyword plus vector search. If the right passage is retrieved but ranked too low, test a reranker. Open WebUI supports hybrid retrieval; Sentence Transformers documents the retrieve-then-rerank pattern. Keep each change only if your checks improve.
Finally, ensure the selected evidence fits the prompt budget. Adding more passages can introduce duplicates, contradictions and extra processing. A citation link is useful only if the linked passage supports the answer.
Technical detailKeep a fully local pipeline local
Check each component separately: document parsing or OCR, embeddings, vector storage, reranking and chat generation. A local chat model does not establish where the other steps run. Use local providers for every stage if offline operation is the requirement.
Download the required models first, then test the complete workflow without a network connection. Account for parser and embedding jobs competing for memory with the chat model; processing a library in advance can simplify the interactive workload.
Keep retrieved documents as evidence, not instructions to the application. Text inside a PDF may contain misleading commands. For ordinary document Q&A, avoid giving the answering model unnecessary tools or actions.
From the community
LocalLLaMA threads describe document libraries that cannot find a known topic and long PDFs whose tables are difficult to parse. The community repeatedly points to extraction and retrieval. The workflow below tests that diagnosis instead of assuming every bad answer has the same cause.
- LocalLLaMA: RAG that actually works? (December 2025)
- LocalLLaMA: beginner help with table-heavy PDFs (January 2026)
These discussions informed the questions. Technical guidance is checked against the primary references; examples and checklists are editorial synthesis, not measured benchmarks.
Make room for the evidence.
Understand how documents and conversation history share the model’s context budget.
Understand context memory