Learn · RAG: chunking, embeddings & vector search · CCAR-F
RAG citations and evaluation
A finance policy assistant end to end: cited answers, permission to say "not found", and a test set that measures retrieval and answers.
Key ideas
- Give Claude retrieved chunks as document blocks with citations enabled, so every claim points to its source passage.
- Instruct Claude to answer only from the documents and to say when the answer isn't there.
- Evaluate retrieval (recall@k, precision/MRR) and generation (faithfulness, correctness, refusal accuracy) separately.
- Build a test set of 50-200 real questions with known source passages, including some that are unanswerable.
- Re-run the test set whenever chunking, embeddings or the reranker change.
- Pick the vector store that fits the existing stack (pgvector on Postgres); chunking, embeddings and reranking matter more.
Worked example: The finance policy assistant, decision by decision
The corpus is about 3,000 pages, too big for one prompt. Each choice below has a reason you can defend in an interview.
Exam traps
- A test set with only answerable questions, which never checks whether the system refuses when it should.
- Judging only the final answer, so retrieval failures look like model failures.
- Asking Claude to cite sources in free text instead of passing documents with citations enabled.
- Changing chunking or the embedding model without re-running the evaluation.