Case study · Retrieval & evaluation

creative-rag

Hybrid retrieval over a film-craft corpus that verifies every citation after generation — a second entailment pass per claim, so it fails loudly instead of hallucinating.

The funnel
  1. 01Query
  2. 02Dense (bge-small) + BM25, in parallel
  3. 03RRF fusion → top-30
  4. 04Cross-encoder rerank (bge-reranker) → top-6
  5. 05Grounded generation
  6. 06Per-claim citation verification
  7. 07Cited answer — or an honest “the corpus doesn’t cover this”
Measured, CI-gated
1.0retrieval hit@6 on the eval suite
0.94mean reciprocal rank
1.0answer faithfulness
40tests, green in CI — retrieval quality is a regression gate on every push
How it works
  • Markdown-aware, heading-based chunking of the craft corpus; derived and non-craft files skipped at ingest.
  • Embedding and reranking run locally via sentence-transformers — no external API in the retrieval path.
  • Citation check is a genuine second LLM pass per claim (entailment), not a prompt instruction.
  • CI ingests a committed sample corpus and gates the retrieval metrics on every push — crag-eval --gate exits nonzero on regression.
  • FastAPI service + CLI + eval harness; Docker image with models baked in; Terraform to AWS ECS Fargate.
  • When the corpus lacks an answer, it says so — the worked example openly returns "no beach-specific lighting recipe in the notes."
  • Consumed by studio-mcp's craft_lookup — the knowledge layer of the studio pair.