RAG Drift Monitor
Compare retrieval distances before and after changing a small document corpus.
This experiment compares eight retrieval queries before and after a corpus change. The setup uses 19 Wikipedia articles, the all-MiniLM-L6-v2 embedding model, and ChromaDB. Four articles were replaced before the second run.
The saved report contains the distances, source documents, and thresholds from April 12, 2026.
Compare the runs
For each query, subtract the baseline distance from the changed-corpus distance:
delta = drifted_dist - baseline_distThe report marks a delta of at least 0.05 as degraded and at least 0.10 as requiring review. Those thresholds belong to this experiment.
The saved results contain six healthy queries, one degraded query, and one requiring review. Four queries changed source documents. Two of those remained below the degraded threshold. The other four queries have zero distance change.
Read the result carefully
A changed source can matter even below the distance threshold. Review the retrieved passage alongside the score.
The report measures retrieval changes. It contains no generated-answer evaluation, relevance labels, or production traffic. It cannot establish an answer-quality improvement or a general false-positive rate.
Inspect the report
The Python summary script recalculates the states from the saved distances:
python summarize-drift.py report.jsonKeep the embedding model, distance metric, chunking, and query set fixed when comparing corpus revisions. Changes to those settings need their own baseline.
Technologies: Python, sentence-transformers, ChromaDB, Streamlit.
Results
Distances from the saved April 12, 2026 report comparing eight queries across corpus revisions.
| Query | Baseline | Changed | Delta | State |
|---|---|---|---|---|
| How does attention work in transformer models? | 0.3287 | 0.3907 | 0.0620 | degraded |
| What causes hallucinations in large language models? | 0.3329 | 0.4344 | 0.1015 | review required |
| How does retrieval-augmented generation improve accuracy? | 0.4529 | 0.4618 | 0.0089 | healthy |
| What is the difference between dense and sparse retrieval? | 0.4989 | 0.5276 | 0.0287 | healthy |
| How are word embeddings trained? | 0.3239 | 0.3239 | 0.0000 | healthy |
| What is cosine similarity used for in semantic search? | 0.2672 | 0.2672 | 0.0000 | healthy |
| How does fine-tuning differ from pretraining? | 0.3479 | 0.3479 | 0.0000 | healthy |
| What are the main challenges in information retrieval evaluation? | 0.4356 | 0.4356 | 0.0000 | healthy |
Related code
- Summarize retrieval drift — Recalculate query states from the distances in a saved retrieval report.