Summarize retrieval drift
Recalculate query states from the distances in a saved retrieval report.
Tested against the saved report and malformed inputs.
#!/usr/bin/env python3"""Summarize a saved retrieval-drift report. Standard library only; no network I/O."""import argparseimport jsonimport mathimport sysfrom pathlib import Path
def summarize(report): if not isinstance(report, dict) or not isinstance(report.get("queries"), list): raise ValueError("Report must contain a queries array") thresholds = report.get("thresholds", {}) degraded = thresholds.get("degraded", 0.05) review = thresholds.get("review", 0.10) if any(isinstance(n, bool) or not isinstance(n, (int, float)) or not math.isfinite(n) for n in [degraded, review]) or not 0 <= degraded <= review: raise ValueError("Thresholds must be finite numbers: 0 <= degraded <= review") counts = {"healthy": 0, "degraded": 0, "review_required": 0} for index, query in enumerate(report["queries"]): if not isinstance(query, dict): raise ValueError(f"Query {index + 1} must be an object") distances = [query.get("baseline_dist"), query.get("drifted_dist")] if any(isinstance(n, bool) or not isinstance(n, (int, float)) or not math.isfinite(n) for n in distances): raise ValueError(f"Query {index + 1} requires finite numeric distances") delta = distances[1] - distances[0] state = "review_required" if delta >= review else "degraded" if delta >= degraded else "healthy" counts[state] += 1 return {"query_count": len(report["queries"]), "thresholds": {"degraded": degraded, "review": review}, "summary": counts}
def main(): parser = argparse.ArgumentParser(description=__doc__) parser.add_argument("report", type=Path, help="Path to a saved JSON report") args = parser.parse_args() try: result = summarize(json.loads(args.report.read_text(encoding="utf-8-sig"))) except (OSError, ValueError, TypeError) as error: print(f"Cannot summarize report: {error}", file=sys.stderr) return 2 print(json.dumps(result, indent=2)) return 0
if __name__ == "__main__": sys.exit(main())Run the summary
Download the example report and the script:
python summarize-drift.py report.jsonThe example produces eight queries: six healthy, one degraded, and one requiring review. Output is JSON on stdout. The input file remains unchanged.
Thresholds
The script computes drifted_dist - baseline_dist for each query. It checks the review threshold first, then the degraded threshold. The defaults are 0.10 and 0.05; values in the report override them.
Saved state labels are ignored. A negative delta is classified as healthy under this rule. A source-document change does not independently change the state.
These thresholds summarize this experiment. They have not been calibrated for other retrieval systems.
Errors
Missing files, invalid JSON, invalid thresholds, and nonnumeric distances produce an error on stderr and exit code 2. A successful run exits with code 0.
The script uses Python’s standard library. It does not load an embedding model or connect to a vector database.
Parameters
| Name | Default | Purpose |
|---|---|---|
report | Required | Path to the report JSON. |
Requirements
- Requires
- Python 3.9 or later.
- A JSON report with a queries array and baseline_dist and drifted_dist on each entry.
- Permissions
- Read access to the report file. No network access required.
- Testing
- Checked against the saved eight-query report and malformed-input fixtures. No model or database is called.
- Version
- 1.0.0