← Code

Summarize retrieval drift

Recalculate query states from the distances in a saved retrieval report.

Tested against the saved report and malformed inputs.

summarize-drift.py
#!/usr/bin/env python3
"""Summarize a saved retrieval-drift report. Standard library only; no network I/O."""
import argparse
import json
import math
import sys
from pathlib import Path
def summarize(report):
if not isinstance(report, dict) or not isinstance(report.get("queries"), list):
raise ValueError("Report must contain a queries array")
thresholds = report.get("thresholds", {})
degraded = thresholds.get("degraded", 0.05)
review = thresholds.get("review", 0.10)
if any(isinstance(n, bool) or not isinstance(n, (int, float)) or not math.isfinite(n)
for n in [degraded, review]) or not 0 <= degraded <= review:
raise ValueError("Thresholds must be finite numbers: 0 <= degraded <= review")
counts = {"healthy": 0, "degraded": 0, "review_required": 0}
for index, query in enumerate(report["queries"]):
if not isinstance(query, dict):
raise ValueError(f"Query {index + 1} must be an object")
distances = [query.get("baseline_dist"), query.get("drifted_dist")]
if any(isinstance(n, bool) or not isinstance(n, (int, float)) or not math.isfinite(n)
for n in distances):
raise ValueError(f"Query {index + 1} requires finite numeric distances")
delta = distances[1] - distances[0]
state = "review_required" if delta >= review else "degraded" if delta >= degraded else "healthy"
counts[state] += 1
return {"query_count": len(report["queries"]), "thresholds": {"degraded": degraded, "review": review}, "summary": counts}
def main():
parser = argparse.ArgumentParser(description=__doc__)
parser.add_argument("report", type=Path, help="Path to a saved JSON report")
args = parser.parse_args()
try:
result = summarize(json.loads(args.report.read_text(encoding="utf-8-sig")))
except (OSError, ValueError, TypeError) as error:
print(f"Cannot summarize report: {error}", file=sys.stderr)
return 2
print(json.dumps(result, indent=2))
return 0
if __name__ == "__main__":
sys.exit(main())

Run the summary

Download the example report and the script:

Terminal window
python summarize-drift.py report.json

The example produces eight queries: six healthy, one degraded, and one requiring review. Output is JSON on stdout. The input file remains unchanged.

Thresholds

The script computes drifted_dist - baseline_dist for each query. It checks the review threshold first, then the degraded threshold. The defaults are 0.10 and 0.05; values in the report override them.

Saved state labels are ignored. A negative delta is classified as healthy under this rule. A source-document change does not independently change the state.

These thresholds summarize this experiment. They have not been calibrated for other retrieval systems.

Errors

Missing files, invalid JSON, invalid thresholds, and nonnumeric distances produce an error on stderr and exit code 2. A successful run exits with code 0.

The script uses Python’s standard library. It does not load an embedding model or connect to a vector database.

Parameters

NameDefaultPurpose
reportRequiredPath to the report JSON.

Requirements

Requires
  • Python 3.9 or later.
  • A JSON report with a queries array and baseline_dist and drifted_dist on each entry.
Permissions
Read access to the report file. No network access required.
Testing
Checked against the saved eight-query report and malformed-input fixtures. No model or database is called.
Version
1.0.0