Model Routing Middleware
Notes on a FastAPI router that selects a model and records fallback requests.
This prototype routes model requests by task type and keeps a fallback path for failed attempts. The work covers request classification, route logging, and cost accounting.
No benchmark or runnable router is published on this page.
Request handling
A useful request record includes the selected model, routing rule, input and output token counts, latency, retries, and final result. Record fallback attempts under the same request ID so their cost remains visible.
Separate API failures from task failures. A timeout or rate limit can trigger a retry. A syntactically valid response may still fail the task’s schema or acceptance checks.
Set an attempt limit and an overall deadline. Exhausting either should produce a visible failure instead of another unbounded retry.
Evaluate the routing rule
Use the same task set for a fixed-model baseline and the router. Include routine tasks, ambiguous requests, malformed inputs, and provider failures.
Compare successful-task cost, total latency, fallback frequency, and task acceptance rate. Include every attempt in the cost total.
A prompt-length threshold alone provides little information about task difficulty. Routing rules need task-specific results before they can support a savings claim.
Logging
Keep request IDs and operational measurements by default. Raw prompts and responses may contain sensitive data; their retention and access need a separate decision.
Technologies: Python, FastAPI, Model routing.