LLM Gateways Can Treat Failover HTTP 200 as Success Despite Incorrect Responses

That 200 OK From Your LLM Gateway Probably Means Nothing

Every AI gateway on the market — LiteLLM, Portkey, OpenRouter, Olla — checks the same things: HTTP status code, response time, token usage. If the backup provider returns HTTP 200 with valid JSON, the gateway declares success. But HTTP 200 only tells you the request completed. It says nothing about whether the response is correct. The Silent Failure Pattern In production monitoring across multi-provider setups, a consistent pattern emerges during failover events: Primary provider returns 5xx or times out Gateway routes to backup provider Backup returns HTTP 200 with complete, well-formed JSON Gateway declares success The response contains subtly wrong content — hallucinated entities, missing fields, contradictory reasoning The consuming agent or application continues processing bad data The gateway logs show "failover successful." Monitoring shows no errors. But the output is wrong. Why Existing Gateways Can't Catch This All major LLM gateways operate at the transport level: # Every gateway does this: def handle_failover(request, providers): for provider in providers: try: response = provider.complete(request) if response.status_code == 200: return response # "Success!" except Exception as e: log(f"Provider failed: {e}") continue # Try next Transport-level checks validate: ✅ HTTP status is 2xx ✅ Response is valid JSON ✅ Response time within threshold ✅ Token usage within limits What they don't validate: ❌ Does the response contain all required fields? ❌ Are field types and values correct? ❌ Does the output contradict itself? ❌ Are there hallucinated entities? ❌ Does the response actually answer the original request? What Response Validation Looks Like in Practice Instead of accepting any 200 OK, add a contract validation step after failover: from dataclasses import dataclass from typing import List, Optional import json @dataclass class ResponseContract: """Define what a valid response looks like.""" required_fields: List[str] forbidden_patterns: List[str] max_tokens: int require_json: bool = True field_constraints: dict = None def validate_response(response: dict, contract: ResponseContract) -> dict: """Validate response against contract. Returns validation result.""" issues = [] # 1. Structural checks (~45µs P50) for field in contract.required_fields: if field not in response: issues.append(f"Missing required field: {field}") # 2. Field type validation if contract.field_constraints: for field, expected_type in contract.field_constraints.items(): if field in response: if not isinstance(response[field], expected_type): issues.append(f"Field {field}: expected {expected_type.__name__}, got {type(response[field]).__name__}") # 3. Content pattern checks if isinstance(response.get("content", ""), str): content = response["content"] for pattern in contract.forbidden_patterns: if pattern.lower() in content.lower(): issues.append(f"Forbidden pattern found: {pattern}") return { "valid": len(issues) == 0, "issues": issues, "issue_count": len(issues), } # Usage example def validated_failover(request, providers, contract): """Failover with response validation.""" for provider in providers: try: response = provider.complete(request) result = validate_response(response, contract) if result["valid"]: return response else: log(f"Provider {provider.name}: contract validation failed - {result['issues']}") # Option: retry with next provider, or surface degradation except Exception as e: log(f"Provider {provider.name} error: {e}") raise AllProvidersFailed("All providers failed or produced invalid responses") This pattern adds 45µs P50 overhead (diagnostic engine microbenchmark, 70,000 fault injections across 7 failure types) — negligible compared to the 700-900ms of a typical LLM API call. Three Failure Categories That Slip Through HTTP 200 Based on the arXiv:2606.14589 taxonomy from a production LLM agent runtime: 1. Schema-Valid but Wrong The response is structurally perfect — all fields present, correct types, valid JSON. The content is just wrong. Example: You ask for pricing of GPT-4o. The backup model returns valid JSON with a plausible price that happens to be outdated or from a different model. Detection: Field-level constraints and cross-field validation (e.g., "model_name + price must match known pricing table"). 2. Contradictory Chain Outputs In multi-step agent workflows, each individual response looks fine — but the combination produces contradictions. Example: Step 1 says "user is in California." Step 3 says "applying NY state tax." Each response is independently valid. Detection: Stateful validation across the conversation context, checking for logical consistency between steps. 3. Hallucinated Grounding The response is coherent, well-structured, and cites sources — but the citations don't exist or don't support the claim. Example: An analysis that cites specific research papers, but the papers don't contain the claimed findings. Detection: Structured predicates that verify assertions against known reference data. Where to Add Validation The validation layer belongs in the proxy, not the application: ┌─────────────┐ ┌──────────────┐ ┌─────────────┐ │ Application │────▶│ Gateway │────▶│ Provider 1 │ │ (Agent/App) │ │ + Validation│ ├─────────────┤ └─────────────┘ │ Layer │ │ Provider 2 │ │ │ ├─────────────┤ │ After every │ │ Provider 3 │ │ response: │ └─────────────┘ │ 1. Validate │ │ 2. If fail → │ │ retry or │ │ flag │ └──────────────┘ Benefits of proxy-level validation: Zero application changes — legacy apps get validation for free Unified policy — one contract definition, enforced everywhere Failover-aware — validation is most critical exactly when failover happens Measurable — track validation pass/fail rates by provider The Benchmark That Matters When evaluating a gateway, add one more row to your comparison spreadsheet: Capability Any current gateway Should be standard Provider routing ✅ ✅ Failover ✅ ✅ Circuit breakers ✅ ✅ Rate limiting ✅ ✅ Cost tracking ✅ ✅ Response validation ❌ ✅ Required Semantic correctness ❌ ✅ Required The microsecond-level overhead (45µs P50, 102µs P99) makes this a no-brainer addition to the proxy layer. Try It Yourself The validation approach shown above is simplified for illustration. A production-grade implementation — with configurable contracts, multi-provider support, and MCP integration — is what we're building at Correctover. But the pattern itself is framework-agnostic. You can add response validation to any gateway today with < 100 lines of Python. References: arXiv:2606.14589 — "When Errors Become Narratives: A Longitudinal Taxonomy of Silent Failures in a Production LLM Agent Runtime" Microbenchmark: 70,000 fault injections, 7 fault types, diagnostic P50=45µs, P99=102µs Correctover.com

1 hour ago

Your AI Gateway's 200 OK Is Lying to You — A Practical Guide to Response Validation

Every AI gateway on the market — LiteLLM, Portkey, OpenRouter, Olla — checks the same things: HTTP status code, response time, token usage. If the backup provider returns HTTP 200 with valid JSON, the gateway declares success. But HTTP 200 only tells you the request completed. It says nothing about whether the response is correct. The Silent Failure Pattern In production monitoring across multi-provider setups, a consistent pattern emerges during failover events: Primary provider returns 5xx or times out Gateway routes to backup provider Backup returns HTTP 200 with complete, well-formed JSON Gateway declares success The response contains subtly wrong content — hallucinated entities, missing fields, contradictory reasoning The consuming agent or application continues processing bad data The gateway logs show "failover successful." Monitoring shows no errors. But the output is wrong. Why Existing Gateways Can't Catch This All major LLM gateways operate at the transport level: # Every gateway does this: def handle_failover(request, providers): for provider in providers: try: response = provider.complete(request) if response.status_code == 200: return response # "Success!" except Exception as e: log(f"Provider failed: {e}") continue # Try next Transport-level checks validate: ✅ HTTP status is 2xx ✅ Response is valid JSON ✅ Response time within threshold ✅ Token usage within limits What they don't validate: ❌ Does the response contain all required fields? ❌ Are field types and values correct? ❌ Does the output contradict itself? ❌ Are there hallucinated entities? ❌ Does the response actually answer the original request? What Response Validation Looks Like in Practice Instead of accepting any 200 OK, add a contract validation step after failover: from dataclasses import dataclass from typing import List, Optional import json @dataclass class ResponseContract: """Define what a valid response looks like.""" required_fields: List[str] forbidden_patterns: List[str] max_tokens: int require_json: bool = True field_constraints: dict = None def validate_response(response: dict, contract: ResponseContract) -> dict: """Validate response against contract. Returns validation result.""" issues = [] # 1. Structural checks (~45µs P50) for field in contract.required_fields: if field not in response: issues.append(f"Missing required field: {field}") # 2. Field type validation if contract.field_constraints: for field, expected_type in contract.field_constraints.items(): if field in response: if not isinstance(response[field], expected_type): issues.append(f"Field {field}: expected {expected_type.__name__}, got {type(response[field]).__name__}") # 3. Content pattern checks if isinstance(response.get("content", ""), str): content = response["content"] for pattern in contract.forbidden_patterns: if pattern.lower() in content.lower(): issues.append(f"Forbidden pattern found: {pattern}") return { "valid": len(issues) == 0, "issues": issues, "issue_count": len(issues), } # Usage example def validated_failover(request, providers, contract): """Failover with response validation.""" for provider in providers: try: response = provider.complete(request) result = validate_response(response, contract) if result["valid"]: return response else: log(f"Provider {provider.name}: contract validation failed - {result['issues']}") # Option: retry with next provider, or surface degradation except Exception as e: log(f"Provider {provider.name} error: {e}") raise AllProvidersFailed("All providers failed or produced invalid responses") This pattern adds 45µs P50 overhead (diagnostic engine microbenchmark, 70,000 fault injections across 7 failure types) — negligible compared to the 700-900ms of a typical LLM API call. Three Failure Categories That Slip Through HTTP 200 Based on the arXiv:2606.14589 taxonomy from a production LLM agent runtime: 1. Schema-Valid but Wrong The response is structurally perfect — all fields present, correct types, valid JSON. The content is just wrong. Example: You ask for pricing of GPT-4o. The backup model returns valid JSON with a plausible price that happens to be outdated or from a different model. Detection: Field-level constraints and cross-field validation (e.g., "model_name + price must match known pricing table"). 2. Contradictory Chain Outputs In multi-step agent workflows, each individual response looks fine — but the combination produces contradictions. Example: Step 1 says "user is in California." Step 3 says "applying NY state tax." Each response is independently valid. Detection: Stateful validation across the conversation context, checking for logical consistency between steps. 3. Hallucinated Grounding The response is coherent, well-structured, and cites sources — but the citations don't exist or don't support the claim. Example: An analysis that cites specific research papers, but the papers don't contain the claimed findings. Detection: Structured predicates that verify assertions against known reference data. Where to Add Validation The validation layer belongs in the proxy, not the application: ┌─────────────┐ ┌──────────────┐ ┌─────────────┐ │ Application │────▶│ Gateway │────▶│ Provider 1 │ │ (Agent/App) │ │ + Validation│ ├─────────────┤ └─────────────┘ │ Layer │ │ Provider 2 │ │ │ ├─────────────┤ │ After every │ │ Provider 3 │ │ response: │ └─────────────┘ │ 1. Validate │ │ 2. If fail → │ │ retry or │ │ flag │ └──────────────┘ Benefits of proxy-level validation: Zero application changes — legacy apps get validation for free Unified policy — one contract definition, enforced everywhere Failover-aware — validation is most critical exactly when failover happens Measurable — track validation pass/fail rates by provider The Benchmark That Matters When evaluating a gateway, add one more row to your comparison spreadsheet: Capability Any current gateway Should be standard Provider routing ✅ ✅ Failover ✅ ✅ Circuit breakers ✅ ✅ Rate limiting ✅ ✅ Cost tracking ✅ ✅ Response validation ❌ ✅ Required Semantic correctness ❌ ✅ Required The microsecond-level overhead (45µs P50, 102µs P99) makes this a no-brainer addition to the proxy layer. Try It Yourself The validation approach shown above is simplified for illustration. A production-grade implementation — with configurable contracts, multi-provider support, and MCP integration — is what we're building at Correctover. But the pattern itself is framework-agnostic. You can add response validation to any gateway today with < 100 lines of Python. References: arXiv:2606.14589 — "When Errors Become Narratives: A Longitudinal Taxonomy of Silent Failures in a Production LLM Agent Runtime" Microbenchmark: 70,000 fault injections, 7 fault types, diagnostic P50=45µs, P99=102µs Correctover.com

1 hour ago