All case studies
Case study 04AI SystemsAutomation

LLM Evaluation & Cost-Aware Routing

Evaluation suites and model routing that cut inference cost 28% without lowering quality.

Role
Senior engineer · evaluation and model routing
Core stack
LangSmithLangGraphPythonModel routingContext engineering
−28%
Inference cost from context budgets and model routing
3
Gates on every change: completion, tools, grounding
p95
Latency and tool failures tracked in production

Overview

The quality and cost layer behind the agent platform: every change is scored on fixed evaluation sets, failure modes are rehearsed on purpose, and each request is routed to the smallest model that meets the bar.

Prompt, tool, and model changes could quietly break behavior that worked the day before, and sending every request to the largest model was expensive. The team needed evidence before shipping and a way to cut cost without trading away quality.

Product walkthrough

The flow, screen by screen

Routing policy
01 / 02

Routing policy. Each task type maps to a model tier and a context budget. Cost and quality are measured on the regression suite before a rule ships.

ArchitectureCorePlatformAsync / control
shipsPull requestprompt · tool · modelGitHub Actionsruns suitesLangSmith suitesfixed datasetsRelease gatepass or blockRequesttask typeModel routertask → tierContext budgethistory · passagesModel tierssmall · medium · largeMonitoringp95 · cost/caseFailure injection
Changes pass LangSmith regression suites before release; in production a router picks the model tier and context budget, while nightly failure injection rehearses recovery.

Approach

  1. 01Established LangSmith regression suites for task completion, tool selection, and grounding, run on every pull request against fixed datasets.
  2. 02Added failure-injection tests for tool timeouts, duplicate deliveries, malformed tool output, rejected approvals, and lost workers.
  3. 03Introduced context budgets that trim history and retrieved passages to what each task actually needs.
  4. 04Routed requests by task type, sending lookups and classification to smaller models and reserving larger ones for multi-step work.
  5. 05Tracked p95 latency, tool failures, and cost per case in production.

Key decisions

Gate releases on evaluations

A fixed dataset turns “it looks better” into a number that can block a merge.

Route by task, verify by evaluation

Every routing rule is checked against the regression suite before it takes real traffic, with a fallback to the next tier.

Break things on purpose

Injected failures prove that retries, idempotency, and recovery paths work before a real incident tests them.

Outcomes

  • Cost came down while task completion and grounding held steady on the same datasets.
  • Recovery paths are exercised every night instead of during incidents.
  • LangSmith
  • LangGraph
  • Python
  • Model routing
  • Context engineering
  • GitHub Actions
  • PostgreSQL

Working on something similar?

I work with clients worldwide, across any time zone. Tell me what you're building.

Send a message Email meMore case studies