Hover a node · click to focus · ←/→ steps
Apps → tagged gateway → AWS / Azure / GCP providers or self-host. Cost tags feed FinOps; sampled traffic closes the eval loop.
When you need this
- Three teams call OpenAI with separate keys and no central log
- LLM bill doubled with no owner
- Need Azure + OpenAI fallback without rewriting apps
- Finance wants showback by product and environment
Prerequisites
- Container runtime or K8s
- Provider API keys in a secret store (Vault / cloud SM)
- Team and environment taxonomy agreed with FinOps
Tools
Deploy as single ingress before adding a second LLM vendor.
Fast pilot; migrate to gateway + OTel for long-term control.
Assess for experimentation; production regulated stacks often prefer self-hosted LiteLLM + direct keys.
Steps
- 1
Deploy proxy as mandatory ingress
Point all apps to the LiteLLM OpenAI-compatible endpoint. Block direct provider calls at network or secret policy level — gateway is the only place keys live.
- 2
Configure team budgets and tags
Set budget limits per team tag. Pass team_id, product, and environment in request metadata from every service. Reject untagged requests in staging first, then prod.
- 3
Add routing and fallback
Define model groups: primary, cheap fallback, and EU-sovereign path when needed. Fail over on latency SLO breach, 429s, or budget threshold — not only hard errors.
- 4
Wire online eval sampling
Sample a % of traffic into Langfuse/Phoenix with the same tags so cost and quality share one dimension.
- 5
Export spend to FinOps
Ship usage logs to the warehouse or Helicone/OpenCost. Weekly showback: top teams, top models, cost per successful task.
Adoption pitfalls
- Leaving provider keys in app env ‘just for local’ — they leak to prod
- Tags optional → 40% of spend unattributable
- Routing by model name only without cost caps
Adoption checklist
- Zero direct provider calls from app code
- Every request tagged with team and environment
- Monthly spend report by team automated
- Rate limits and budget alerts configured
- Fallback path tested under provider outage drill
SEER REAL assessment / sprint
Assessment inventories shadow keys and spend owners. Sprint 1 usually stands up LiteLLM + tags + one FinOps dashboard; Sprint 2 adds fallbacks and eval sampling.