Hover a node · click to focus · ←/→ steps
Apps → tagged gateway → AWS / Azure / GCP providers or self-host. Cost tags feed FinOps; sampled traffic closes the eval loop.
When you need this
- APM green but answers getting worse
- Cannot trace RAG latency across services
- Incident response starts from user screenshots
- Need GenAI semantic conventions across teams
Prerequisites
- OTel collector
- Backend (Jaeger, Grafana Tempo, etc.)
- Agreement on GenAI attribute names
Tools
Instrument gateway and app tier first; expand to training jobs later.
Self-host for regulated workloads; cloud for faster pilot.
Strong OSS alternative or complement to Langfuse; evaluate self-host ops.
Steps
- 1
Instrument gateway and app tier
Add GenAI semantic conventions: model name, token counts, retrieval latency, error types, team tags. Propagate trace IDs through LiteLLM.
- 2
Define SLOs on model signals
p95 latency, error rate, weekly eval score — not only HTTP 200. Separate SLOs for retrieval vs generation.
- 3
Correlate with product metrics
Join traces to ticket volume, conversion, or CSAT drops for incident triage.
- 4
Alert on drift, not only outages
Pair OTel metrics with Langfuse/Phoenix/Evidently eval alerts. Page on faithfulness or latency drift, not just 5xx.
Adoption pitfalls
- Logging full prompts to APM without redaction
- No sampling policy → cost explosion
- Traces without team tags → useless for FinOps
Adoption checklist
- Every LLM request has a trace ID
- Dashboard shows tokens and latency by model
- On-call runbook links traces to eval failures
- SLO review in weekly ops meeting
SEER REAL assessment / sprint
Assessment checks whether quality signals exist beyond uptime. Sprint typically instruments the gateway + one RAG app and ships a model SLO dashboard.