Hover a node · click to focus · ←/→ steps
Labels on every GPU job + gateway tags on every token. FOCUS-aligned showback across AWS / Azure / GCP.
When you need this
- GPU cluster costs spike without owner
- Platform team blamed for all cloud spend
- Need chargeback for internal ML products
- Idle training jobs burn budget overnight
Prerequisites
- Kubernetes with GPU nodes
- Label policy agreed with finance (team, model, env, cost_center)
- Prometheus or equivalent metrics path
Tools
Requires consistent K8s labels before numbers are trustworthy.
Assess for platform teams; less critical for app-only ML teams.
Steps
- 1
Define mandatory labels
Require team, model, environment, cost_center on every GPU workload. Enforce with admission controller or OPA — unlabeled pods fail to schedule on GPU pools.
- 2
Install OpenCost / Kubecost
Scrape allocation metrics and map GPU hours to namespaces and labels. Separate training vs inference pools in the dashboard.
- 3
Weekly showback ritual
Top 5 spend drivers with team leads. Right-size or power off idle dev GPUs off-hours. Track idle % as a first-class KPI.
- 4
Optional: Infracost on infra PRs
Preview GPU node pool changes in Terraform PRs so platform cannot silently grow capacity without cost visibility.
- 5
Align with FOCUS export
Map Kubecost dimensions to FOCUS columns so cloud + GPU story matches the FinOps common schema.
Adoption pitfalls
- Tool installed before label policy → garbage attribution forever
- Shared namespaces with no team label
- Ignoring spot / preemptible savings for non-prod training
Adoption checklist
- 100% GPU workloads have team and model labels
- Monthly showback deck automated
- Idle GPU policy documented and enforced
- Budget owner assigned per namespace
SEER REAL assessment / sprint
Assessment finds unlabeled GPU spend and missing owners. Sprint work usually ships label policy + Kubecost + one showback meeting cadence.