When you need this
- Cannot reproduce last quarter’s model
- Training data lives on someone’s laptop path
- Need lineage for EU evidence packs
- Retrain jobs silently use newer dirty data
Prerequisites
- Remote storage (S3/GCS/Azure Blob) for DVC cache
- Git repo for code + .dvc pointers
- One critical training pipeline to pilot
Tools
Trial once datasets leave laptops; pair with registry for model promotion.
Start with registry + experiment tracking before full deployment automation.
Trial before auto-retrain; bad data amplification is the #1 silent failure mode.
Steps
- 1
Track the critical dataset
dvc add the training/validation sets; push to remote. Commit .dvc files. Never commit raw blobs to Git.
- 2
Define the pipeline
dvc.yaml stages: prepare → train → evaluate. Parameters in params.yaml. Outputs include metrics and model artifact.
- 3
Gate data quality
Great Expectations (or similar) before train stage. Fail the pipeline on schema/drift violations.
- 4
Link to MLflow
Log DVC data hash / git commit as MLflow tags on every run. Registry promotion requires the hash present.
- 5
CI reproducibility check
On PR, dvc repro (or pull + verify) for the affected pipeline. Block merge if metrics drift beyond threshold without intent.
Adoption pitfalls
- DVC remote with no access control
- Tracking everything before one pipeline works
- Forgetting to push cache → broken CI on other machines
Adoption checklist
- Production models tagged with data hash
- Remote cache backed up / lifecycle policy set
- Data quality gate before train
- Someone can reproduce last prod train from Git + DVC
SEER REAL assessment / sprint
Assessment asks for lineage on the last prod model. Sprint versions one dataset + pipeline with DVC and wires hashes into MLflow.