Skip to content

DVC

DVC (Data Version Control) is a version control system for machine learning projects. This sheet covers data versioning, pipelines, metrics, and experiment management.

Table of Contents

Installation / Setup

# Install DVC
pip install dvc

# Initialize DVC in git repo
dvc init

# Add remote storage (S3, GCS, Azure, etc.)
dvc remote add -d myremote s3://mybucket/dvc
dvc remote modify myremote access_key_id YOUR_KEY
dvc remote modify myremote secret_access_key YOUR_SECRET

Core Syntax

Data Versioning

# Track a file or directory
dvc add data/training.csv
dvc add data/

# Commit to git (DVC stores .dvc files in git)
git add data/training.csv.dvc data/.gitignore
git commit -m "Add training data"

# Pull data from remote
dvc pull

# Push data to remote
dvc push

# Checkout specific version
git checkout HEAD~3
dvc checkout

Pipelines

# Define stages in dvc.yaml
# (see Practical Examples for full yaml)

# Reproduce pipeline
dvc repro

# Reproduce specific stage
dvc repro prepare

# Show pipeline DAG
dvc dag

# Force re-run
dvc repro --force

Metrics

# Add metrics
dvc metrics add metrics.json

# Show metrics
dvc metrics show

# Compare metrics across commits
dvc metrics diff HEAD~3

# Add plots
dvc plots add confusion_matrix.png

# Show plots
dvc plots show

Experiments

# Run experiment
dvc exp run

# Run experiment with custom params
dvc exp run -S train.lr=0.001

# Show experiments
dvc exp show

# Compare experiments
dvc exp diff

# Push experiments
dvc exp push origin experiment_name

# Pull experiments
dvc exp pull origin experiment_name

Common Operations

dvc.yaml Pipeline

# dvc.yaml
stages:
  prepare:
    cmd: python prepare.py
    deps:
      - prepare.py
      - data/raw
    outs:
      - data/prepared

  train:
    cmd: python train.py
    deps:
      - train.py
      - data/prepared
    params:
      - train.lr
      - train.epochs
    outs:
      - model.pkl
    metrics:
      - metrics.json:
          cache: false

  evaluate:
    cmd: python evaluate.py
    deps:
      - evaluate.py
      - model.pkl
      - data/prepared
    metrics:
      - metrics.json:
          cache: false

params.yaml

# params.yaml
prepare:
  val_size: 0.2
  test_size: 0.1

train:
  lr: 0.001
  epochs: 10
  batch_size: 32

evaluate:
  threshold: 0.5

Tracking Files

# Track output files (ignored by git)
dvc add model.pkl
dvc add data/prepared/

# Add to .gitignore automatically
git add .gitignore
git commit -m "Update .gitignore"

# Track with DVC
git add model.pkl.dvc
git commit -m "Add model"

Practical Examples

Example 1 — Full ML pipeline

# dvc.yaml
stages:
  prepare:
    cmd: python src/prepare.py
    deps:
      - src/prepare.py
      - data/raw
    outs:
      - data/prepared/train
      - data/prepared/test

  featurize:
    cmd: python src/featurize.py
    deps:
      - src/featurize.py
      - data/prepared
    outs:
      - data/features

  train:
    cmd: python src/train.py
    deps:
      - src/train.py
      - data/features
    params:
      - train
    outs:
      - model.pkl
    metrics:
      - metrics.json:
          cache: false

  evaluate:
    cmd: python src/evaluate.py
    deps:
      - src/evaluate.py
      - model.pkl
      - data/features
    metrics:
      - metrics.json:
          cache: false

# Run full pipeline
dvc repro

# Show metrics
dvc metrics show

# Compare with previous run
dvc metrics diff HEAD~1

Example 2 — Experiment tracking

# Run experiment with different params
dvc exp run -S train.lr=0.01
dvc exp run -S train.lr=0.001
dvc exp run -S train.lr=0.0001

# Show all experiments
dvc exp show

# Compare experiments
dvc exp diff exp-1 exp-2

Example 3 — Data versioning workflow

# Initial data
dvc add data/training.csv
git add data/training.csv.dvc
git commit -m "Add initial training data"

# Update data
cp new_data.csv data/training.csv
dvc add data/training.csv
git add data/training.csv.dvc
git commit -m "Update training data v2"

# Checkout old version
git checkout HEAD~1
dvc checkout
# Now data/training.csv is back to v1

Common Patterns

Pattern: reproducible experiments

# 1. Track code changes
git add src/
git commit -m "Update model architecture"

# 2. Track data changes
dvc add data/
git add data/.dvc
git commit -m "Update dataset"

# 3. Run pipeline
dvc repro

# 4. Commit results
git add metrics.json dvc.lock
git commit -m "Update metrics"

Pattern: collaborative workflows

# Pull latest code and data
git pull
dvc pull

# Make changes
# ... edit code ...

# Run and commit
dvc repro
git add .
git commit -m "Update experiment"
git push
dvc push

Pattern: hyperparameter tuning

# Run multiple experiments
for lr in 0.1 0.01 0.001; do
    dvc exp run -S train.lr=$lr
done

# Show results
dvc exp show --sort-by metrics.accuracy

Best Practices

  • Track everything — code in git, data/models in DVC.
  • Use .dvc files — they're small and git-friendly.
  • Use remote storage — S3, GCS, Azure for large files.
  • Define pipelines — reproducible, documented workflows.
  • Track metrics — version control for evaluation results.
  • Use params.yaml — separate config from code.
  • Tag important runs — git tag v1.0 for releases.
  • Use .gitignore — DVC manages it automatically.

Common Mistakes

Mistake Fix
Not tracking data Use dvc add for large files
Mixing git and DVC Code in git, data in DVC
Not using remote Always add remote storage
Hardcoding paths Use dvc.yaml deps and outs
Not versioning params Use params.yaml

Troubleshooting

Symptom Cause / Fix
Data not found Run dvc pull; check remote
Pipeline not reproducible Check deps; check params
Metrics not tracked Check dvc.yaml metrics section
Remote not configured Run dvc remote add
Lock file conflict Run dvc repro to update

Official Documentation

Last Updated

2026-09-06 — Initial version.