The Complete DVC Guide

This is part one of three. It covers everything you need to do real work with DVC, not a teaser. By the end you can version a dataset, reproduce a pipeline, share data with a colleague through remote storage, compare experiment metrics, and answer "which data produced this model?" with a commit hash. Mid-level and Senior take the same topics further; nothing here is thrown away.

Each section ends with a Try it task. Do them as you go. They take a few minutes each, and these ideas only stick once you have watched dvc repro skip a stage it did not need to run.

The problem DVC solves: code in Git, data elsewhere

Git is excellent at versioning text and hopeless at versioning data. Commit a 2 GB training set and you have permanently added 2 GB to every clone of that repository, forever, because Git stores full history. Change one row and you add another 2 GB.

So teams do the obvious thing and keep data out of Git. Then the real problem starts: the model in production was trained on something, and nobody can say what. The notebook says data/train_v3_final_fixed.csv, that file is on a laptop that was reimaged, and the person who made it left.

DVC's central trick is to store a small text pointer in Git and the actual bytes somewhere cheap. The pointer file is a few hundred bytes and contains a content hash. Git versions the pointer; DVC moves the bytes. One git checkout then brings back the exact code and tells DVC which data belongs to it.

That single idea gives you four things, and they are the reason DVC exists rather than a bespoke script:

You need little to follow along: Python, Git, and a folder. A tiny CSV and a five-line training script are the best place to start, because a broken pipeline costs you nothing.

Install and initialise

DVC is a Python package, and it sits on top of Git rather than replacing it.

pip install "dvc[s3]"        # or dvc[gs], dvc[azure], dvc[ssh], dvc[all]
dvc --version

The bracketed extra pulls in the driver for your remote storage. Plain pip install dvc works fine for local experimentation and then fails with a confusing "URL is not supported" the moment you configure an S3 remote, so install the extra you need up front.

git init                     # DVC needs a Git repository
dvc init
git status

dvc init creates a small amount of scaffolding and stages it for you:

git commit -m "Initialise DVC"

Your first tracked dataset, in four commands

This is the core loop. Four commands, and afterwards your data is versioned.

mkdir -p data
# put a file in data/raw.csv — any size, even 10 rows

dvc add data/raw.csv
git add data/raw.csv.dvc data/.gitignore
git commit -m "Track the raw dataset"

Now look at what happened, because this is the whole mental model:

cat data/raw.csv.dvc

outs:
- md5: 8f2c4b19e0a7d3f1c6b8a2e4d7091f3b
  size: 1048576
  hash: md5
  path: raw.csv

The .dvc file holds a hash, a size, and a path. The file itself is not in Git. data/.gitignore now contains /raw.csv, which DVC wrote for you, so Git will never try to store it.

cat data/.gitignore          # /raw.csv
git status                   # clean; the CSV is ignored

The bytes went into DVC's cache, filed under that hash:

ls .dvc/cache/files/md5/8f/2c4b19e0a7d3f1c6b8a2e4d7091f3b

Inside the cache: content addressing and links

Understanding this saves you from a whole category of confusion later, and it takes two minutes.

DVC's cache is content-addressable: a file's location is derived from the hash of its contents. Two identical files, whatever they are called, occupy one cache entry. Change one byte and you get a different hash and a new entry.

dvc add data/raw.csv         # hash A
echo "one more row" >> data/raw.csv
dvc add data/raw.csv         # hash B — a new entry; A is still there

That has an important consequence: the cache accumulates. Every version you have ever added is still on disk until you clean it up, which is what makes going back in time possible, and also why the cache grows.

By default DVC does not copy the file into your workspace, it links it from the cache, so you are not storing two copies:

dvc config cache.type reflink,copy      # try reflink, fall back to a copy
dvc config core.check_link_support true

Going back in time: two commands, in order

This is the payoff. Do it on purpose early so you trust it.

# Make a second version and commit it
echo "new rows" >> data/raw.csv
dvc add data/raw.csv
git commit -am "Dataset v2"

# Travel back
git checkout HEAD~1 data/raw.csv.dvc
dvc checkout data/raw.csv
head data/raw.csv                     # v1 is back

Two commands, and the order matters. git checkout moves the pointer; dvc checkout makes the working file match it. Miss the second step and you have v1's pointer with v2's file on disk, which dvc status will tell you about immediately.

dvc status
# data/raw.csv.dvc:
#         changed outs:
#                 modified:           data/raw.csv

Remote storage: where the bytes live

So far everything lives on your machine. A remote is where the bytes go so a colleague (or a CI runner, or a training box) can get them.

# Local folder: perfect for learning, and genuinely useful for a shared NAS
dvc remote add -d myremote /tmp/dvc-storage

# S3
dvc remote add -d storage s3://my-bucket/dvc-store

# Google Cloud Storage
dvc remote add -d storage gs://my-bucket/dvc-store

# Azure Blob
dvc remote add -d storage azure://my-container/dvc-store

# Any SSH box
dvc remote add -d storage ssh://user@host/srv/dvc-store

git add .dvc/config
git commit -m "Configure the default DVC remote"

The -d makes it the default, so dvc push and dvc pull need no arguments. The configuration lands in .dvc/config, which is committed. It contains the location, not the credentials.

dvc push                     # upload cached content to the remote
dvc pull                     # download what the current pointers need
dvc fetch                    # download to the cache without touching the workspace
dvc status -c                # compare local cache against the remote

Those four are the whole workflow, and the mental model mirrors Git exactly:

Two config files: committed versus local

There are two config files, and knowing which is which prevents both leaked secrets and "it works on my machine".

dvc remote add -d storage s3://my-bucket/dvc-store    # → .dvc/config, committed
dvc remote modify --local storage access_key_id AKIA…  # → .dvc/config.local, ignored

The pattern that follows from this is worth adopting immediately: shared truth in config, personal reality in config.local. A remote URL is shared truth. Your access key is personal reality. So is a cache on a different disk because your home directory is small.

# Useful local-only settings
dvc cache dir /mnt/big-disk/dvc-cache          # move the cache off a small root disk
dvc remote modify --local storage ssh_private_key_path ~/.ssh/id_dvc
dvc config --local core.jobs 8                  # more parallel transfers on a fast link

Tracking directories, and the .dir object

Datasets are rarely one file. dvc add works on directories, and the behaviour is slightly different in a way worth knowing.

dvc add data/images/          # one .dvc file for the whole directory
cat data/images.dvc

outs:
- md5: 3a7f9c2e8b1d4506.dir      # note the .dir suffix
  size: 524288000
  nfiles: 12000
  hash: md5
  path: images

The .dir suffix marks this as a directory hash, the hash of a small JSON listing that maps every file in the tree to its own hash. Individual files are still deduplicated in the cache, so adding a directory where one image changed uploads exactly one new object.

Pipelines: from dvc add to dvc.yaml

dvc add versions data somebody produced. A pipeline versions the process, which is what makes a result reproducible rather than recorded.

The idea is a dependency graph. Each stage declares what it needs and what it makes, and DVC reruns a stage only when one of its declared inputs changed.

stages:
  prepare:
    cmd: python src/prepare.py data/raw.csv data/prepared.csv
    deps:
      - src/prepare.py
      - data/raw.csv
    outs:
      - data/prepared.csv

  train:
    cmd: python src/train.py data/prepared.csv models/model.pkl
    deps:
      - src/train.py
      - data/prepared.csv
    outs:
      - models/model.pkl

dvc repro                    # run whatever is out of date, in dependency order
git add dvc.yaml dvc.lock .gitignore
git commit -m "Add the prepare → train pipeline"

Four keys carry almost all of the meaning:

Notice what you no longer do: you do not dvc add a pipeline output. outs already tracks it. Running dvc add on something a stage produces is a common early mistake and DVC will refuse it, because two mechanisms would then own the same file.

dvc.lock, and how DVC knows what changed

dvc.yaml is what you wrote. dvc.lock is what happened, and it is the file that makes reproducibility real.

schema: '2.0'
stages:
  prepare:
    cmd: python src/prepare.py data/raw.csv data/prepared.csv
    deps:
    - path: data/raw.csv
      hash: md5
      md5: 8f2c4b19e0a7d3f1c6b8a2e4d7091f3b
      size: 1048576
    - path: src/prepare.py
      hash: md5
      md5: 1a2b3c4d5e6f7a8b9c0d1e2f3a4b5c6d
      size: 842
    outs:
    - path: data/prepared.csv
      hash: md5
      md5: 9e8d7c6b5a4f3e2d1c0b9a8f7e6d5c4b
      size: 987654

Every dependency and output is recorded with its exact hash at the moment the stage last ran successfully. That is how dvc repro decides what to do. It hashes the current inputs and compares them against the lock.

Two more commands make the graph legible:

dvc dag                      # ASCII dependency graph
dvc dag --outs               # the same, keyed on outputs
dvc status                   # which stages are out of date, and why

Parameters: config DVC can see

Hard-coded hyperparameters make a stage un-reproducible: you changed n_estimators from 100 to 200, DVC saw no dependency change, and skipped the stage. params fixes that.

prepare:
  test_size: 0.2
  random_state: 42

train:
  n_estimators: 100
  max_depth: 8
  learning_rate: 0.05

stages:
  train:
    cmd: python src/train.py
    deps:
      - src/train.py
      - data/prepared.csv
    params:
      - train.n_estimators
      - train.max_depth
      - train.learning_rate
    outs:
      - models/model.pkl

import yaml

with open("params.yaml") as f:
    params = yaml.safe_load(f)["train"]

model = RandomForestClassifier(
    n_estimators=params["n_estimators"],
    max_depth=params["max_depth"],
)

Now changing a number in params.yaml marks the stage out of date, as changing the code would. You can list individual keys as above, or - train to depend on the whole section.

dvc params diff              # what changed since the last commit
dvc params diff HEAD~3       # or against any revision

Metrics and plots: results that diff in Git

A pipeline that produces a model but no numbers is a pipeline you cannot reason about. DVC has two output kinds for exactly this.

stages:
  evaluate:
    cmd: python src/evaluate.py
    deps:
      - src/evaluate.py
      - models/model.pkl
      - data/test.csv
    metrics:
      - metrics.json:
          cache: false
    plots:
      - plots/roc.json:
          cache: false
          x: fpr
          y: tpr
      - plots/confusion.csv:
          cache: false
          template: confusion
          x: actual
          y: predicted

{
  "accuracy": 0.9231,
  "f1": 0.9104,
  "auc": 0.9687,
  "train_seconds": 42.7
}

dvc metrics show             # the current numbers
dvc metrics diff             # versus the last commit
dvc metrics diff HEAD~5      # versus any revision
dvc plots show               # render plots to an HTML file
dvc plots diff HEAD~1        # overlay two revisions on one chart

The cache: false on those outputs is deliberate and worth understanding: metrics files are small text, so committing them to Git rather than caching them in DVC means dvc metrics diff can compare any two commits without downloading anything.

Experiments: cheap runs, no commits

Committing every attempt pollutes your history. dvc exp gives you a lightweight way to run many variations, compare them, and keep only the ones worth keeping.

# Run with a parameter override, without editing any file
dvc exp run --set-param train.n_estimators=300

# Queue several and run them together
dvc exp run --queue --set-param train.max_depth=4
dvc exp run --queue --set-param train.max_depth=8
dvc exp run --queue --set-param train.max_depth=16
dvc exp run --run-all --jobs 3

# Compare
dvc exp show
dvc exp show --only-changed          # hide columns that are identical everywhere

dvc exp show prints a table of every experiment with its parameters and metrics side by side, which is the fastest way to see that max_depth=8 beat both neighbours.

 ───────────────────────────────────────────────────────────────
  Experiment          accuracy   f1      max_depth  n_estimators
 ───────────────────────────────────────────────────────────────
  workspace           0.9231     0.9104  8          100
  main                0.9105     0.8977  6          100
  ├── exp-a1b2c       0.9402     0.9311  8          300
  ├── exp-d4e5f       0.9188     0.9042  4          100
  └── exp-g7h8i       0.9377     0.9265  16         100
 ───────────────────────────────────────────────────────────────

Then promote the winner and discard the rest:

dvc exp apply exp-a1b2c      # bring it into your workspace
git add . && git commit -m "n_estimators=300: accuracy 0.940"

dvc exp branch exp-a1b2c my-branch    # or put it on its own branch
dvc exp remove --queued               # clean up
dvc exp gc --workspace                # garbage-collect experiments not referenced

Reading another repo's data: get versus import

Two commands let you use a DVC repository as a data source, which is how datasets get shared across projects.

# Look at what a repo tracks, without cloning it
dvc list https://github.com/iterative/dataset-registry get-started

# Download a file from it into the current directory
dvc get https://github.com/iterative/dataset-registry get-started/data.xml

# Download AND record where it came from
dvc import https://github.com/iterative/dataset-registry get-started/data.xml

The difference between the last two is the whole point:

dvc import --rev v1.2.0 https://github.com/org/data-repo datasets/train.parquet
dvc update datasets/train.parquet.dvc         # pull the newer upstream version

dvc import writes a pointer that names the source repository, path, and revision. That means your project can state "we depend on version v1.2.0 of the shared training set" as a committed fact, and updating is a deliberate command rather than a silent drift.

Garbage collection: reclaiming the cache safely

The cache accumulates every version of everything you have ever added. That is the feature; it is also why your disk fills up.

dvc gc --workspace           # keep only what the current workspace needs
dvc gc --all-branches        # keep what every branch needs
dvc gc --all-tags            # keep what every tag needs
dvc gc --all-commits         # keep what every commit needs — the safest
dvc gc --cloud               # also clean the remote (dangerous, read below)
dvc gc --dry-run --workspace # show what would go, delete nothing

The flags are all about which revisions count as "in use". The default, no flag at all, is --workspace, which is the most aggressive: anything not needed by the files currently checked out is deleted.

Two other space commands worth knowing:

du -sh .dvc/cache            # how big is the local cache
dvc cache dir                # where is it
dvc cache dir /mnt/big/cache # move it, e.g. off a small root disk

Reading a broken pipeline, in order

Debugging DVC has an order, and following it beats guessing.

1. Run dvc status firstIt names the stage, the specific dependency, and the reason. Most problems are answered here and you can stop.
2. Then dvc status -cCompares your cache against the remote. This is the answer to "my colleague cannot pull my data". You never pushed it.
3. Check the graph with dvc dagA stage that never reruns often has a missing deps entry; a cycle error means two stages each claim the other's output.
4. Run the cmd by handCopy the command out of dvc.yaml and run it in your shell. If it fails there, DVC is innocent and you are debugging your script.
5. Force a rerun with dvc repro -fIgnores the lock and reruns everything. If forcing fixes it, your deps are incomplete, and the real fix is declaring the missing dependency.
6. Add -v for verbose outputdvc repro -v or dvc pull -v prints what DVC is doing, including which remote it contacted and why a transfer failed.

Four failures cover most of what you will hit at this level, and each has a one-line cause:

dvc status                   # local truth
dvc status -c                # remote truth
dvc dag                      # the graph as DVC sees it
dvc repro -f                 # rebuild everything, ignoring the lock
dvc repro --dry              # show what would run, run nothing
dvc repro -s train           # just one stage, and its upstream
dvc repro -s train --single-item   # exactly one stage, no upstream

Putting it all together

Everything above in one project. Nothing here is new. Read it as a whole and you should be able to justify every line.

.
├── .dvc/
│   ├── config              # remote URL — committed
│   └── config.local        # credentials — gitignored
├── .dvcignore
├── data/
│   ├── .gitignore          # written by DVC
│   ├── raw.csv.dvc         # pointer, committed
│   └── raw.csv             # the bytes, gitignored
├── src/
│   ├── prepare.py
│   ├── train.py
│   └── evaluate.py
├── models/
│   └── .gitignore          # written by DVC
├── params.yaml             # committed
├── metrics.json            # committed (cache: false)
├── plots/roc.csv           # committed (cache: false)
├── dvc.yaml                # committed
└── dvc.lock                # committed

prepare:
  test_size: 0.2
  random_state: 42

train:
  n_estimators: 300
  max_depth: 8

stages:
  prepare:
    cmd: python src/prepare.py
    deps:
      - src/prepare.py
      - data/raw.csv                # the dvc add-tracked input
    params:
      - prepare.test_size
      - prepare.random_state
    outs:
      - data/train.csv              # cached: large, binary-ish, regenerable
      - data/test.csv

  train:
    cmd: python src/train.py
    deps:
      - src/train.py
      - data/train.csv              # produced upstream; the link in the graph
    params:
      - train.n_estimators
      - train.max_depth
    outs:
      - models/model.pkl            # cached: the artifact

  evaluate:
    cmd: python src/evaluate.py
    deps:
      - src/evaluate.py
      - models/model.pkl
      - data/test.csv
    metrics:
      - metrics.json:
          cache: false              # small text → Git, so diffs need no pull
    plots:
      - plots/roc.csv:
          cache: false
          x: fpr
          y: tpr

# The everyday loop
dvc repro                          # run only what is out of date
dvc metrics diff                   # did the numbers move?
git add dvc.lock metrics.json plots/ params.yaml
git commit -m "Deeper trees: accuracy 0.921 → 0.940"
dvc push                           # share the data and the model
git push

# What a colleague runs
git clone <repo> && cd <repo>
dvc pull                           # exactly your data and model
dvc repro                          # "Everything is up to date" — reproduced

Ten decisions in there are the whole lesson of this page:

What you can now do, and what comes next

You can version datasets and models with Git-sized pointers, restore any past version, share data through remote storage, express your workflow as a dependency graph that reruns only what changed, parameterise it, record metrics and plots that diff across commits, run and compare experiments without polluting history, depend on datasets from other repositories, keep your disk under control, and debug a pipeline methodically. That is a working practitioner's toolkit, enough to own the data and reproducibility story on a real project.

Mid-level takes every one of those topics further: the hashing and link internals that explain performance, dvc.yaml templating with foreach and matrix, output modifiers like persist and cache: false, external and cloud-versioned outputs, dvc import-url, multiple and per-role remotes, experiment queues and cloud-based experiments, the CI patterns that make dvc repro a pull-request check, DVCLive for in-training logging, and monorepo layouts.

Senior then covers what you own when data versioning is your responsibility: the trust and access model across remotes, credentials that are never stored, immutability and retention on the storage side, cost and lifecycle policy, hashing at terabyte scale, data registries as a product, lineage and audit for regulated work, GDPR deletion against an immutable cache, CI/CD for data with self-hosted runners, and where DVC stops and a feature store or lakehouse begins.

Part one of three. A fast review of everything in the Beginner Detailed track, in about twenty-five minutes. Fast review first, common questions at the end. Mid-level reviews this plus its own material; Senior reviews all three.

The thirty-second answer

DVC is version control for data and machine-learning pipelines, built on top of Git. Instead of committing large files, DVC stores a small text pointer containing a content hash in Git and keeps the actual bytes in a remote such as S3. One git checkout then restores the exact code, and dvc checkout restores the exact data that belongs to it, so a commit reproduces a result rather than describing one.

Then add the sentence that shows you have used it: "the part people underestimate is dvc.lock. It records the hash of every input and output from the last successful run, which is how dvc repro reruns only the stages whose inputs changed."

The model

Why not just Git?

Know the follow-up: "why not Git LFS?" LFS solves the size problem and nothing else: no pipelines, no dependency graph, no metrics, no experiments, and it is tied to your Git host rather than to storage you choose.

The everyday commands

dvc init                          # scaffold in an existing Git repo
dvc add data/raw.csv              # track a file you did not generate
dvc status                        # working files versus pointers
dvc status -c                     # local cache versus remote
dvc checkout                      # restore working files from the cache
dvc push / dvc pull / dvc fetch   # cache ↔ remote
dvc repro                         # run whatever is out of date
dvc dag                           # the dependency graph
dvc metrics show / diff           # the numbers, and how they moved
dvc plots show / diff             # curves, rendered to HTML
dvc exp run / show / apply        # experiments
dvc gc --dry-run --all-commits    # what cleanup would delete
dvc doctor                        # version, platform, available remotes

The Git parallel is exact, and stating it is the fastest way to show you understand the model:

What dvc add does

dvc add data/raw.csv

outs:
- md5: 8f2c4b19e0a7d3f1c6b8a2e4d7091f3b
  size: 1048576
  hash: md5
  path: raw.csv

Four things happen, and being able to list them is the core question at this level:

1. Hash the contentsMD5 by default, which is why dvc add on a large file is I/O-bound.
2. Move it to the cache.dvc/cache/files/md5/8f/2c4b…, where the first two characters shard the directory.
3. Write the pointerThe .dvc file, which you commit to Git.
4. Add a .gitignore entrySo Git never tries to store the data itself.

The cache is content-addressable: identical files share one entry, and every version you have ever added is still there until you garbage-collect. A directory gets a .dir hash, the hash of a JSON listing mapping each file to its own hash, so individual files inside it are still deduplicated.

Time travel, in two commands

git checkout HEAD~1 data/raw.csv.dvc      # move the pointer
dvc checkout data/raw.csv                  # make the file match it

git checkout moves pointers; dvc checkout moves data. Skip the second and dvc status reports the mismatch immediately. dvc install adds Git hooks that run dvc checkout for you after every git checkout, which removes the most common source of "my data is the wrong version".

If the cache does not have that version, dvc checkout fails and you need dvc pull, which is exactly the "missing cache files" error people hit when the data was never pushed.

Configuration: two files

dvc remote add -d storage s3://bucket/dvc                # → config, committed
dvc remote modify --local storage access_key_id AKIA…    # → config.local, ignored
dvc cache dir /mnt/big/cache                              # a bigger disk

Credentials never belong in .dvc/config. Use the provider's own mechanism (environment variables, an IAM role, gcloud auth, an SSH agent) or --local if it must be a file.

Pipelines

stages:
  prepare:
    cmd: python src/prepare.py
    deps: [src/prepare.py, data/raw.csv]
    params: [prepare.test_size]
    outs: [data/train.csv, data/test.csv]

  train:
    cmd: python src/train.py
    deps: [src/train.py, data/train.csv]
    params: [train.n_estimators, train.max_depth]
    outs: [models/model.pkl]

  evaluate:
    cmd: python src/evaluate.py
    deps: [src/evaluate.py, models/model.pkl, data/test.csv]
    metrics:
      - metrics.json:
          cache: false

Never dvc add a pipeline output. outs already tracks it, and DVC will refuse, because two mechanisms cannot own the same file.

dvc.lock and how repro decides

stages:
  train:
    cmd: python src/train.py
    deps:
    - path: data/train.csv
      md5: 8f2c4b19e0a7d3f1c6b8a2e4d7091f3b
      size: 1048576
    outs:
    - path: models/model.pkl
      md5: 9e8d7c6b5a4f3e2d1c0b9a8f7e6d5c4b

Always commit dvc.lock. Treat it exactly like package-lock.json: generated, never hand-edited, always committed. Without it a colleague rebuilds everything and cannot verify they got your result.

Metrics, plots, and cache: false

dvc metrics show
dvc metrics diff HEAD~1
dvc plots diff HEAD~1 --open

cache: false on a metrics file means dvc metrics diff can compare any two commits without downloading anything. Caching a 200-byte JSON forces a dvc pull to read a number, which is the wrong trade.

Experiments

dvc exp run --set-param train.max_depth=16     # no commit, no branch
dvc exp run --queue -S train.max_depth=4,8,16
dvc exp run --run-all --jobs 3
dvc exp show --only-changed
dvc exp apply exp-a1b2c                         # promote the winner
dvc exp remove --queued

An experiment is a hidden commit under refs/exps/ plus cached outputs. It is cheap to create, cheap to discard, and comparable in one table, which is why you stop committing every attempt.

Sharing data across projects

dvc list https://github.com/org/data-registry
dvc get https://github.com/org/data-registry data/train.csv        # anonymous copy
dvc import --rev v1.2.0 https://github.com/org/data-registry data/train.csv
dvc update data/train.csv.dvc

Garbage collection

dvc gc --dry-run --all-commits    # always dry-run first
dvc gc --workspace                # keep only the current checkout's data — aggressive
dvc gc --all-commits              # keep everything reachable from any commit — safest
dvc gc --cloud                    # ALSO deletes on the remote — dangerous

The flag chooses which revisions count as in use. The default is --workspace, which on a repository with ten branches deletes the data for nine of them. Add --cloud and you have done it to the team's remote, where it is not recoverable.

The traps, and why they happen

Three ideas explain nearly every beginner failure: pointers and data are separate, the cache is local until you push, and DVC only knows what you declared.

Common interview questions

1. What is DVC and what problem does it solve?It versions data and ML pipelines on top of Git. Large files stay out of Git; a small hash-bearing pointer goes in instead, and the bytes live in object storage. The result is that one commit identifies both the code and the exact data, so "which data produced this model?" becomes a lookup rather than an investigation.
2. Why not just commit data to Git? Why not Git LFS?Git stores full history of every version, so a 2 GB file changed ten times is 20 GB in every clone forever. LFS fixes the size problem but gives you nothing else (no pipelines, no dependency graph, no metrics, no experiments) and ties storage to your Git host.
3. What is inside a .dvc file?A hash, a size, and a path, a few hundred bytes. It is the pointer Git versions; DVC uses the hash to find the content in the cache or the remote.
4. Walk me through what dvc add does.Hashes the file, moves it into the content-addressed cache under that hash, writes the .dvc pointer, and adds a .gitignore entry so Git ignores the data. Then you commit the pointer.
5. How do you restore an old version of a dataset?git checkout the .dvc file to that revision, then dvc checkout to make the working file match. Two commands, because Git moves pointers and DVC moves data. dvc install adds hooks that automate the second.
6. What is the difference between dvc add and outs in dvc.yaml?dvc add is for inputs you did not generate, like a raw dataset someone handed you. outs is for files a stage produces. Never both on the same file; DVC refuses because two mechanisms would own it.
7. How does dvc repro know what to rerun?It compares the current hash of every declared dependency against the hashes recorded in dvc.lock from the last successful run. Any mismatch, or a missing output, means the stage runs, and that stage's new outputs cascade to everything downstream.
8. Why must dvc.lock be committed?It is the record of which exact inputs produced which exact outputs. Without it, a colleague cannot verify they reproduced your result, and dvc repro rebuilds everything. Same role as package-lock.json: generated, never hand-edited, always committed.
9. How do you resolve a dvc.lock merge conflict?Never by hand. The hashes describe a run that happened, so stitching two halves together produces a lock describing a run that never occurred. Take one side, then run dvc repro and commit the regenerated file.
10. Why do parameters need declaring in params?Otherwise DVC sees no dependency change when you edit params.yaml and skips the stage, a silent staleness. Declaring the key makes a value change invalidate the stage as a code change would.
11. Why do metrics use cache: false?They are small text, so committing them to Git means dvc metrics diff can compare any two revisions instantly with no download. Putting them in the cache would require a dvc pull just to read a number.
12. How does a colleague get your data?git clone, then dvc pull. The clone brings the pointers and dvc.lock; the pull brings the bytes from the remote into the cache and then into the workspace.
13. Where do credentials go?Not in .dvc/config, which is committed. Use the provider's own credential chain (environment variables, an IAM role, gcloud auth, an SSH agent) or .dvc/config.local, which is gitignored, if it must be a file.
14. What is the DVC cache, and why does it grow?Content-addressed local storage under .dvc/cache. Every version you have ever added is a separate object, which is what makes time travel possible, and why you eventually need dvc gc.
15. What does dvc gc do, and what is the risk?Deletes cached objects that are not reachable from the revisions you name. The default --workspace only considers the current checkout, so on a multi-branch repo it deletes most of your data. Always --dry-run first, prefer --all-commits, and treat --cloud as destructive because it hits the shared remote.
16. dvc get versus dvc import?get downloads a file and records nothing, an anonymous copy. import downloads it and writes a .dvc file naming the source repository, path, and pinned revision, so it becomes a versioned dependency you can later dvc update.
17. What is a DVC experiment and why not just commit?A lightweight hidden commit under refs/exps/ with its own cached outputs. It needs no branch and no commit, several can be queued and run in parallel, and dvc exp show compares them in one table. Committing every attempt instead pollutes history with dead ends nobody can read.
18. How would you make an existing project reproducible?dvc init, dvc add the raw inputs, express the scripts as stages in dvc.yaml with every input declared in deps, move hyperparameters into params.yaml, write metrics with cache: false, then dvc repro until a second run reports up to date. The acceptance test is a fresh clone that pulls and reports up to date without rebuilding.
19. What is the first command when something is wrong?dvc status, which names the stage and the specific dependency. Then dvc status -c to compare against the remote, which answers "my colleague cannot pull my data". You never pushed it.
20. Which storage backends does DVC support?S3 and anything S3-compatible such as MinIO, Google Cloud Storage, Azure Blob, SSH, HDFS, WebDAV, HTTP for read-only, and a plain local or network directory. The driver comes from a pip extra, which is why pip install dvc alone fails on an S3 URL.

Sixty-second self-test

• Give the thirty-second answer, then the sentence that shows you have used it.
• Say what a .dvc file contains and roughly how big it is.
• List the four things dvc add does.
• Name the two commands to restore an old dataset version, and why there are two.
• Explain how dvc repro decides what to run.
• Say why dvc.lock must be committed and never hand-edited.
• Explain why a parameter must be declared in params.
• Say why metrics use cache: false.
• Give the difference between dvc get and dvc import.
• Say what dvc gc --workspace deletes, and what makes --cloud dangerous.
• Name where credentials belong, and where they must not go.
• State the first two commands to run when something is wrong.

Part one of three. Almost every beginner problem with DVC comes from one of three things: pointers and data being separate, the cache being local until you push, or DVC only knowing what you declared. Start with the error table, then work through the habits and practice cards underneath it.

Common errors at this level

The habits that pay off most

Practice cards

Short, self-contained exercises. Each one takes a few minutes and leaves you with a fact you will not forget.

1. Watch the two-command time travelMake three versions of a file, committing each. git checkout an old .dvc file, run dvc status before dvc checkout, and read the mismatch it reports. Then dvc install and repeat to see the step disappear.
2. Simulate a colleaguerm -rf .dvc/cache data/raw.csv, then dvc pull. Getting your file back after deleting both the file and the cache is the moment DVC stops being abstract.
3. Cause a silent skipRemove a script from a stage's deps, change the script, and run dvc repro. Nothing errors and nothing reruns. This is the most dangerous failure at this level.
4. Watch the cascadeEdit a late-stage script and run dvc status: one stage. Edit an early one: every downstream stage. Then run dvc dag and confirm the picture matches.
5. Prove parameters need declaringChange a value in params.yaml that is not listed under params and confirm the stage is skipped. Declare it and repeat.
6. Produce a metrics diff worth sharingAdd a metrics stage, commit, change one parameter, repro, and run dvc metrics diff. That table is the artifact to paste into a pull request.
7. Compare three experimentsQueue three variations of one parameter, run them, and read dvc exp show --only-changed. Apply the winner, commit, and discard the rest.
8. Read a dry-run garbage collectionWith three dataset versions committed, compare dvc gc --dry-run --workspace against --all-commits. The difference in the two lists is the lesson.
9. Break the pointer/data pairing on purposeDelete a tracked output with rm, run dvc status, then dvc checkout. Knowing DVC restores it from the cache saves you from re-running a long pipeline.

Debugging order

Follow this rather than guessing. The first two steps answer most problems.

1. dvc statusNames the stage, the specific dependency, and the reason. Most problems end here.
2. dvc status -cCompares your cache against the remote. This is the answer to "my colleague cannot pull my data". You never pushed it.
3. dvc dagA stage that never reruns usually has a missing deps entry; a cycle error means two stages each claim the other's output.
4. Run the cmd by handCopy it out of dvc.yaml and run it in your shell. If it fails there, DVC is innocent and you are debugging your script.
5. dvc repro -fIgnores the lock and reruns everything. If forcing fixes it, your deps are incomplete, and declaring the missing input is the real fix.
6. Add -vdvc repro -v or dvc pull -v prints every hash comparison and every transfer, including which remote it contacted and why something failed.

dvc status                   # local truth
dvc status -c                # remote truth
dvc dag                      # the graph as DVC sees it
dvc repro --dry              # what would run, run nothing
dvc repro -s train           # one stage and its upstream
dvc repro -s train --single-item   # exactly one stage
dvc repro -f                 # ignore the lock, rebuild everything
dvc doctor                   # version, platform, available remote drivers

dvc add versus outs: the decision

Half of all beginner DVC confusion lives here, and one question resolves it: did a pipeline stage produce this file?

Getting it wrong produces one of two errors. dvc add on a stage output is refused outright. Forgetting outs is worse: the file is untracked, so it is not cached, not pushed, and, because DVC clears a stage's declared path before running, an undeclared sibling file can be deleted without warning.

Set up so DVC catches things for you

Do these four things once per project and an entire class of mistakes disappears.

dvc install                   # Git hooks: checkout, pre-push warning
dvc config core.autostage true # git add .dvc files automatically
dvc config cache.type reflink,copy
dvc doctor                     # confirm remote drivers are present

core.autostage is the quiet win: DVC stages the .dvc files and .gitignore entries it creates, so you stop forgetting to git add them.

Then a .dvcignore, which works like .gitignore but for DVC's own scanning. It stops DVC hashing files you never want tracked:

*.tmp
.ipynb_checkpoints/
__pycache__/
*.log
.DS_Store

Writing a dvc.yaml that ages well

stages:
  prepare:
    cmd: python src/prepare.py               # no hard-coded paths in the command
    deps:
      - src/prepare.py                       # the script itself
      - src/common.py                        # anything it imports
      - data/raw.csv
    params:
      - prepare.test_size                    # every value the script reads
    outs:
      - data/train.csv
      - data/test.csv

Reading dvc status output

Learning to read this saves more time than any other single skill at this level.

data/raw.csv.dvc:
        changed outs:
                modified:           data/raw.csv        ← the file differs from the pointer

train:
        changed deps:
                modified:           src/train.py        ← this input changed
        changed outs:
                deleted:            models/model.pkl    ← the output is missing

evaluate:
        changed command                                 ← you edited `cmd` in dvc.yaml

Small things worth doing from day one

Name datasets by what they are, not by version. customers.parquet, not customers_v3_final.parquet. Versions are Git's job now, and the old naming habit is what DVC removes.

Add a desc: to important outputs. It shows up in dvc list and answers "what is this?" six months later.

Keep raw data immutable. dvc add it once and never edit it in place. Regenerate derived data through the pipeline instead.

Commit the pointer in the same commit as the code that reads it. Splitting them means an intermediate commit that cannot reproduce.

Write the metrics number into the commit message. "Deeper trees: F1 0.910 → 0.931" makes git log a results log.

Check dvc status -c before you finish for the day. It is a two-second habit that prevents "the pointer is pushed but the data is on my laptop".

# A pre-push sanity check worth aliasing
dvc status && dvc status -c && git status --short

A starter setup worth keeping

Copy this into a new project and delete what you do not need. Every line is something from this page.

*.tmp
.ipynb_checkpoints/
__pycache__/
*.log
.DS_Store

prepare:
  test_size: 0.2
  random_state: 42

train:
  n_estimators: 300
  max_depth: 8

stages:
  prepare:
    cmd: python src/prepare.py
    deps: [src/prepare.py, src/common.py, data/raw.csv]
    params: [prepare.test_size, prepare.random_state]
    outs: [data/train.csv, data/test.csv]

  train:
    cmd: python src/train.py
    deps: [src/train.py, src/common.py, data/train.csv]
    params: [train.n_estimators, train.max_depth]
    outs:
      - models/model.pkl                 # cached: the artifact

  evaluate:
    cmd: python src/evaluate.py
    deps: [src/evaluate.py, models/model.pkl, data/test.csv]
    metrics:
      - metrics.json:
          cache: false                   # small text → Git, diffable forever
    plots:
      - plots/confusion.csv:
          cache: false
          template: confusion
          x: actual
          y: predicted

# One-time setup
git init && dvc init
dvc remote add -d storage s3://my-bucket/dvc
dvc install
dvc config core.autostage true
git add . && git commit -m "Initialise DVC"

# The everyday loop
dvc repro
dvc metrics diff
git commit -am "Deeper trees: F1 0.910 → 0.931"
dvc push && git push

# What a colleague runs
git clone <repo> && cd <repo> && dvc pull && dvc repro

Eight details in there are the whole lesson of this page: a .dvcignore written before the first add, hooks and autostage installed once, every script and its imports in deps, hyperparameters in params.yaml, cache: false on metrics and plots, dvc push before git push, a metrics number in the commit message, and a colleague's three-command reproduction as the acceptance test.

Mid-level tips go deeper on every one of these: cache internals and why CI never hits the cache, sharding small files, output modifiers in practice, entrypoint-style pipeline patterns, Compose-style override layouts for environments, experiment queue hygiene, remote tuning, and diagnosing "reproduces locally, not in CI".

This is part two of three. It picks up exactly where Beginner ended and takes every topic from there further, then adds the machinery you have not met yet. Nothing is dropped and nothing is repeated for its own sake. Where you already know the basics, we go straight to the depth.

Where this picks up

Each section starts with the problem it solves, and ends with a Try it you can do on a real project in a few minutes.

Hashing, the cache layout, and why it matters

Beginner told you the cache is content-addressed. Performance questions start arriving at this level, and they all trace back to the mechanism.

DVC hashes file contents with MD5 by default, chosen for speed rather than cryptographic strength, which is the honest answer when an interviewer asks. The hash becomes the object's address:

.dvc/cache/files/md5/
├── 8f/
│   └── 2c4b19e0a7d3f1c6b8a2e4d7091f3b        ← a file's contents
└── 3a/
    └── 7f9c2e8b1d4506….dir                  ← a JSON listing for a directory

The first two hex characters become the directory, which keeps any single directory from holding millions of entries. A .dir object is a JSON array mapping relative paths to per-file hashes:

[
  {"md5": "a1b2c3…", "relpath": "cats/001.jpg"},
  {"md5": "d4e5f6…", "relpath": "cats/002.jpg"},
  {"md5": "789abc…", "relpath": "dogs/001.jpg"}
]

Three consequences follow, and each one shows up as a real problem:

Hashing is the bottleneck, not transfer. dvc add on a 50 GB dataset reads all 50 GB to compute hashes. On a slow disk that dominates everything. DVC caches file hashes by (path, size, mtime, inode) in .dvc/tmp/, so a second add of unchanged files is fast, which is why the first run after a git clone feels much slower than every run afterwards.

Small files are disproportionately expensive. A million 4 KB files means a million hash operations, a million cache entries, and a .dir listing of a million rows. The same bytes as 200 shard files is faster.

Deduplication is global to the cache. Identical content anywhere (two branches, two projects on a shared cache, two paths in one directory) occupies one object. That is why a shared cache across projects can cut disk usage sharply.

dvc config core.hash_algorithm md5           # the default
dvc config core.hash_algorithm sha256        # slower, cryptographically sound
dvc data status --granular                   # per-file view of what changed
du -sh .dvc/cache && find .dvc/cache -type f | wc -l

Link types, measured

Beginner said reflink,copy is the safe default. On a large dataset the difference between the options is minutes and gigabytes.

dvc config cache.type reflink,hardlink,symlink,copy   # try in order
dvc config cache.protected true                       # make working files read-only
dvc config core.check_link_support true               # verify at runtime
dvc doctor                                            # what your filesystem supports

The reflink,copy ordering means "use copy-on-write if the filesystem can, otherwise make a real copy". That is the right default: no extra space where possible, and always safe.

The pattern for a machine where space is tight and speed matters:

dvc config --local cache.type hardlink,symlink
dvc config --local cache.protected true
# and then: never edit outputs in place, always regenerate through the pipeline

Pipeline templating: foreach, matrix, and vars

A pipeline with one stage per feature set, per model type, or per dataset shard quickly becomes hundreds of near-identical blocks. foreach and matrix collapse them.

stages:
  featurize:
    foreach:
      - tfidf
      - word2vec
      - embeddings
    do:
      cmd: python src/featurize.py --method ${item}
      deps:
        - src/featurize.py
        - data/prepared.csv
      outs:
        - data/features-${item}.pkl

That generates three stages named featurize@tfidf, featurize@word2vec, and featurize@embeddings. foreach also accepts a dictionary, which is how you vary several values at once:

stages:
  train:
    foreach:
      small:
        depth: 4
        estimators: 100
      large:
        depth: 16
        estimators: 500
    do:
      cmd: python src/train.py --depth ${item.depth} --estimators ${item.estimators}
      deps: [src/train.py, data/train.csv]
      outs: [models/model-${key}.pkl]

matrix takes the cross product, which is the right tool for a genuine grid:

stages:
  benchmark:
    matrix:
      model: [rf, xgb, lgbm]
      split: [0.1, 0.2]
    do:
      cmd: python src/bench.py --model ${item.model} --split ${item.split}
      outs:
        - out/${item.model}-${item.split}.json

Six stages from six lines. vars lets you pull shared values in from a file or define them inline:

vars:
  - params.yaml                    # everything in params.yaml is available
  - config/paths.yaml:data         # just the `data` key from another file
  - stage_defaults:
      timeout: 3600

stages:
  train:
    cmd: python src/train.py --timeout ${stage_defaults.timeout}
    deps:
      - ${data.train_path}

dvc dag                            # confirm the generated stages
dvc repro featurize@tfidf          # target one generated stage
dvc repro 'featurize@*'            # or a glob

Output modifiers

Beginner used plain outs. Every output takes modifiers, and choosing them correctly is most of what separates a pipeline that works from one that is pleasant to operate.

stages:
  train:
    cmd: python src/train.py
    outs:
      - models/model.pkl                  # the default: cached, gitignored
      - models/summary.txt:
          cache: false                    # small text → Git, diffable forever
      - logs/training.log:
          persist: true                   # survives the next run; append-friendly
      - checkpoints/:
          persist: true
          push: false                     # cached locally, never uploaded
      - data/huge-intermediate.parquet:
          remote: bigstore                # cached to a specific remote

Two of those deserve real attention.

persist: true changes the execution contract. Normally DVC removes a stage's outputs before running it, so the stage always starts clean. With persist, the previous output is left in place, which is what you need for checkpoint-resume training or an append-only log, and which also means the stage is no longer a function of its inputs. Use it deliberately.

push: false is the cost lever. A 200 GB intermediate that every machine regenerates in four minutes does not belong in object storage. Keeping it cached locally but out of the remote is often the single largest storage saving available.

dvc status                       # persist outputs are reported differently
dvc repro --force                # persist outputs are still not deleted
dvc push --remote bigstore       # push only to a specific remote

External data and import-url

So far every input was a file in your repository. Real pipelines start from somewhere else: an S3 landing zone, an HTTP endpoint, a database export dropped on a share.

# Track a remote file as a dependency, with version tracking
dvc import-url s3://landing-bucket/exports/2024-05-01/events.parquet data/events.parquet

# HTTP works too
dvc import-url https://data.example.org/reference/taxonomy.csv data/taxonomy.csv

# Record the source without downloading the bytes now
dvc import-url --no-download s3://landing/huge.parquet data/huge.parquet

# Later, check whether upstream changed and pull it if so
dvc update data/events.parquet.dvc
dvc update --no-download data/huge.parquet.dvc

The .dvc file records both the source URL and the upstream object's own hash or etag, so dvc update can tell you whether the source changed:

deps:
- md5: 4f8a2b1c9e7d3506
  size: 1073741824
  path: s3://landing-bucket/exports/2024-05-01/events.parquet
outs:
- md5: 4f8a2b1c9e7d3506
  size: 1073741824
  path: events.parquet

You can also reference external data directly as a dependency inside a stage, which avoids a separate download step:

stages:
  prepare:
    cmd: python src/prepare.py
    deps:
      - src/prepare.py
      - s3://landing-bucket/exports/current/events.parquet    # external dep
    outs:
      - data/prepared.parquet

Multiple remotes

One remote is the common case. Several become necessary as soon as cost, geography, or access differ across your data.

dvc remote add -d origin s3://prod-bucket/dvc            # the default
dvc remote add archive s3://glacier-bucket/dvc           # cheap, cold
dvc remote add scratch /mnt/fast-nvme/dvc-scratch        # local, fast
dvc remote add readonly https://data.example.org/dvc     # HTTP, pull-only

dvc remote list
dvc push --remote archive
dvc pull --remote readonly

Per-remote settings cover most real operational needs:

dvc remote modify origin region eu-west-1
dvc remote modify origin sse AES256                      # server-side encryption
dvc remote modify origin acl bucket-owner-full-control
dvc remote modify origin jobs 16                          # parallel transfers
dvc remote modify archive verify true                     # re-hash on download
dvc remote modify --local origin credentialpath ~/.aws/dvc-creds
dvc remote modify origin version_aware true               # use bucket versioning

An output can name its own remote, which is how you tier storage without moving anything by hand:

outs:
  - models/model.pkl                       # → default remote
  - data/raw-archive.parquet:
      remote: archive                      # → cold storage

Experiments at scale

Beginner ran a few experiments in sequence. At this level you want dozens, running in parallel, without three of them fighting over the same workspace.

# Queue a sweep from a range
dvc exp run --queue -S train.n_estimators=100,200,400,800
dvc exp run --queue -S 'train.max_depth=range(4, 20, 4)'

# Run the queue with N workers
dvc exp run --run-all --jobs 4

# Watch and manage the queue
dvc queue status
dvc queue logs exp-a1b2c
dvc queue stop
dvc queue remove --all

The critical mechanic: a queued experiment runs in an isolated temporary workspace, so four workers do not overwrite each other's models/model.pkl. That is what --temp does implicitly, and what makes parallel sweeps safe.

dvc exp run --temp                       # run in a temp dir, leave your workspace alone
dvc exp run --temp --set-param train.lr=0.01

Comparing and filtering becomes the main activity once you have thirty results:

dvc exp show --only-changed              # hide constant columns
dvc exp show --sort-by accuracy --sort-order desc
dvc exp show --num 5                     # last 5 commits' experiments
dvc exp show --drop 'train_seconds|.*_std'
dvc exp show --csv > results.csv         # or --json, --md
dvc exp diff exp-a1b2c exp-d4e5f         # two experiments, side by side

Promotion has three shapes, each appropriate to a different situation:

dvc exp push origin exp-a1b2c            # share an experiment
dvc exp list origin                      # what others have pushed
dvc exp pull origin exp-a1b2c            # fetch theirs

DVCLive: logging from inside training

metrics.json written at the end of a run tells you the final score. DVCLive logs during the run, which gives you per-epoch curves and lets you compare training dynamics rather than just endpoints.

pip install dvclive

from dvclive import Live

with Live() as live:
    live.log_params({"lr": lr, "batch_size": bs, "epochs": epochs})

    for epoch in range(epochs):
        train_loss = train_one_epoch(model, loader)
        val_loss, val_acc = validate(model, val_loader)

        live.log_metric("train/loss", train_loss)
        live.log_metric("val/loss", val_loss)
        live.log_metric("val/accuracy", val_acc)
        live.next_step()                       # advances the step counter

    live.log_artifact("models/model.pkl", type="model", name="classifier")

DVCLive writes into dvclive/ in a layout DVC already understands, so the pipeline wiring is short:

stages:
  train:
    cmd: python src/train.py
    deps: [src/train.py, data/train.csv]
    params: [train.lr, train.batch_size, train.epochs]
    outs:
      - models/model.pkl
    metrics:
      - dvclive/metrics.json:
          cache: false
    plots:
      - dvclive/plots/metrics:
          x: step
          cache: false

dvc plots show                     # per-epoch curves
dvc plots diff HEAD~1              # this run's curves against the last
dvc exp show                       # final values appear as columns automatically

There are framework integrations that remove the manual loop entirely: callbacks for PyTorch Lightning, Keras, Hugging Face, XGBoost, LightGBM, and more:

from dvclive.lightning import DVCLiveLogger
trainer = Trainer(logger=DVCLiveLogger(log_model=True))

Plot templates and comparison

Beginner produced a plot. At this level the question is how to make the plot answer a question without hand-writing a chart every time.

plots:
  - plots/confusion.csv:
      template: confusion
      x: actual
      y: predicted
      cache: false
  - plots/roc.json:
      template: linear
      x: fpr
      y: tpr
      title: ROC curve
      x_label: False positive rate
      y_label: True positive rate
      cache: false
  - plots/importance.csv:
      template: bar_horizontal
      x: importance
      y: feature
      cache: false

Top-level plot definitions let you combine several files into one chart, which is how you compare across stages or models:

plots:
  - accuracy-comparison:
      x: epoch
      y:
        logs/rf.csv: accuracy
        logs/xgb.csv: accuracy
      title: RF versus XGBoost

dvc plots show                                   # write and open an HTML report
dvc plots diff HEAD~3 HEAD --targets plots/roc.json
dvc plots diff --open                            # open in a browser directly
dvc plots templates                              # list available templates
dvc plots templates confusion --out .dvc/plots   # customise one

DVC in CI

A pipeline that only reproduces on your laptop is not reproducible. Putting dvc repro in CI turns reproducibility from a claim into a check.

name: Pipeline
on:
  pull_request:
  push:
    branches: [main]

jobs:
  repro:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
        with:
          fetch-depth: 0                 # DVC needs history for metrics diff

      - uses: actions/setup-python@v5
        with:
          python-version: '3.11'
          cache: pip

      - run: pip install -r requirements.txt

      # Credentials via OIDC — no stored keys
      - uses: aws-actions/configure-aws-credentials@v4
        with:
          role-to-assume: ${{ vars.DVC_ROLE_ARN }}
          aws-region: ${{ vars.AWS_REGION }}

      # Cache DVC's own cache between runs, keyed on the lock file
      - uses: actions/cache@v4
        with:
          path: .dvc/cache
          key: dvc-${{ hashFiles('dvc.lock') }}
          restore-keys: dvc-

      - run: dvc pull --allow-missing
      - run: dvc repro

      - name: Fail if the pipeline was not up to date
        run: |
          dvc status --json | tee status.json
          test "$(cat status.json)" = "{}" || {
            echo "::error::dvc.lock is stale — run 'dvc repro' and commit"; exit 1; }

      - name: Report metrics on the pull request
        if: github.event_name == 'pull_request'
        run: |
          {
            echo "## Metrics"
            dvc metrics diff origin/main --md
            echo "## Parameters"
            dvc params diff origin/main --md
          } >> "$GITHUB_STEP_SUMMARY"

      - run: dvc push
        if: github.ref == 'refs/heads/main'

Five details in there carry the weight:

fetch-depth: 0. dvc metrics diff origin/main needs the other revision present. A shallow clone silently produces an empty diff.

Caching .dvc/cache keyed on dvc.lock. Without it, every CI run downloads every input. With it, an unchanged pipeline pulls almost nothing.

dvc pull --allow-missing. Pulls what exists and does not fail on outputs that have not been produced yet, which is the normal state on a branch.

The dvc status gate. This is the check that matters: it fails the pull request when someone edited a script or a parameter without rerunning the pipeline, so dvc.lock cannot drift from the code.

dvc push only on main. Feature branches produce experiments, not canonical artifacts. Pushing from every branch fills your remote with dead ends.

Monorepos and multiple projects

One repository with several models, or one dataset shared by several repositories, both need a layout decision.

Subdirectory pipelines. DVC supports a dvc.yaml per subdirectory, each with its own stages, sharing one .dvc/cache and one remote:

.
├── .dvc/                     # one cache, one remote, one config
├── projects/
│   ├── churn/
│   │   ├── dvc.yaml
│   │   ├── dvc.lock
│   │   └── params.yaml
│   └── fraud/
│       ├── dvc.yaml
│       ├── dvc.lock
│       └── params.yaml
└── shared/
    └── data/features.parquet.dvc

dvc repro projects/churn/dvc.yaml
dvc repro --all-pipelines             # every dvc.yaml in the repo
dvc dag projects/fraud/dvc.yaml

A stage in one project can depend on an output from another by relative path, which makes shared feature engineering a first-class dependency rather than a copy.

Separate repositories with import. When teams need independence, keep the data in its own repository and consume it:

dvc import --rev v2.1.0 git@github.com:org/data-registry.git datasets/customers.parquet
dvc update --rev v2.2.0 datasets/customers.parquet.dvc

Debugging, one level deeper

Beginner's checklist covers "it does not work". At this level the failures are subtler: it runs, it succeeds, and the result is wrong or the rebuild was unnecessary.

dvc repro --dry                     # what would run, and in what order
dvc repro --downstream train        # train and everything after it
dvc repro --force-downstream        # rerun downstream even if hashes matched
dvc repro --single-item train       # exactly one stage, no upstream
dvc repro --pull                    # fetch missing deps from the remote as needed
dvc repro --no-commit               # run but do not save outputs to the cache
dvc repro -v                        # verbose: every hash comparison

dvc data status --granular --unchanged   # per-file, including what did not change
dvc diff HEAD~3                          # which tracked files changed between revisions
dvc diff --targets data/ HEAD~1

dvc data status --granular is the underused one. Where dvc status says "modified: data/images", the granular form lists the individual files added, modified, and deleted, which is how you discover that a "changed dataset" is one corrupted image.

Four subtle failures and how to find them:

Putting it all together

Everything from this level in one project. Nothing here is new. Read it as a whole and you should be able to justify every line.

prepare:
  test_size: 0.2
  random_state: 42

featurize:
  methods: [tfidf, embeddings]
  max_features: 50000

train:
  n_estimators: 300
  max_depth: 8
  learning_rate: 0.05

vars:
  - params.yaml
  - config/paths.yaml:paths

stages:
  # External, dated source — immutable, so the result stays reproducible
  ingest:
    cmd: python src/ingest.py
    deps:
      - src/ingest.py
      - s3://landing/exports/2024-05-01/events.parquet
    outs:
      - data/raw.parquet

  prepare:
    cmd: python src/prepare.py
    deps: [src/prepare.py, data/raw.parquet]
    params: [prepare.test_size, prepare.random_state]
    outs:
      - data/train.parquet
      - data/test.parquet

  # One block, two generated stages
  featurize:
    foreach: ${featurize.methods}
    do:
      cmd: python src/featurize.py --method ${item}
      deps: [src/featurize.py, data/train.parquet]
      params: [featurize.max_features]
      outs:
        - data/features-${item}.pkl:
            push: false                 # regenerable in minutes; not worth storing
  train:
    cmd: python src/train.py
    deps:
      - src/train.py
      - requirements.txt                # pinned deps are a real dependency
      - data/features-tfidf.pkl
    params: [train.n_estimators, train.max_depth, train.learning_rate]
    outs:
      - models/model.pkl                # cached and pushed: the artifact
      - logs/train.log:
          persist: true                 # appended across runs
    metrics:
      - dvclive/metrics.json:
          cache: false                  # small text → Git, diffable
    plots:
      - dvclive/plots/metrics:
          x: step
          cache: false

  evaluate:
    cmd: python src/evaluate.py
    deps: [src/evaluate.py, models/model.pkl, data/test.parquet]
    metrics:
      - metrics.json:
          cache: false
    plots:
      - plots/confusion.csv:
          template: confusion
          x: actual
          y: predicted
          cache: false

# .dvc/config — committed
[core]
    remote = origin
    autostage = true
['remote "origin"']
    url = s3://prod-bucket/dvc
    jobs = 16
['remote "archive"']
    url = s3://glacier-bucket/dvc
[cache]
    type = reflink,copy
    protected = true

# The everyday loop
dvc repro                                  # only what changed
dvc metrics diff && dvc params diff        # what moved, and why
dvc plots diff HEAD~1 --open               # the curves, side by side
git add dvc.lock metrics.json plots dvclive params.yaml
git commit -m "Deeper trees, tfidf features: F1 0.910 → 0.931"
dvc push && git push

# A sweep, in parallel, isolated
dvc exp run --queue -S train.max_depth=4,8,16,24
dvc exp run --run-all --jobs 4
dvc exp show --only-changed --sort-by f1 --sort-order desc
dvc exp branch exp-a1b2c deeper-trees
dvc exp remove --all && dvc gc --dry-run --all-commits

Ten decisions in there are the whole lesson of this level:

Where you are now

You can explain the cache layout and why hashing dominates, choose link types deliberately, template a pipeline instead of copy-pasting stages, use every output modifier correctly, depend on external and versioned sources, tier storage across remotes, run parallel isolated sweeps and share them, log training dynamics with DVCLive, render declarative plots for reviewers, enforce reproducibility in CI, lay out a monorepo, and debug the subtle failures where nothing errors.

Senior takes every one of these further, with a security, scale, or ownership dimension. Who can read and who can write each remote, and how credentials stop being stored at all. Immutability, retention, and object-lock on the storage side, and what that means for dvc gc. Cost modelling: storage class, egress, and the real price of a 200 GB intermediate. Hashing and transfer at terabyte scale, and when to stop using DVC for a given dataset. Data registries as a product with a release cadence. Lineage and audit trails for regulated work. GDPR deletion against a content-addressed, immutable cache. CI/CD with GPU and self-hosted runners. Where DVC stops and a feature store, lakehouse, or table format begins.

Part two of three. A cumulative review of Beginner and Mid-level material, organised by topic rather than by level, in about thirty-five minutes. Fast review first, common questions at the end. Senior reviews all three.

Foundations, in one screen

DVC versions data and ML pipelines on top of Git. A small hash-bearing pointer goes into Git; the bytes go into object storage. dvc.lock records every input and output hash from the last successful run, which is how dvc repro reruns only what changed.

git checkout moves pointers; dvc checkout moves data. dvc install adds hooks so you rarely need the second by hand.

Never dvc add a pipeline output, because outs owns it. Always commit dvc.lock and never hand-edit it. Credentials never go in .dvc/config, which is committed; use the provider's credential chain or .dvc/config.local.

The cache, precisely

├── 8f/2c4b19e0a7d3f1c6b8a2e4d7091f3b        ← a file's contents
└── 3a/7f9c2e8b1d4506….dir                  ← JSON listing for a directory

MD5 by default, chosen for speed rather than cryptographic strength, which is the honest answer. The first two hex characters shard the directory. A .dir object maps relative paths to per-file hashes, so files inside a tracked directory are still deduplicated individually.

Three consequences worth stating without prompting:

Hashing is the bottleneck, not transfer. dvc add on 50 GB reads 50 GB. DVC caches hashes by (path, size, mtime, inode) in .dvc/tmp/, which is why the second add is near-instant and the first run after a clone feels slow.

Small files are disproportionately expensive. A million 4 KB files means a million hash operations, a million objects, and a .dir listing of a million rows. The fix is sharding.

Deduplication is global to the cache. Identical content anywhere (two branches, two projects on a shared cache) occupies one object.

Link types

dvc config cache.type reflink,copy        # the safe default
dvc config cache.protected true           # working files read-only
dvc doctor                                 # what your filesystem supports

Pipeline templating

vars:
  - params.yaml
  - config/paths.yaml:paths

stages:
  featurize:
    foreach: [tfidf, word2vec, embeddings]
    do:
      cmd: python src/featurize.py --method ${item}
      deps: [src/featurize.py, data/prepared.csv]
      outs: [data/features-${item}.pkl]

  benchmark:
    matrix:
      model: [rf, xgb, lgbm]
      split: [0.1, 0.2]
    do:
      cmd: python src/bench.py --model ${item.model} --split ${item.split}
      outs: [out/${item.model}-${item.split}.json]

foreach generates one stage per item; matrix takes the cross product: six stages from six lines. foreach also accepts a dictionary, giving ${key} and ${item.field}. Generated names use @: featurize@tfidf. vars pulls shared values from a file or defines them inline.

Output modifiers

Two carry real weight. persist: true changes the execution contract: DVC normally deletes outputs before rerunning, so persisting means the stage is no longer a function of its inputs. push: false is the cost lever: a 200 GB intermediate that rebuilds in four minutes should not be in object storage.

External data

dvc import-url s3://landing/2024-05-01/events.parquet data/events.parquet
dvc import-url --no-download s3://landing/huge.parquet data/huge.parquet
dvc update data/events.parquet.dvc

deps:
  - s3://landing/exports/current/events.parquet   # external dep, hashed remotely

Multiple remotes

dvc remote add -d origin s3://prod-bucket/dvc
dvc remote add archive s3://glacier-bucket/dvc
dvc remote modify origin jobs 16          # transfer parallelism
dvc remote modify origin verify true      # re-hash on download
dvc remote modify origin sse AES256
dvc remote modify origin version_aware true
dvc push --remote archive

An output can name its own remote with remote:, which is how you tier storage without moving anything by hand.

Experiments at scale

dvc exp run --queue -S train.max_depth=4,8,16,24
dvc exp run --run-all --jobs 4            # each in an isolated temp workspace
dvc queue status / logs / stop
dvc exp show --only-changed --sort-by f1 --sort-order desc
dvc exp show --csv > results.csv
dvc exp branch exp-a1b2c deeper-trees
dvc exp push origin exp-a1b2c
dvc exp remove --all

The critical mechanic: a queued experiment runs in an isolated temporary workspace, which is what makes four parallel workers safe rather than a race over models/model.pkl.

DVCLive and plots

from dvclive import Live

with Live() as live:
    live.log_params({"lr": lr, "batch_size": bs})
    for epoch in range(epochs):
        live.log_metric("train/loss", train_loss)
        live.log_metric("val/accuracy", val_acc)
        live.next_step()
    live.log_artifact("models/model.pkl", type="model")

metrics:
  - dvclive/metrics.json:
      cache: false
plots:
  - dvclive/plots/metrics:
      x: step
      cache: false

DVCLive turns a final number into a curve, so you can see why a run won rather than only that it did. Framework callbacks exist for Lightning, Keras, Hugging Face, XGBoost, and LightGBM.

dvc plots diff HEAD~1 --open
dvc plots templates

DVC in CI

- uses: actions/checkout@v4
  with: { fetch-depth: 0 }              # metrics diff needs history
- uses: aws-actions/configure-aws-credentials@v4
  with: { role-to-assume: ${{ vars.DVC_ROLE_ARN }}, aws-region: eu-west-1 }
- uses: actions/cache@v4
  with:
    path: .dvc/cache
    key: dvc-${{ hashFiles('dvc.lock') }}
    restore-keys: dvc-
- run: dvc pull --allow-missing
- run: dvc repro
- name: Fail if the lock is stale
  run: test "$(dvc status --json)" = "{}"
- run: dvc metrics diff origin/main --md >> "$GITHUB_STEP_SUMMARY"
- run: dvc push
  if: github.ref == 'refs/heads/main'

Monorepos

dvc repro projects/churn/dvc.yaml
dvc repro --all-pipelines

One .dvc/cache and one remote, a dvc.yaml per subdirectory, cross-project dependencies as relative paths. The alternative is separate repositories consumed with dvc import --rev.

Choose on ownership, not tidiness: one team owning everything favours a monorepo; a dataset with its own owner and cadence deserves its own repository.

Debugging, one level deeper

dvc repro --dry                     # what would run
dvc repro --downstream train        # train and everything after
dvc repro --force-downstream        # rerun downstream even if hashes matched
dvc repro --single-item train       # exactly one stage
dvc repro --pull                    # fetch missing deps as needed
dvc repro -v                        # every hash comparison
dvc data status --granular          # per-file changes inside a directory
dvc diff HEAD~3                     # which tracked files changed

Common interview questions

1. What is in the DVC cache, and how is it laid out?Content-addressed objects under .dvc/cache/files/md5/, sharded by the first two hex characters of the hash. A tracked directory gets a .dir object, a JSON listing mapping each relative path to its own hash, so files inside a directory are still deduplicated individually.
2. Why MD5 rather than something cryptographic?Speed. DVC hashes every byte of every tracked file, so the algorithm is on the critical path. It is a content address, not a security control. You can switch to sha256 with core.hash_algorithm, but it changes every object's location, so it is a migration rather than a setting.
3. What dominates dvc add time?Hashing, not transfer. Adding 50 GB reads 50 GB from disk. DVC caches file hashes keyed on path, size, mtime, and inode, which is why the second add is near-instant and the first run after a clone feels slow.
4. Explain the link types and the risk.reflink is copy-on-write: no extra space and safe to edit. hardlink and symlink use no extra space but the working file is the cache object, so editing in place rewrites cached history for every commit that shared that hash, with no warning. copy is safe and doubles the space. Default to reflink,copy, and set cache.protected true if you use hardlinks.
5. How do you handle a dataset of a million small files?Shard it. A million 4 KB objects means a million hash operations, a million transfers, and a `.dir` listing of a million rows. Packaging into tar shards, Parquet, or WebDataset cuts dvc add by an order of magnitude, turns request-bound transfers into throughput-bound ones, and speeds up the data loader as a bonus.
6. How do you avoid copy-pasting ten near-identical stages?foreach for a list or a dictionary, matrix for a cross product, and vars for shared values pulled from a file. Generated stages are addressable with an @ suffix: featurize@tfidf.
7. Name the output modifiers and when you use each.cache: false puts a small text output in Git so it is diffable with no pull. persist: true stops DVC deleting the output before a rerun, which is needed for logs and checkpoints and means the stage is no longer a function of its inputs. push: false keeps a regenerable intermediate out of the remote. remote: sends an output to a specific, usually colder, remote.
8. How do you depend on data that lives outside the repository?dvc import-url to track and download it with upstream version tracking, --no-download to record the pointer without pulling, or an external path directly in a stage's deps so DVC hashes the remote object. All three are reproducible; a moving path is not, so pin a dated immutable one when the result must be rebuildable later.
9. Why would you use several remotes?Cost, geography, and access. A hot default plus a cold archive, tiered per output with remote:. Or a read-only remote for contributors and fork CI. Access control lives in the storage provider; the remote definition just means everyone agrees where data is.
10. How do you make dvc pull faster?Raise jobs on the remote or pass -j. In CI, cache .dvc/cache keyed on dvc.lock so an unchanged pipeline pulls almost nothing. Co-locate the runner with the bucket, since cross-region egress often costs more than the compute.
11. What does verify true do and why would you set it?It re-hashes objects after download and refuses ones that do not match. Without it, a corrupted or maliciously replaced object is restored silently and reported as success, because the pointer records what DVC expected and nothing checks the bytes.
12. How do you run twenty experiments in parallel without them clobbering each other?Queue them and use --run-all --jobs N. Each queued experiment executes in an isolated temporary workspace, so four workers do not all write models/model.pkl. Then dvc exp show --only-changed --sort-by to compare, and dvc exp branch to promote.
13. Where does the disk go during a sweep?Every experiment is a Git ref under refs/exps/ plus its own cached outputs. Thirty runs of a 2 GB model is 60 GB. Nothing cleans up automatically: dvc exp remove drops the refs, then dvc gc --all-commits reclaims the space.
14. What does DVCLive add over writing metrics.json at the end?Per-step logging, so you get curves rather than endpoints. That is what lets you see overfitting after epoch twelve rather than only that the final score was worse. It writes into a layout DVC understands, so wiring is two entries in dvc.yaml, and framework callbacks exist for the common trainers.
15. How do you enforce that the pipeline is reproducible?Run dvc repro in CI and then assert dvc status is empty as a required check. That fails the pull request when someone edited a script or a parameter without rerunning, so dvc.lock cannot drift from the code. Without that gate, "reproducible" is a claim.
16. Why does dvc metrics diff origin/main return nothing in CI?A shallow clone. actions/checkout defaults to depth 1, so the other revision is not present and the diff is silently empty. fetch-depth: 0 fixes it.
17. Monorepo or separate repositories?Decide on ownership. One team owning everything favours a monorepo: one cache, one remote, maximum deduplication, cross-project dependencies as plain paths, one CI config. A dataset with its own owner and release cadence deserves its own repository, consumed with dvc import --rev so consumers pin explicitly.
18. A stage reruns on every dvc repro even with no changes. Debug it.Something in deps changes every run: a timestamped log, a generated file, a directory containing a cache. Find it with dvc repro -v, which prints the hash comparison and names the differing dependency.
19. How do you resolve a dvc.lock merge conflict?Never by hand. The hashes describe a run that happened; a stitched file describes one that did not, and checking it out restores inconsistent data. Take one side, usually theirs, then run dvc repro and commit the regenerated lock.
20. One image in a 50,000-image directory changed. How do you find it?dvc data status --granular, which lists individual added, modified, and deleted files rather than reporting the whole directory as modified. That is how you discover a "changed dataset" is one corrupted file.
21. What is version_aware and when would you use it?Instead of hash-addressed object names, DVC writes files at their natural paths and relies on the bucket's own versioning for history. The bucket becomes browsable and readable by non-DVC tools, valuable when a data engineering team shares it, at the cost of cross-file deduplication.

Final self-test

• Describe the cache layout, including what a .dir object contains.
• Say why MD5, and what changing the algorithm costs.
• Name what dominates dvc add time, and why a second add is fast.
• Give the hardlink corruption failure in two sentences.
• Say what you do with a million small files, and the three benefits.
• Name four output modifiers and one use for each.
• Explain why a moving external dependency breaks reproducibility.
• Give three ways to make dvc pull faster.
• Say what verify true prevents.
• Explain how queued experiments avoid clobbering each other.
• Say where the disk goes during a sweep, and the two-step cleanup.
• Name the CI check that makes reproducibility enforceable.
• Give the reason dvc metrics diff is empty in CI.
• State the correct way to resolve a dvc.lock conflict.

Part two of three. At this level your pipelines run and your data is versioned, so the problems change character. They are no longer "why is my data the wrong version" but "why does CI rebuild everything", "why is dvc add taking twenty minutes", and "why does this reproduce locally and not in CI". Start with the error table, then the practices and practice cards underneath it.

Common errors at this level

Cumulative: Beginner's errors still apply, and these are the ones that appear once things work.

The practices that pay off most

Practice cards

1. Measure the sharding winGenerate 100,000 small files and time dvc add. Shard into 200 tars and time it again. Compare the two .dir object sizes and the object counts pushed.
2. Find your invalidation pointRun dvc repro -v after a small change and read which dependency's hash differed. It is usually earlier in the graph than you assumed.
3. Watch the hardlink corruptionOn a throwaway repo, set cache.type hardlink, add a file, commit, then edit the file in place. Check out the old revision and confirm you get the new content. Then enable cache.protected and try again.
4. Prove persist changes the contractAdd a log output, repro twice without persist and note it is recreated. Add persist: true and repro twice more.
5. Budget a sweepMeasure du -sh .dvc/cache, run a four-way sweep of a large model, and measure again. Multiply by your usual sweep count per month.
6. Break CI on purposeChange a script without running dvc repro, push, and confirm the dvc status gate fails. That failing check is the whole value of the workflow.
7. Time the CI cacheRun the pipeline workflow with and without the .dvc/cache cache step and compare wall-clock time.
8. Collapse ten stages into oneConvert a set of near-identical stages to foreach and confirm dvc dag shows an identical graph.
9. Resolve a lock conflict correctlyCreate the conflict deliberately on two branches. Resolve it by taking one side and running dvc repro, not by editing the YAML.

Making CI fast

Three changes, in order of return.

- uses: actions/checkout@v4
  with: { fetch-depth: 0 }                    # 2. metrics diff needs history

- uses: actions/cache@v4                       # 1. the biggest win
  with:
    path: .dvc/cache
    key: dvc-${{ hashFiles('dvc.lock') }}
    restore-keys: dvc-

- run: dvc pull --allow-missing -j 16          # 3. transfer parallelism
- run: dvc repro
- run: test "$(dvc status --json)" = "{}"      # the gate

Sharding, and when to do it

dvc add on a directory hashes every file. Beyond roughly ten thousand files that starts to hurt, and beyond a hundred thousand it dominates everything.

import tarfile, pathlib, itertools

files = sorted(pathlib.Path("data/raw-images").rglob("*.jpg"))
out = pathlib.Path("data/shards"); out.mkdir(parents=True, exist_ok=True)

for i in range(0, len(files), 512):
    with tarfile.open(out / f"shard-{i // 512:05d}.tar", "w") as tar:
        for f in files[i:i + 512]:
            tar.add(f, arcname=f.name)

stages:
  shard:
    cmd: python src/shard.py
    deps: [src/shard.py, data/raw-images]
    outs:
      - data/shards           # ~2,000 tars instead of 1,000,000 jpegs

Cache hygiene

The cache keeps every version of everything, which is the feature and the problem.

du -sh .dvc/cache                       # how big
find .dvc/cache -type f | wc -l         # how many objects
dvc cache dir                            # where
dvc cache dir --local /mnt/big/cache     # move it, machine-locally

dvc exp remove --all                     # drop experiment refs first
dvc gc --dry-run --all-commits           # then read what becomes collectable
dvc gc --all-commits                     # reclaim it

The order matters. Experiment outputs are reachable while their refs exist, so gc will not touch them. Dropping the refs first is what makes the space collectable.

"Reproduces locally, not in CI"

Five differences, roughly in the order they are the answer.

# In both environments
dvc status
python -c "import sklearn, numpy; print(sklearn.__version__, numpy.__version__)"
sha256sum requirements.lock
git rev-parse HEAD

Experiment queue hygiene

dvc exp run --queue -S train.max_depth=4,8,16,24
dvc exp run --run-all --jobs 4          # isolated temp workspaces
dvc queue status                         # queued / running / done
dvc queue logs exp-a1b2c                 # a specific run's output
dvc queue stop                           # after the current tasks finish
dvc queue kill exp-a1b2c                 # a specific run, now
dvc queue remove --all                   # clear the queue

dvc exp show --only-changed --sort-by f1 --sort-order desc
dvc exp show --csv > results.csv
dvc exp branch exp-a1b2c deeper-trees    # promote
dvc exp push origin exp-a1b2c            # share
dvc exp remove --all                     # then gc

Habits worth adopting now

Declare the dependency lock file in deps. requirements.lock or poetry.lock with hashes. It is the difference between reproducing the code and reproducing the run.

Prefer dated immutable external paths. exports/2024-05-01/events.parquet, never exports/current/.

Put jobs in .dvc/config. Everyone gets the faster transfers, not just whoever remembered the flag.

Set verify true on remotes you do not fully control. It costs download time and catches corruption instead of restoring it.

One responsibility per stage. A stage that prepares and trains reruns training when only preparation changed.

Write the metric into the commit message. git log --oneline becomes a results log.

stages:
  train:
    cmd: python src/train.py
    deps:
      - src/train.py
      - src/model.py                     # imports count
      - requirements.lock                # the environment is a dependency
      - data/features.pkl
    params: [train.n_estimators, train.max_depth]
    outs:
      - models/model.pkl
      - logs/train.log:
          persist: true
    metrics:
      - dvclive/metrics.json:
          cache: false
    plots:
      - dvclive/plots/metrics:
          x: step
          cache: false

Senior tips go further: the hardening pass every remote should get, verifying controls rather than trusting them, cost governance, incident playbooks for a deleted remote and an unreproducible model, running DVC as a platform, and the erasure problem.

This is part three of three. It closes the series by taking every topic from Beginner and Mid one level further (each one now has a security, scale, or ownership dimension) and adds the work you own when data versioning is your responsibility rather than your tool.

Up to this point the questions have been about making a pipeline reproduce. From here they are different: who can read and who can overwrite the data, whether a result from eighteen months ago can still be reconstructed and proven, what your storage costs, what happens when a customer asks you to delete their records from an immutable content-addressed store, and where DVC stops being the right tool.

Where this picks up

I am starting with the trust model, because every other decision in this track depends on it.

The trust model: the bucket decides, not DVC

A DVC remote is a storage bucket. Everything about who can do what to your data is decided in the storage provider, not in DVC, and DVC's design means the consequences differ from what people expect.

Three properties of DVC's model make the access question sharper than it looks:

Content-addressed writes are silent overwrites. Pushing an object writes to a path derived from its hash. Identical content is a no-op, but a client that can write can also replace an object at a hash path with different bytes, and every commit referencing that hash now resolves to the wrong data with no error anywhere. That is the reason write access matters more than it first appears.

A pointer is not proof. dvc.lock records the hash DVC expected. Nothing verifies the bytes on download unless you ask it to:

dvc remote modify origin verify true      # re-hash after every download
dvc pull --verify                          # or per-invocation

Deletion is the dangerous permission. dvc gc --cloud deletes objects on the remote, permanently, based on what the running client considers reachable. A developer with delete permission and a stale clone can remove data every other branch depends on.

The access shape that works:

s3://ml-data-prod/dvc/            ← read: everyone   write: CI on main only   delete: nobody
s3://ml-data-dev/dvc/             ← read+write: developers                    delete: lifecycle only
s3://ml-data-public/dvc/          ← read: anonymous / fork CI                 write: CI on main

Credentials that are never stored

Mid put credentials in .dvc/config.local. At this level the goal is that there is nothing to store.

permissions:
  id-token: write            # request an OIDC token
  contents: read

steps:
  - uses: aws-actions/configure-aws-credentials@v4
    with:
      role-to-assume: ${{ vars.DVC_ROLE_ARN }}   # trust policy pins repo + branch
      aws-region: ${{ vars.AWS_REGION }}
  - run: dvc pull && dvc repro && dvc push

The role's trust policy is where the security lives, and it should pin the subject as narrowly as your workflow allows:

{
  "Condition": {
    "StringEquals": {
      "token.actions.githubusercontent.com:sub":
        "repo:my-org/my-repo:ref:refs/heads/main"
    }
  }
}

For humans, the equivalent is the provider's own SSO: aws sso login, gcloud auth application-default login, an SSH agent for ssh:// remotes. DVC picks these up automatically because it uses each provider's standard credential chain.

dvc remote modify --local origin profile ml-readonly    # a named SSO profile
dvc remote modify --local origin ssh_private_key_path ~/.ssh/id_dvc
dvc doctor                                              # confirm the driver sees credentials

Immutability, retention, and object lock

DVC's cache is append-only by convention. Making it append-only by enforcement is a storage-side decision, and it is what turns "we can reproduce that" into something you can defend.

# S3: versioning plus governance-mode object lock
aws s3api put-bucket-versioning --bucket ml-data-prod \
  --versioning-configuration Status=Enabled

aws s3api put-object-lock-configuration --bucket ml-data-prod \
  --object-lock-configuration '{
    "ObjectLockEnabled":"Enabled",
    "Rule":{"DefaultRetention":{"Mode":"GOVERNANCE","Days":365}}
  }'

The interaction with dvc gc is the part people miss:

Cost: what your data costs

Mid introduced push: false as a cost lever. "Storage is cheap" stops being true once a team runs sweeps, so the full cost model matters.

Four charges apply, and only the first is the one people think about:

The arithmetic that changes behaviour: a 2 GB model, thirty experiments per sweep, four sweeps a month, kept for a year is roughly 2.9 TB of accumulated artifacts for one project. At standard-class pricing that is a real line item, and almost none of it will ever be read again.

Five levers, roughly in order of return:

1. push: false on regenerable intermediatesIf it takes four minutes to rebuild and 200 GB to store, do not store it. This is usually the largest single saving and it costs nothing.
2. Garbage-collect experiments on a scheduledvc exp remove plus a lifecycle rule. Unreferenced sweep artifacts are the biggest source of silent growth.
3. Lifecycle to a colder classObjects untouched for 90 days move to infrequent-access or archive. Reproducibility is preserved; only retrieval latency changes.
4. Shard small filesA million 4 KB objects is a request-count problem and a hashing problem. Tar or Parquet shards fix both, and speed up your data loader as a bonus.
5. Co-locate compute and storageEgress from another region or another cloud can exceed the storage cost outright. Put the runner where the bucket is.

# Where is the money? Object counts and sizes by prefix
aws s3 ls --summarize --human-readable --recursive s3://ml-data-prod/dvc/ | tail -3

# Lifecycle: standard → IA at 90 days → Glacier at 365
aws s3api put-bucket-lifecycle-configuration --bucket ml-data-prod \
  --lifecycle-configuration file://lifecycle.json

# DVC-side hygiene
dvc exp remove --all
dvc gc --dry-run --all-commits --cloud
du -sh .dvc/cache

Scale: terabytes and millions of files

DVC handles large data well and many small files badly. Knowing exactly where the wall is means you can design around it instead of hitting it.

The sharding pattern, which is the single most effective change for image and audio datasets:

stages:
  shard:
    cmd: python src/shard.py --in data/raw-images --out data/shards --size 512
    deps: [src/shard.py, data/raw-images]
    outs:
      - data/shards            # ~2,000 tar files instead of 1,000,000 jpegs

That gives DVC two thousand objects instead of a million: dvc add drops from tens of minutes to under one, the .dir listing shrinks by three orders of magnitude, and transfers become throughput-bound rather than request-bound. Your data loader gets faster too, because sequential reads beat a million opens.

Other levers at this scale:

dvc remote modify origin jobs 32                 # transfer parallelism
dvc config core.checksum_jobs 16                 # hashing parallelism
dvc remote modify origin version_aware true      # natural paths + bucket versioning
dvc config cache.type reflink,hardlink           # avoid duplicating TBs locally
dvc add --to-remote data/huge.parquet            # transfer straight to the remote
dvc import-url --no-download s3://…/huge.parquet  # track without pulling

version_aware matters at this scale: instead of hash-addressed objects, DVC stores files at their natural paths and relies on the bucket's own versioning for history. That makes the bucket browsable and readable by non-DVC tools, a real benefit when a data engineering team needs the same files, at the cost of losing cross-file deduplication.

Data registries as a product

Mid used dvc import to consume a dataset. Owning the producing side is a different job, and treating it as a product rather than a folder is what makes it adopted.

data-registry/
├── .dvc/config                     # the canonical remote
├── datasets/
│   ├── customers/
│   │   ├── 2024-05.parquet.dvc
│   │   └── README.md               # schema, provenance, licence, owner
│   └── transactions/
│       └── 2024-05.parquet.dvc
├── CODEOWNERS
└── README.md                       # how to consume, and the release policy

# Producer: publish a dated, immutable version and tag it
dvc add datasets/customers/2024-05.parquet
git add datasets/customers/2024-05.parquet.dvc
git commit -m "customers: 2024-05 snapshot, 4.2M rows"
git tag customers-2024.05
dvc push && git push --tags

# Consumer: pin an explicit revision
dvc import --rev customers-2024.05 \
  git@github.com:org/data-registry.git datasets/customers/2024-05.parquet

# Consumer: update deliberately, later
dvc update --rev customers-2024.06 datasets/customers/2024-05.parquet.dvc

What makes it a product rather than a bucket:

stages:
  validate:
    cmd: python validate.py datasets/customers/2024-05.parquet
    deps: [validate.py, datasets/customers/2024-05.parquet]
    metrics:
      - reports/customers-2024-05.json:
          cache: false          # row counts, null rates, schema hash — in Git

Lineage, provenance, and audit

dvc.lock already records which inputs produced which outputs. For regulated work you need to turn that into an answer a reviewer or an auditor accepts, and the gap between the two is smaller than it looks, but it is not zero.

The question is always some form of: "This model made a decision affecting a customer. Prove what trained it."

# Which commit is this model from? Start with its hash.
git log --all -S "9e8d7c6b5a4f3e2d" -- dvc.lock

# Then reconstruct that revision completely
git checkout <sha>
dvc pull
dvc repro --dry               # confirm nothing is out of date

# What differed between two revisions?
dvc diff <old-sha> <new-sha>
dvc params diff <old-sha> <new-sha>
dvc metrics diff <old-sha> <new-sha>

That last row is the honest gap. dvc.lock pins your data and your code; it does not pin your Python version, your CUDA driver, or the exact wheel of scikit-learn that was resolved. Closing it takes three things:

stages:
  train:
    cmd: python src/train.py
    deps:
      - src/train.py
      - requirements.lock       # fully pinned, hash-verified
      - Dockerfile              # the environment as code
      - data/train.parquet

Pin the dependency set with hashes (pip-compile --generate-hashes, poetry.lock, uv.lock). Declare it as a dependency, so a change invalidates the stage. Record the container digest, not a tag, so the environment is addressable:

live.log_params({
    "git_sha": subprocess.check_output(["git", "rev-parse", "HEAD"]).decode().strip(),
    "image_digest": os.environ.get("IMAGE_DIGEST", "unknown"),
    "data_hash": read_lock_hash("data/train.parquet"),
})

Deleting data from a content-addressed store

This is the hardest problem in the guide, and precision matters because the naive answer is wrong.

A subject exercises their right to erasure. Their records are in customers-2024-03.parquet, which is in your DVC cache, on your remote, in three developers' local caches, and referenced by every commit that trained a model on it.

Deleting the object breaks history. Every commit referencing that hash now fails to check out. Not deleting it violates the law. And rewriting Git history to remove the pointer does not touch the data at all.

If you are already in the bad situation, the sequence is:

1. Establish the blast radiusWhich hashes contain the data, and which commits, tags, and models reference those hashes. This is a git log -S over dvc.lock and every .dvc file.
2. Decide what must surviveUsually the model and its metrics must remain auditable while the training data must go. That is an acceptable outcome in most regimes, and it needs to be written down.
3. Rebuild the dataset without the subjectProduce a new version, push it, and update the pointers on active branches.
4. Delete the old objects everywhereRemote, all local caches, all CI caches, and any registry mirror. Old versions in a versioned bucket count.
5. Record the tombstoneA committed note stating which hash was erased, when, why, and under what authority. An unresolvable pointer with no explanation is worse than one with a documented reason.
6. Close the classMove personal data out of the versioned store so the next request is a key deletion rather than an incident.

CI/CD at scale, and CML

Mid put dvc repro in CI on a hosted runner. Real training needs GPUs, long timeouts, and data locality, and reporting that a reviewer reads.

name: Train
on:
  pull_request:
  workflow_dispatch:

permissions:
  id-token: write
  contents: read
  pull-requests: write

jobs:
  train:
    runs-on: [self-hosted, gpu]          # or a cloud runner provisioned per job
    timeout-minutes: 480
    steps:
      - uses: actions/checkout@v4
        with: { fetch-depth: 0 }

      - uses: aws-actions/configure-aws-credentials@v4
        with:
          role-to-assume: ${{ vars.DVC_ROLE_ARN }}
          aws-region: ${{ vars.AWS_REGION }}

      - run: pip install -r requirements.lock --require-hashes
      - run: dvc pull --allow-missing -j 32
      - run: dvc repro

      - name: Comment the report on the pull request
        env:
          REPO_TOKEN: ${{ secrets.GITHUB_TOKEN }}
        run: |
          {
            echo "## Metrics"
            dvc metrics diff origin/main --md
            echo "## Parameters"
            dvc params diff origin/main --md
            echo "## Data"
            dvc diff origin/main --md
          } > report.md
          dvc plots diff origin/main --show-vega > plots.json || true
          cml comment create report.md

      - run: dvc push
        if: github.ref == 'refs/heads/main'

Four scale concerns and their answers:

Where DVC stops

Naming the boundary is itself a senior signal, and it lands better than defending DVC for every shape of data.

The combination that works in practice: DVC for file-shaped data and reproducible pipelines, a table format for warehouse-shaped data, an orchestrator for scheduling, and DVC's own experiment tracking or MLflow for the model registry. They compose because they solve different problems; the mistake is expecting any one of them to do all four.

Running DVC as a platform

Once more than one team versions data, the highest-leverage work stops being technical and becomes about defaults. Four decisions cover most of it.

Publish a template repository, not documentation. A dvc.yaml skeleton, a .dvc/config with the right remote and cache settings, a CI workflow with OIDC and the dvc status gate, and a params.yaml. Teams inherit every good decision by cloning; a wiki page is read once and ignored.

Own the remote layout and its policy. One prefix per environment, one bucket policy, versioning on, delete removed from human credentials, lifecycle rules for retention. Teams should not each invent this.

Make the safe path the fast path. If dvc pull from the sanctioned remote is fast and well-documented while copying from a colleague's laptop is slow, nobody copies from laptops. Regional co-location and a tuned jobs setting do more for compliance than a policy statement.

Measure and publish the numbers. Without them, "our data is under control" is an opinion:

Incident playbooks

Under pressure, ordering matters more than knowledge. Five shapes, with the first move for each.

A remote object is corrupt or missing

Confirm it first: dvc pull -v names the hash and the remote. Then check whether bucket versioning has a previous version of that key; if so, restore it and you are done. If not, find any machine or CI cache that still holds the object (find .dvc/cache -name '<rest-of-hash>') and push from there. If nothing has it, the data is gone: identify every commit referencing the hash, rebuild the artifact if its inputs survive, and record a tombstone if it cannot be rebuilt. Then turn on verify true and versioning so the next occurrence is recoverable.

dvc gc --cloud deleted live data

Stop all pushes immediately, because a subsequent push can overwrite the delete markers you need. Restore from bucket versioning. This is the entire reason versioning is non-optional. Then remove delete permission from every human credential and move retention to an audited lifecycle policy. The post-mortem action is a permission change, not a training session.

A model cannot be reproduced

Work through the layers in order, because each is cheaper to check than the next: is the code commit findable, does dvc pull succeed for every dep in dvc.lock, does dvc repro --dry report up to date, and does an actual rerun produce matching metrics. The break is almost always the last two, and almost always the environment: an unpinned wheel, a different CUDA version, an undeclared random seed. Fix it by adding the missing pin to deps, not by documenting the discrepancy.

The cache filled the disk mid-sweep

A credential leaked

Rotate before anything else, because the credential may already be in a log, a cache, or a fork. Then audit the bucket's access logs for the exposure window, looking specifically for writes and deletes rather than only reads. Replace the mechanism, not just the value: if a long-lived key leaked, the fix is OIDC, because rotating a key you still store only resets the clock.

The review checklist

Take this checklist away from this level. Run it against any DVC project (your own, or one you are reviewing) and it catches nearly everything covered here before it becomes an incident.

The complete picture

The series' final artefact: every level's topics, hardened. Nothing in it is new.

vars:
  - params.yaml

stages:
  # Immutable dated source: reproducible in eighteen months
  ingest:
    cmd: python src/ingest.py
    deps:
      - src/ingest.py
      - requirements.lock
      - s3://landing/exports/2024-05-01/events.parquet
    outs:
      - data/raw.parquet

  # Sharded once, so hashing and transfer stay bounded
  shard:
    cmd: python src/shard.py --size 512
    deps: [src/shard.py, data/raw.parquet]
    outs:
      - data/shards

  prepare:
    cmd: python src/prepare.py
    deps: [src/prepare.py, requirements.lock, data/shards]
    params: [prepare.test_size, prepare.random_state]
    outs:
      - data/train.parquet
      - data/test.parquet:
          remote: archive          # cold storage: read rarely, kept for audit

  featurize:
    foreach: ${featurize.methods}
    do:
      cmd: python src/featurize.py --method ${item}
      deps: [src/featurize.py, data/train.parquet]
      params: [featurize.max_features]
      outs:
        - data/features-${item}.pkl:
            push: false            # four minutes to rebuild; do not store 200 GB

  train:
    cmd: python src/train.py
    deps:
      - src/train.py
      - requirements.lock          # hash-pinned; a wheel change invalidates the stage
      - Dockerfile                 # the environment, as code
      - data/features-tfidf.pkl
    params: [train.n_estimators, train.max_depth, train.learning_rate]
    outs:
      - models/model.pkl
      - logs/train.log:
          persist: true
    metrics:
      - dvclive/metrics.json:
          cache: false
    plots:
      - dvclive/plots/metrics:
          x: step
          cache: false

  evaluate:
    cmd: python src/evaluate.py
    deps: [src/evaluate.py, models/model.pkl, data/test.parquet]
    metrics:
      - metrics.json:
          cache: false
    plots:
      - plots/confusion.csv:
          template: confusion
          x: actual
          y: predicted
          cache: false

# .dvc/config — committed. No credentials anywhere in it.
[core]
    remote = origin
    autostage = true
    checksum_jobs = 16
['remote "origin"']
    url = s3://ml-data-prod/dvc
    jobs = 32
    verify = true                  # corruption is caught, not restored
    sse = AES256
['remote "archive"']
    url = s3://ml-data-archive/dvc
    verify = true
[cache]
    type = reflink,copy
    protected = true               # in-place edits fail loudly

# Verify, do not assume
dvc status && dvc status -c                 # local and remote agreement
dvc pull --verify                           # re-hash everything on download
dvc repro --dry                             # nothing stale
git log --all -S "$MODEL_HASH" -- dvc.lock  # trace a model to its commit
dvc gc --cloud --dry-run --all-commits      # never without --dry-run first
aws s3api get-bucket-versioning --bucket ml-data-prod    # confirm survivability

Where the series leaves you

Across the three levels you have gone from dvc add on a CSV to owning data versioning as a platform. The same topics carried all the way through, each time with more depth:

You should now be able to look at a DVC project and see not just what it does but what it guarantees: whether a result from last year can be rebuilt, whether the environment is pinned as well as the data, who can overwrite the bytes a model was trained on, whether a corrupted object would be noticed, whether a mistaken gc is survivable, what it costs per month, and whether the versioned artifacts contain data somebody could lawfully demand you erase.

That review instinct outlasts any specific command, because, as with pipelines and containers, most data incidents are prevented at review time rather than at recovery time.

Part three of three, and the one to read if you only read one. A cumulative review of the entire series (foundations, pipeline machinery, and the governance and scale work a senior owns) organised by topic rather than by level. About fifty minutes. Fast review first, common questions at the end.

Part one: foundations

DVC versions data and ML pipelines on top of Git. A small hash-bearing pointer goes into Git; the bytes go into object storage. dvc.lock records every input and output hash from the last successful run, which is how dvc repro reruns only the stages whose inputs changed.

Why not Git alone: full history of every large binary, in every clone, forever. Why not LFS: it fixes size and nothing else: no pipelines, no graph, no metrics, no experiments, and storage tied to your Git host.

git checkout moves pointers; dvc checkout moves data. dvc install adds hooks so the second is automatic.

dvc add, in four steps

Hash the contents → move to the content-addressed cache → write the .dvc pointer → add a .gitignore entry. A clean git status afterwards is the signal it worked.

The commands and the Git parallel

The rules that never change

Never dvc add a pipeline output, because outs owns it. Always commit dvc.lock, never hand-edit it. Declare every input in deps or the stage silently skips. Parameters go in params.yaml and must be declared. Metrics and plots use cache: false so they diff from Git with no pull. Credentials never go in .dvc/config.

The traps

Part two: pipeline and cache machinery

The cache, precisely

├── 8f/2c4b19e0a7d3f1c6b8a2e4d7091f3b       ← file contents
└── 3a/7f9c2e8b1d4506….dir                 ← JSON listing for a directory

MD5 by default: a content address chosen for speed, not a security control. Two-character shard directories. A .dir object maps relative paths to per-file hashes, so files inside a directory still deduplicate individually. Hashing dominates, not transfer; DVC caches hashes by (path, size, mtime, inode), which is why a second add is near-instant.

Link types

Default reflink,copy; add cache.protected true if you use hardlinks so in-place edits fail loudly.

Templating and modifiers

stages:
  featurize:
    foreach: [tfidf, embeddings]        # matrix: for a cross product
    do:
      cmd: python src/featurize.py --method ${item}
      outs:
        - data/features-${item}.pkl:
            push: false                  # regenerable; do not store 200 GB

External data, remotes, experiments

dvc import-url tracks an external source with dvc update; --no-download records the pointer only; an external path in deps is hashed remotely. A moving external path is a reproducibility hole, so pin dated, immutable paths.

jobs is the biggest pull speed lever. verify true re-hashes on download. version_aware uses natural paths plus bucket versioning: browsable, no cross-file dedup.

Queued experiments run in isolated temporary workspaces, which is what makes --run-all --jobs 4 safe. They are Git refs under refs/exps/ plus cached outputs, so thirty runs of a 2 GB model is 60 GB, cleared with dvc exp remove then dvc gc --all-commits.

DVCLive, and CI

DVCLive logs per step, turning endpoints into curves. In CI: OIDC for credentials, fetch-depth: 0 for metrics diffs, .dvc/cache cached on dvc.lock, and dvc status empty as a required check, the gate that makes reproducibility enforceable. Push only from the protected branch.

Part three: trust, scale, and ownership

The trust model

Three properties make access sharper than it looks:

Content-addressed writes are silent overwrites. A client with write access can replace the object at a hash path, and every commit referencing that hash now resolves to wrong data with no error.

A pointer is not proof. Nothing verifies bytes on download unless verify true is set.

Deletion is the dangerous permission.

Credentials with nothing stored

The trust policy is where the security lives, so pin the sub to a branch. For humans, the provider's SSO; DVC uses each provider's standard credential chain. A fork build must never hold a write credential.

Immutability and retention

Retention belongs in a lifecycle policy (declarative, audited, and not triggerable by a stale clone) not in dvc gc --cloud.

Cost

Levers in order: push: false on regenerable intermediates, scheduled experiment cleanup, lifecycle to a colder class, shard small files, co-locate compute and storage.

Scale, and where DVC stops

DVC has no scheduler. You run dvc repro from an orchestrator: the orchestrator schedules, DVC decides what needs to run and guarantees the lineage.

Lineage and audit

That last row is the honest gap. dvc.lock pins data and code, not the Python version, the wheels, or the CUDA driver. Close it by hash-pinning a lock file and declaring it, plus the Dockerfile or image digest, in deps. The strongest audit answer is a scheduled rebuild of a past release, not a diagram.

Erasure versus content addressing

Deleting an object breaks history; keeping it violates the law; rewriting Git history does not touch the bytes.

The resolution is always architectural and upstream of the tool: do not put erasable data in an immutable store.

The review checklist

Common interview questions

1. Explain DVC to an engineer who has never used it.Version control for data and ML pipelines, on top of Git. A small pointer containing a content hash goes into Git; the bytes go into object storage. dvc.yaml declares stages with their inputs and outputs, and dvc.lock records the hashes of what ran, so one commit reproduces a result rather than describing one, and dvc repro reruns only what changed.
2. What is the single most dangerous DVC command, and why?dvc gc --cloud --workspace. It deletes every object on the shared remote that is not needed by the currently checked-out revision, which on a repository with twenty branches destroys nineteen branches' data permanently, and with a zero exit code. The mitigations are removing delete permission from human credentials, doing retention as an audited lifecycle policy, and enabling bucket versioning.
3. Why does write access to a data remote matter more than it appears?Objects live at paths derived from their hash. A client that can write can also replace the object at a hash path with different bytes, and every commit referencing that hash then resolves to the wrong data with no error anywhere, because a pointer records what DVC expected, and nothing checks the bytes unless verify true is set.
4. How do you give CI credentials without storing a secret?OIDC or workload identity: the CI provider mints a short-lived token, and the cloud role's trust policy pins the subject to a specific repository and branch. Nothing is stored, so nothing can leak, and a throwaway branch cannot assume the role. For humans, the provider's own SSO, since DVC uses each provider's standard credential chain.
5. What must a fork pull request not be able to do?Write to the data remote. A fork build runs contributor code with whatever identity the job holds, so a write credential means a contributor can overwrite datasets. Give fork builds a read-only remote or none, and split the workflow so the privileged half (push, promotion, registry updates) never executes fork-supplied code.
6. How do you make a data remote trustworthy?Bucket versioning so any overwrite or delete is recoverable, object lock so deletion needs a specific privilege, verify true so corruption is refused rather than silently restored, encryption at rest, and delete permission removed from every human credential. Retention becomes a declarative lifecycle policy rather than someone running gc.
7. Where does the money go?Four charges: storage by bytes and class, requests by object count, egress by bytes leaving the region, and retrieval from cold classes. The surprise is usually requests on a million-small-file dataset, or egress because CI runs in a different region from the bucket. The largest single saving is almost always push: false on intermediates that regenerate in minutes.
8. Where is DVC's scaling wall, and what do you do about it?Many small files. A million 4 KB objects means a million hash operations, a million transfers, and a .dir listing of a million rows. Sharding into tars, Parquet, or WebDataset typically improves dvc add by more than tenfold, turns request-bound transfers into throughput-bound ones, and speeds up the data loader. Large files are fine, because hashing is I/O-bound and parallel.
9. Would you use DVC for a billion-row warehouse table?No. That is a table-format problem: Iceberg, Delta, or Hudi give you time travel and schema evolution natively, and DVC would be duplicating the warehouse. Track the query and its result hash so the pipeline stays reproducible, and let the lakehouse version the table. Insisting DVC handles every data shape is the wrong answer here.
10. Does DVC replace Airflow?No. It has no scheduler, no retries, and no distributed execution. You run dvc repro from an orchestrator: the orchestrator handles scheduling and failure, DVC decides what needs to run and guarantees the lineage. They compose because they solve different problems.
11. A model made a decision affecting a customer eighteen months ago. Prove what trained it.Find the commit from the model's hash with git log -S over dvc.lock, check that revision out, dvc pull, and confirm dvc repro --dry reports up to date. That gives you the data hashes, the code, the parameters, and the metrics. Then be honest about the gap: it does not prove the environment unless the lock file and image digest were also declared as dependencies.
12. What does dvc.lock not pin?The environment. Python version, resolved wheel versions, system libraries, GPU driver. Close it by hash-pinning a dependency lock file and declaring it in deps, recording the container digest rather than a tag, and logging both alongside the metrics.
13. How do you prove reproducibility rather than claim it?A scheduled job that picks a past release, checks it out on a clean runner, pulls, repros, and asserts the metrics match. It fails on the day an upstream dependency, a deleted object, or an unpinned library breaks reproducibility, rather than on the day someone needs it. That single job is worth more than any lineage documentation.
14. A customer demands erasure of their records. They are in a versioned dataset. What now?Establish the blast radius, which hashes contain the data and which commits and models reference them. Decide what must survive, usually the model and its metrics while the training data goes, and write that down. Rebuild the dataset without the subject, delete the old objects from the remote, every local cache, every CI cache, and any mirror, including old versions in a versioned bucket. Record a tombstone explaining the unresolvable pointer. Then close the class by moving personal data out of the versioned store.
15. Why are content addressing and erasure in tension?An immutable, replicated, hash-addressed store is precisely the wrong shape for "make this specific record disappear everywhere". The same tension exists in Git, in backups, and in any append-only log. The resolution is architectural and upstream of the tool: version derived, aggregated, or pseudonymised data, or encrypt per subject so erasure means destroying a key.
16. Can you enable compliance-mode object lock and still honour GDPR?Not on the same objects. Compliance mode means nobody can delete before expiry, including you, and including under a lawful erasure request. If you are subject to both obligations, keep personal data out of the immutable store or store it encrypted per subject. Decide before enabling it, because compliance mode cannot be walked back.
17. How do you run a data registry as a product?Dated immutable paths rather than latest.parquet, a Git tag per release so consumers pin a readable version, a README per dataset with schema, provenance, licence and owner, CODEOWNERS so someone is accountable, a validation stage writing row counts and schema hashes to a cache: false metrics file, and a published deprecation policy. Consumers use dvc import --rev and dvc update deliberately.
18. What is the failure mode of a shared registry, and the real fix?A team needs one extra column, cannot get it quickly, and forks. Six months later there are nine variants and no canonical source. The fix is turnaround time on requests, not policy. If a reasonable schema change takes three weeks, forking is rational and you will lose. Treat it as a product with a service level or do not call it canonical.
19. How do you run training in CI at scale?Self-hosted or per-job cloud runners with GPUs, in the same region as the bucket because egress often exceeds compute cost. A warm .dvc/cache on protected-branch runners, ephemeral runners for anything triggered by a pull request. OIDC for credentials, and CML comments carrying metrics, parameter, and data diffs plus rendered plots so a reviewer needs to run nothing.
20. Why are self-hosted runners a risk?They keep state. A persistent runner with a warm cache is fast and is also a shared, writable environment that has executed every contributor's code. Keep warm-cache runners on protected branches, make pull-request runners ephemeral, and never let a fork-triggered job hold the remote's write credential.
21. What would you measure to know your data versioning is healthy?Bytes and objects per project with the growth trend, the fraction of bytes in unreferenced experiments, the percentage of pipelines where dvc status is clean in CI, the age of the oldest successfully reproduced release, the number of remotes with human write access, and median dvc pull time in CI. Without numbers, "our data is under control" is an opinion.
22. How would you roll DVC out to six teams?A template repository: a working pipeline, the CI workflow with OIDC and the dvc status gate, the shared .dvc/config with the right remote and cache settings. Teams inherit every good decision by cloning, which works and a wiki page does not. Then own the remote layout and its policy centrally, make the sanctioned path the fast path, and publish the metrics.
23. What do you look for reviewing a DVC project?Not what it does, but what it guarantees. Can a result from last year be rebuilt; is the environment pinned as well as the data; who can overwrite the bytes a model trained on; would a corrupted object be noticed; is a mistaken delete survivable; what does it cost per month; and does any versioned artifact contain data somebody could lawfully demand you erase.

Final self-test

• Give the whole-series answer in four sentences, then name the most dangerous command.
• Explain why a write credential on a remote is worse than it appears.
• Say how CI gets credentials with nothing stored, and what the trust policy pins.
• Name four controls that make a remote trustworthy.
• List the four cost charges, and the largest single saving.
• Say where DVC's scaling wall is, and the fix plus its three benefits.
• Explain why a warehouse table is not a DVC problem.
• Say what DVC does and does not do relative to an orchestrator.
• Trace a model to its training data in three commands.
• Name what dvc.lock does not pin, and how to close the gap.
• Give the metric that proves reproducibility rather than claiming it.
• Explain the tension between erasure and content addressing, and the architectural fix.
• Say why compliance-mode object lock conflicts with an erasure obligation.
• Name five properties that make a data registry a product.
• Give two risks of self-hosted runners and their mitigations.
• Name three health metrics you would publish, and run the review checklist from memory.

Part three of three. The problems at this level are rarely about syntax. They are about a remote sixty people depend on, a model nobody can rebuild, a storage bill nobody owns, and an erasure request against an immutable store. Start with the error table, then the practices, verification, and playbooks underneath it.

Common errors at this level

Cumulative. Everything from Beginner and Mid still applies. These cause incidents rather than failed runs.

The practices that pay off most

Practice cards

1. Recover an overwriteOn a test bucket with versioning on, overwrite a DVC object by hand and restore the previous version. If you cannot do this, versioning is not enabled and that is your top priority.
2. Watch verify refuseCorrupt one object in a test remote, run dvc pull without verify and note it succeeds with wrong data. Enable verify true and repeat.
3. Read the destruction dry-runRun dvc gc --cloud --dry-run --workspace on a multi-branch repository and read the object count. Show that number to anyone holding delete permission.
4. Prove the trust policy is scopedTry to assume your CI role from a throwaway branch and confirm the refusal. If it succeeds, the sub condition is not pinned.
5. Rebuild something oldPick a release from three months ago, check it out on a clean machine, dvc pull, dvc repro, and compare the metrics. Note exactly where it broke.
6. Find the unpinned layerList everything that revision does not pin: Python version, wheels, CUDA, image digest. Add the lock file and Dockerfile to deps and confirm a change invalidates the stage.
7. Measure the real billTotal bytes, object count, and the fraction in experiment artifacts unread for three months. Calculate the lifecycle saving.
8. Classify for erasureTake one dataset and decide whether a subject could lawfully demand its deletion. If yes, count how many commits and models reference it.
9. Run the checklist on someone else's projectThe review checklist below, against a repository you did not write. It will find something.

The hardening pass every remote should get

# 1. Versioning: makes every other mistake survivable
aws s3api put-bucket-versioning --bucket ml-data-prod \
  --versioning-configuration Status=Enabled

# 2. Object lock in governance mode: deletion needs a specific privilege
aws s3api put-object-lock-configuration --bucket ml-data-prod \
  --object-lock-configuration '{"ObjectLockEnabled":"Enabled",
    "Rule":{"DefaultRetention":{"Mode":"GOVERNANCE","Days":365}}}'

# 3. Encryption at rest
aws s3api put-bucket-encryption --bucket ml-data-prod \
  --server-side-encryption-configuration \
  '{"Rules":[{"ApplyServerSideEncryptionByDefault":{"SSEAlgorithm":"AES256"}}]}'

# 4. Access logging, so a mistake is traceable
aws s3api put-bucket-logging --bucket ml-data-prod --bucket-logging-status file://logging.json

# 5. Lifecycle: retention as declarative policy, not a manual command
aws s3api put-bucket-lifecycle-configuration --bucket ml-data-prod \
  --lifecycle-configuration file://lifecycle.json

# .dvc/config — committed, no credentials
[core]
    remote = origin
    autostage = true
    checksum_jobs = 16
['remote "origin"']
    url = s3://ml-data-prod/dvc
    jobs = 32
    verify = true                 # corruption refused, not restored
    sse = AES256
['remote "archive"']
    url = s3://ml-data-archive/dvc
    verify = true
[cache]
    type = reflink,copy
    protected = true              # in-place edits fail loudly

Verifying, not assuming

Every control above should be checked rather than trusted.

# Is versioning actually on?
aws s3api get-bucket-versioning --bucket ml-data-prod

# Does verify catch a corrupted object?
dvc pull --verify

# Can a developer credential delete?
aws s3api delete-object --bucket ml-data-prod --key dvc/files/md5/xx/test 2>&1 | tail -1

# Is anything sensitive in the committed config?
git log -p --all -- .dvc/config | grep -iE 'key|secret|token|password' || echo clean

# Is the environment genuinely pinned?
grep -c 'sha256:' requirements.lock
dvc dag --outs | grep -q requirements.lock && echo "declared as a dep"

# Would an old release rebuild?
git checkout <old-tag> && dvc pull && dvc repro --dry

# How much would a cloud gc destroy?
dvc gc --cloud --dry-run --workspace | wc -l

Automate the mechanical checks so they do not depend on anyone remembering:

- name: dvc.lock must not be stale
  run: test "$(dvc status --json)" = "{}"

- name: dvc.lock must accompany dvc.yaml changes
  run: |
    git diff --name-only origin/main | grep -q '^dvc.yaml$' \
      && { git diff --name-only origin/main | grep -q '^dvc.lock$' \
           || { echo "::error::dvc.yaml changed without dvc.lock"; exit 1; }; } || true

- name: No credentials in the committed config
  run: |
    grep -iE 'access_key|secret|password|token' .dvc/config \
      && { echo "::error::credential in .dvc/config"; exit 1; } || true

- name: Bucket versioning must be enabled
  run: |
    test "$(aws s3api get-bucket-versioning --bucket ml-data-prod --query Status --output text)" = "Enabled"

Cost governance

# Where the bytes are
aws s3 ls --summarize --human-readable --recursive s3://ml-data-prod/dvc/ | tail -3

# DVC-side hygiene
du -sh .dvc/cache
find .dvc/cache -type f | wc -l
dvc exp remove --all
dvc gc --dry-run --all-commits

Incident playbooks

dvc gc --cloud deleted live data

Stop all pushes immediately, because a subsequent push can overwrite the delete markers you need to recover. Restore from bucket versioning. This is the entire reason versioning is non-optional. Then remove delete permission from every human credential and move retention to a lifecycle policy. The post-mortem action is a permission change, not a training session.

A remote object is corrupt or missing

Confirm it with dvc pull -v, which names the hash and the remote. Check bucket versioning for a previous version of that key; if present, restore and you are done. If not, find any machine or CI cache that still holds the object and push from there. If nothing has it, identify every commit referencing the hash, rebuild the artifact if its inputs survive, and record a tombstone if it cannot be rebuilt. Then enable verify true and versioning so the next occurrence is recoverable.

A model cannot be reproduced

Work the layers in order, cheapest first: is the commit findable from the model's hash, does dvc pull succeed for every dep in dvc.lock, does dvc repro --dry report up to date, and does an actual rerun produce matching metrics. The break is almost always the last two, and almost always the environment. Fix it by adding the missing pin to deps, not by documenting the discrepancy.

The cache filled the disk mid-sweep

A credential leaked

Rotate before anything else. It may already be in a log, a cache, or a fork. Then audit the bucket's access logs for the exposure window, looking for writes and deletes rather than only reads, because a write is the damaging case. Replace the mechanism, not just the value: if a long-lived key leaked, the fix is OIDC. Rotating a key you still store only resets the clock.

An erasure request lands

Establish the blast radius: which hashes contain the data, and which commits, tags, and models reference those hashes. Decide what must survive, usually the model and its metrics remain auditable while the training data goes, and write that decision down. Rebuild the dataset without the subject, push it, and update pointers on active branches. Delete the old objects from the remote, every local cache, every CI cache, and any mirror, including old versions in a versioned bucket. Record a committed tombstone naming the erased hash, the date, and the authority. Then close the class by moving personal data out of the versioned store.

Running DVC as a platform

Publish a template repository, not documentation. A working pipeline, the CI workflow with OIDC and the dvc status gate, and a .dvc/config with the right remote, verify, jobs, and cache settings. Teams inherit every good decision by cloning; a wiki page is read once.

Own the remote layout and its policy. One prefix per environment, versioning on, delete removed from humans, lifecycle rules for retention, access logging. Teams should not each invent this.

Make the safe path the fast path. If dvc pull from the sanctioned remote is fast and documented while copying from a colleague's laptop is slow, nobody copies from laptops. Regional co-location and a tuned jobs do more for compliance than a policy.

Measure and publish. Without numbers, "our data is under control" is an opinion:

Machine-learning specifics

Weights are data, not code. Track them, but keep them out of the same lifecycle as source. A retrain should not require a code release, and a code fix should not require re-pushing gigabytes.

Model artifacts accumulate fastest. Every experiment caches one. A 2 GB model across thirty sweeps a month is where the storage bill comes from, and dvc exp remove plus a lifecycle rule is the answer.

Register promoted models explicitly. A Git tag plus dvc.lock gives you data, code, and metrics for a release. live.log_artifact(..., type="model") makes the artifact discoverable rather than implied.

Log the environment alongside the metrics. Git SHA, image digest, and the input data hash written into the run's parameters. Without those three, a reproducible pipeline is still an unreproducible model.

Seed everything and say so. Python, NumPy, the framework, any shuffling, and the split. An unseeded run is an unreproducible run no matter how well the data is versioned.

live.log_params({
    "git_sha": git_sha(),
    "image_digest": os.environ.get("IMAGE_DIGEST", "unknown"),
    "data_hash": lock_hash("data/train.parquet"),
    "seed": SEED,
})

The checklist to run before shipping

Most data incidents are prevented at review time rather than at recovery time. Reading a DVC project for what it guarantees, rather than what it does, is the highest-leverage habit you can build.