How does one add individual files with DVC?

Viewed 112

Suppose I run the following commands:

# set up DVC

mkdir foo
cd foo && git init
dvc init
git add * && git commit -m "dvc init"


# make a data file

mkdir -p bar/biz
touch bar/biz/boz


# add the data file

dvc add bar/biz/boz

And DVC outputs the following:

To track the changes with git, run:

  git add bar/biz/.gitignore bar/biz/boz.dvc

This last part is what I would like to avoid. Preferably, DVC would only change the top level .gitignore (located at the project root, where git init was executed), and will change only DVC files at the top level.

And here's why:

I have a rather large dataset developed in an original work more or less ad-hoc. This data is not systematically organized, nor do I want to organize it as-is.

Instead, I want to incrementally add this old, bespoke data to the DVC directory tree. And each time I add some of the data to the tree, I want to check it in with DVC as I would if I were modifying code or mixing one project's code into another.

However, DVC wants to create a local file and gitignore at every location I add. This creates a mess and I have no reasonable faith that it will be easy to maintain all of these atomic and distributed datastores.


The question:

What is the preferred way to incrementally add data in DVC so that DVC uses the root gitignore and root DVC files/items?

1 Answers

Assuming bar/ is the dataset directory you're incrementally adding to, you can instead

dvc add bar

This creates a bar.dvc file and writes to .gitignore at the top level.

When you update content in bar/, dvc add it again or use dvc commit to register the new dataset version. The new files get added to the project cache and the .dvc file gets an updated md5 hash that identifies to the latest directory structure.

Some docs:
https://dvc.org/doc/start/data-management#making-changes
https://dvc.org/doc/command-reference/add
https://dvc.org/doc/user-guide/project-structure/internal-files#structure-of-the-cache-directory

Related