pip install dvc
pip install numpy
pip install pyyaml
./pipeline/get_data.py # generates random numbers and creates a new file
./pipeline/process_data.py # computes some parameter dependant statistics on the data, saves metrics
./environment.yml # conda environment
./params.yaml # specifies parameters for the pipeline
After running the DVC code, we will have the following:
./metrics/ # contains output metrics
./dvc.yaml # defines pipeline
./dvc.lock # specifies current DVC state
After running the DVC code, we will have the following:
./data/
./.dvc
dvc init
dvc repro
git add dvc.lock
git commit -m "ran dvc"
execute dvc repro
again... note that everything is skipped! no parameters or dependancies changed.
After the above has been executed...
dvc exp run -S 'process_data.test_size=0.5'
dvc exp show