Data Access Issues about presto

[gabrieltseng]

Hi ! Apologies for the delay; I somehow missed this issue.

ModuleNotFoundError: No module named 'dataops'

This is a typo in the README - the import path is now presto.dataops

ERROR: (gcloud) Invalid choice: 'storage'.

I suspect this error has to do with the update to the gcloud storage CLI from gsutil. alpha seems to no longer be required - which version of gcloud are you using? But adding alpha seems like it might work, based on the error message. Based on this, the updated command would be:

gcloud storage buckets create gs://$(python -c "from presto.dataops import EE_BUCKET; print(EE_BUCKET)") --location=us-central1 
gcloud storage buckets create gs://$(python -c "from presto.dataops import EE_BUCKET; print(NPY_BUCKET)") --location=us-central1 
gcloud storage buckets create gs://$(python -c "from presto.dataops import EE_BUCKET; print(TAR_BUCKET)") --location=us-central1 

or, with the alpha:

gcloud alpha storage buckets create gs://$(python -c "from presto.dataops import EE_BUCKET; print(EE_BUCKET)") --location=us-central1 
gcloud alpha storage buckets create gs://$(python -c "from presto.dataops import EE_BUCKET; print(NPY_BUCKET)") --location=us-central1 
gcloud alpha storage buckets create gs://$(python -c "from presto.dataops import EE_BUCKET; print(TAR_BUCKET)") --location=us-central1 

But I haven't tested this yet. Once I do I will update the README.

However all this step does it make the buckets; another option is to just manually make them in the google cloud console - their names are :

• EE_BUCKET = "lem-earthengine2"
• NPY_BUCKET = "lem-npy2"
• TAR_BUCKET = "lem-assets2"

I hope this helps, and apologies again for the delay.

[mburges-cvl]

Thanks for the answer, but I have another question, what exactly are contained in:

presto/data/dynamic_world_samples_active_shards.geojson

and

presto/data/dynamic_world_samples.geojson

?

Are these the individual files for training (once all active shards, that can be downloaded and once all shards from the original dw dataset)? So could I hypothetically just download these individual files?

Thanks!

[gabrieltseng]

presto/data/dynamic_world_samples.geojson provides all the locations used in the exports.

Yes - you can use this file to just download all the data from earthengine.

presto/data/dynamic_world_samples_active_shards.geojson describes which files we have actually exported and used for training (so can be deleted if you are doing a fresh re-export). A new file with your active shards would be written during the export.