liambindle / bashdatacatalog

A simple portable file cataloging tool for bash

bashdatacatalog

The bashdatacatalog is a command-line tool that facilitates synchronizing local data collections with a remote data source. With the bashdatacatalog, you can run queries on your local data collections to answer questions like "What files am I missing?" or "What files aren't bitwise identical to remote data?". Queries can include a date range, in which case collections with temporal assets are filtered-out accordingly. The bashdatacatalog can format the results of queries as: a URL download list, a Globus transfer list, an rsync transfer list, or simply a file list.

The bashdatacatalog was written to facilitate downloading input data for users of the GEOS-Chem atmospheric chemistry model. The canonical GEOS-Chem input data repository has >1 M files and >100 TB of data, and the input data required for a simulation depends on the model version and simulation parameters such as start and end date.

Note: Consider giving the bashdatacatalog a Star ⭐ if you find it useful. This increase visibility and helps justify maintaining this repository.

📋 Key Terms

Data is organized with collections and catalogs.

collection - A data collection is a directory (folder) that has data files. A data collection may have any number of files, any types of files, and have subdirectories.

catalog - A file that lists collections and collection settings. A catalog file includes (1) the local paths to data collections, (2) the URLs of the data sources, and (3) boolean flags to enable/disable data collections.

💿 Install

You can install the bashdatacatalog with the following command. Follow the prompts and restart your terminal.

$ bash <(curl -s https://raw.githubusercontent.com/LiamBindle/bashdatacatalog/main/install.sh)

Note: This command upgrades the bashdatacatalog if it's already installed.

🎉 Example Usage

1. Download a catalog file:

   $ curl https://raw.githubusercontent.com/LiamBindle/bashdatacatalog/main/sandbox/catalog1.csv -o catalog1.csv

2. Fetch collection metadata:

   $ bashdatacatalog-fetch catalog1.csv

3. Run listing queries (e.g., download all missing files with 4 parallel download streams):

   $ bashdatacatalog-list -am -f xargs-curl catalog1.csv | xargs -P 4 curl

💡 Documentation

See the Wiki for documentation on using the bashdatacatalog: https://github.com/LiamBindle/bashdatacatalog/wiki.

If you are a GEOS-Chem user, see the standalone instructions for GEOS-Chem users here.

For options and arguments to the bashdatacatalog-list command, see

$ bashdatacatalog-list -h

bashdatacatalog's Issues

md5sum behaviour with older coreutils

Hi Liam!

Just tried running the bashdatacatalog-selftest and got some md5sum error messages:

md5sum: invalid option -- 's'
Try 'md5sum --help' for more information.`

I had a quick look and on my system this is because the coreutils that ships with the Linux distro is 8.22 (CentOS 7 - end of life in 2024 so still some life left in it yet!) and there is no --ignore-missing option to md5sum. The command errors out with a missing option, so anything that uses md5sum in bashdatacalog will fail.

I guess this is a 'my system' problem, but worth being aware of, as anybody else running with an older version of coreutils will see the same issue. I'll see how things go dropping that option from the command!

No error message for range START END (rather than START,END)

line 313: unexpected EOF while looking for matching `''

Dear Liam,
I install bashdatacatalog following the Installation Instructions(https://github.com/LiamBindle/bashdatacatalog/wiki/1.-Installation)
$ bash <(curl -s https://raw.githubusercontent.com/LiamBindle/bashdatacatalog/main/install.sh)
But when run bashdatacatalog-list, an error "line 313: unexpected EOF while looking for matching `''" occurred. Could you help me ?

Feng

Bashdatacatalog list contains unnecessary files

Problem description

Hi Liam, I am using bashdatacatalog to download missing files. It is a great tool but I do have some questions:

1. When there is existing entries for one year, it will fetch all missing files in that entry even though it is not in the time range specified. For example, I have downloaded some offline dust files for one week in 2019, and bashdatacatalog-fetch all files in the whole year 2019 although I have specified I time range is 2017-12-01,2018-12-31 using bashdatacatalog-list.

2. Also, I thought by specifying the meteorology is GEOS-FP in MeteorologicalInputs.csv , it will only fetch fine resolution such as 0.25 degree OFFLINE_DUST instead of 0.5 degree. Now it will also download 0.5 degree files

3. Another problem is it will download a lot of files for vary old years, which currently I am not sure why. For example, for the emissions of SfcVMR in HEMCO_Config.rc: --> SfcVMR : true # 1750-2014, I think it only needs files for certain years? But it will download all years' files including ./HEMCO/SfcFix/v2019-12/GMI/2x2.5/surface_VMRs_GMI_1959.2x25.nc

Sorry it may be too messy here to pass all information. To be specific, here is the example containing all the problems I have stated above. I want to get all missing files for simulations of GCHP v13.2 from 2017-12-01 to 2018-12-31 with meteorology GEOS-FP and thus only high-resolution files are needed.
By running bashdatacatalog-list -am -r 2017-12-01,2018-12-31 InputDataCatalogs/*.csv InputDataCatalogs/**/*.csv &> down-list-orig.txt , I got the file attached below which contains files for year 2019 and 1959, as well as offline emission files with resolution of 0.5 degree:
down-list-orig.txt

The csv files I used are:
EmissionsInputs.csv
MeteorologicalInputs.csv
ChemistryInputs.csv

HEMCO_Config.rc is also attached here for reference:
HEMCO_Config.rc.txt

curl --url argument sometimes not supported

See geoschem/input-data-catalogs#2