Problem description

Hi Liam, I am using bashdatacatalog to download missing files. It is a great tool but I do have some questions:

1. When there is existing entries for one year, it will fetch all missing files in that entry even though it is not in the time range specified. For example, I have downloaded some offline dust files for one week in 2019, and bashdatacatalog-fetch all files in the whole year 2019 although I have specified I time range is 2017-12-01,2018-12-31 using bashdatacatalog-list.

2. Also, I thought by specifying the meteorology is GEOS-FP in MeteorologicalInputs.csv , it will only fetch fine resolution such as 0.25 degree OFFLINE_DUST instead of 0.5 degree. Now it will also download 0.5 degree files

3. Another problem is it will download a lot of files for vary old years, which currently I am not sure why. For example, for the emissions of SfcVMR in HEMCO_Config.rc: --> SfcVMR : true # 1750-2014, I think it only needs files for certain years? But it will download all years' files including ./HEMCO/SfcFix/v2019-12/GMI/2x2.5/surface_VMRs_GMI_1959.2x25.nc

Sorry it may be too messy here to pass all information. To be specific, here is the example containing all the problems I have stated above. I want to get all missing files for simulations of GCHP v13.2 from 2017-12-01 to 2018-12-31 with meteorology GEOS-FP and thus only high-resolution files are needed.
By running bashdatacatalog-list -am -r 2017-12-01,2018-12-31 InputDataCatalogs/*.csv InputDataCatalogs/**/*.csv &> down-list-orig.txt , I got the file attached below which contains files for year 2019 and 1959, as well as offline emission files with resolution of 0.5 degree:
down-list-orig.txt

The csv files I used are:
EmissionsInputs.csv
MeteorologicalInputs.csv
ChemistryInputs.csv

HEMCO_Config.rc is also attached here for reference:
HEMCO_Config.rc.txt

Dear Liam,
I install bashdatacatalog following the Installation Instructions(https://github.com/LiamBindle/bashdatacatalog/wiki/1.-Installation)
$ bash <(curl -s https://raw.githubusercontent.com/LiamBindle/bashdatacatalog/main/install.sh)
But when run bashdatacatalog-list, an error "line 313: unexpected EOF while looking for matching `''" occurred. Could you help me ?

Feng

Hi Liam!

Just tried running the bashdatacatalog-selftest and got some md5sum error messages:

md5sum: invalid option -- 's'
Try 'md5sum --help' for more information.`

I had a quick look and on my system this is because the coreutils that ships with the Linux distro is 8.22 (CentOS 7 - end of life in 2024 so still some life left in it yet!) and there is no --ignore-missing option to md5sum. The command errors out with a missing option, so anything that uses md5sum in bashdatacalog will fail.

I guess this is a 'my system' problem, but worth being aware of, as anybody else running with an older version of coreutils will see the same issue. I'll see how things go dropping that option from the command!

See geoschem/input-data-catalogs#2