opendatacubepipelines / tesp Goto Github PK

The outcome from the meeting was to ZIP the contents of the ARD package into a single ZIP archive.

This will reduce the inode count significantly, and prevent hitting the storage quotas at NCI.

Update the seach_s2 tool to be sensor independent. Most of it should be straight forward, but a unique list should still be returned of the L1 datasets, irrespective of a zip archive, tarball, or directory.

Currently the workflow will attempt to package all 3 surface reflectance products. Ideally this should be configurable.

Currently if --archive-list is to work --index-datacube-env needs to be specified.
But specifying --index-datacube-env is used to give the location of the datacube-env file and flag that ARD produced should be indexed.

The above logic means that you can't do a run that archives and doesn't index ARD produced. This is an unintended constraint and should be fixed.

We need something like:

        brdf:
            dataset_ids: [...]
            model: definitive # one case to rule all ???
            alpha_1:
                band_1: <value>
                band_n: <value>
            alpha_2:
                band_1: <value>
                band_n: <value>

Currently the package tasks creates 2 layers for the work directory in order to handle multiple granules delivered in one package (multi-granule).
Setting cleanup=true in the configuration file will only delete the directory for the granule and not the package, thus not fully cleaning up after the job has been completed. (The Package task currently cannot delete the package directory because the tasks in a multi-granule processing aren't aware of the completion of each other).

Issue is fairly minor, results in additional inode usage until the process can be cleaned up.

Current implementation of our lineage tracks ancillary data-sets separately to imagery files. This behaviour was carried over from previous versions of our packaging code base.

Ideally, the source ancillaries would be tracked as source data-sets; however this is currently blocked due to further enhancements required on our ancillary fetch and sync tooling (to enabling immutable files and versioning).

Update the fmask wrapper to include the landsat suite of platforms/sensors.

The RGB order of bands for the quicklook images are incorrect.

For indexing a moderate amount of scenes (500) at the end of batch job, only 4GB of memory is currently requested when submitting to the PBS scheduler.

As we request 48 CPU's (entire node), we might as well request the entire memory (192GB) for the node as well.

The outcome from the meeting regarding the ARD package, was to remove the pyramids/overviews from the SUPPLEMENTARY & QA GeoTIFF's (satellite/solar angles etc).

This is to reduce the storage footprint thus reducing the on-going costs.