Master index file about gamma-astro-data-formats

[cdeil]

I don't have time for the whole data handling stuff this week, sorry.
HGPS EdB meeting Wednesday, HAP freeze on Friday, both still require a lot of work.

You could just use a FITS table for now and then we change to JSON later on.
Changing to a different table format should just be a one- or few-line change in your script.

[mimayer]

Alright, so I guess the FITS table handling will then go into ctools-1.0 if this is fine with you.

[mimayer]

I looked a bit into json and here is a proposal how the file could look like:

{      
    "hess-fits-prod1": {
    "hduindex": "hduindex.fits.gz",
    "obsindex": "obsindex.fits.gz"
     },
    
    "hess-fits-prod2": {
    "hduindex": "hduindex2.fits.gz",
    "obsindex": "obsindex2.fits.gz"
     }
}

If you have time, please let me know if you have any feedback. This can be implemented rather quickly then.

[cdeil]

It would be better to have a top-level key so that it's possible to add other info to the file,
and to use a list. So basically what we have now in Gammapy, but with different and extra keys (similar to what you proposed as columns in proposal B):
https://gammapy.readthedocs.org/en/latest/data/dm.html#data-registry-config-file

JSON example:

>>> config = dict()
>>> config['datasets'] = list()
>>> dataset = dict(name='spam', base_dir='ham', obs_index='sausage')
>>> config['datasets'].append(dataset)
>>> config['datasets'].append(dataset)
>>> import json
>>> print(json.dumps(config, indent=4))
{
    "datasets": [
        {
            "obs_index": "sausage",
            "name": "spam",
            "base_dir": "ham"
        },
        {
            "obs_index": "sausage",
            "name": "spam",
            "base_dir": "ham"
        }
    ]
}

[mimayer]

Thanks for the input!
So following proposal B, we would now have:

{
    "datasets": [
        {
            "name": "hess-fits-prod1",
            "obsindx": "relative/path/to/prod1/obs-index.fits.gz",
            "hduindx": "relative/path/to/prod1/hdu-index.fits.gz"
        },
        {
           "name": "hess-fits-prod2",
           "obsindx": "relative/path/to/prod2/obs-index.fits.gz",
           "hduindx": "relative/path/to/prod2/hdu-index.fits.gz"
        }
    ]
}

Is this acceptable? I would then edit the spec to json and implement this in ctools.

[cdeil]

@mimayer - As I said before, I don't want to agree to a high-level index format now. There's several open questions, e.g. where to put base_dir, whether paths should be relative to index files or absolute, whether the format on the server and user's machine is the same or different and whether / how we store info about remotes and the state of synchronisation to the user's machine. It's not possible for me to invent a good format here without prototyping it, which I can start in Gammapy in January, because I don't have time for this now.

I think this was also the conclusion of the discussion in the last telcon, that it'll take some time to hash out the high-level index scheme and data handling on the server / user machine works.

My strong preference would be that you document the high-level index format you're using in your ctool for now in the ctools docs. I can agree to provide it on the HESS server also after we come up with something better / more extensive in January.

It's not clear to me yet if this will remain tool-specific long-term or if we will agree on global data indexing schemes on the server and / or user's machine to be shared by ctools and Gammapy.

OK?

[mimayer]

@cdeil. I agree there is no need for an extensive discussion about the format now until we know exactly what we want. Nevertheless, we basically decided e.g. on the BASE_DIR content here: #8 (comment 2...)

My strong preference would be that you document the high-level index format you're using in your ctool for now in the ctools docs. I can agree to provide it on the HESS server also after we come up with something better / more extensive in January.

Thanks! I guess my main question was to ensure that you agree to provide and maintain such a file on the server in HD. We agreed that JSON is simplest since human-editable (i.e. you can simply add a new production by hand). I will edit the specs with the updated format.

So bottomline to answer your question: OK!

[mimayer]

I have updated the prototype description to the master index file to JSON: http://gamma-astro-data-formats.readthedocs.org/en/latest/data_storage/super_index/index.html

I am about to implement this in ctools and also write docs there.

[cdeil]

Closing this old issue.
This master index table was recently removed in #109, for discussion see #7 .