Event list spec required columns about gamma-astro-data-formats

[mimayer]

@cdeil:
I completely agree. I was drafting the spec from the ctool perspective where things like CORE or XMAX are required to read the events. Of course this needs improvement on the ctools side.
I made an update - what do you think?

I am not sure if the changes can be implemented in ctools quickly. If not, can we agree on exporting optional columns like "CORE" and "XMAX" for the first version?

[cdeil]

Better, but I don't see why OBS_ID, MULTIP, DETX, DETY should be required, I think they should be optional. BUNCH_ID should definitely be optional, I don't even know if CTA will have bunches, and certainly it's a detail that shouldn't be present at the DL3 level.

Concerning ctools, well, the first step is to make a change request or drop an email to the ctools mailing list saying that we think something should be changed and in what way and why. @mimayer Can you do that to get the conversation started?

[kosack]

+1 for having BUNCH_ID be optional (it's a technical detail of the HESS DAQ implementation, and is not general)

[mimayer]

Fine with me: OBS_ID, MULTIP, BUNCH_ID should be optional.
DETX and DETY should be required as these values are important to create background models and other instrument specific stuff, right?

[mimayer]

I made the change

[cdeil]

DETX and DETY should be required as these values are important to create background models and other instrument specific stuff, right?

I think they should be optional. They can be computed from RA / DEC.

@mimayer - Let me know if you disagree about DETX and DETY ... if so we'll move the discussion to a separate issue or the ctools or CTA mailing list. Can you run a ctools analysis with an event list that doesn't have DETX / DETY at the moment? I think last time I checked ctobssim didn't even fill it, so if it needs to be present at the moment, it can't be hard to change to make it optional?

[mimayer]

I think they should be optional. They can be computed from RA / DEC.

This depends on the definition of DETX, DETY. Some kind of instrument coordinates are always needed I would say.

Can you run a ctools analysis with an event list that doesn't have DETX / DETY at the moment? I think last time I checked ctobssim didn't even fill it, so if it needs to be present at the moment, it can't be hard to change to make it optional?

I think in principle you can - not sure though. ctobssim fills DETX and DETY, was implemented a year ago.

[mimayer]

What's the harm of having DETX and DETY required?

[cdeil]

What's the harm of having DETX and DETY required?

Storage space ... data size is at the moment dominated by the event list, and adding DETX / DETY means an increase by ~ 40%.

Doesn't sound like you agree to make it optional yet.

I'll write up http://gamma-astro-data-formats.readthedocs.org/en/latest/info/coordinates.html#field-of-view and then file a separate issue to see if you agree with the definition and we can discuss if / why it's OK to have it optional there.

[mimayer]

Just note that without that info no background model can be created as you need to stack DETX and DETY in instrument coordinates (not-aligned with Ra-DEC).
All in all, ctools analysis would suffer a lot if that info was removed (just my opinion).

But if you insist in removing them - fine.
You probably won't export them in HAP at all then? Just asking because then background modelling using this chain will then lose precision for sure.

[cdeil]

Software that builds background models will need to be able to transform RA/DEC to DETX/DETY anyways (to exclude bright gamma-ray sources and maybe stars), so just having RA / DEC is enough, no?

For HAP I think we'll keep FOV coordinates and a lot of the other optional columns (like Hillas parameters, core location, ...) for now for testing. Maybe we'll provide a second slim version that has just the parameters needed for high-level analysis, that would then be smaller by a factor of a few (factor 3?) to download.

[mimayer]

Software that builds background models will need to be able to transform RA/DEC to DETX/DETY anyways (to exclude bright gamma-ray sources and maybe stars), so just having RA / DEC is enough, no?

Well no, there is need to transform externally between Ra/Dec and DETX/DETY if for every event the coordinates are given. You can compute rotation angles for each event time stamp (also interpolate), without the need for telescope location, etc. Therefore DETX/DETY is required and necessary to extract this information.

For HAP I think we'll keep FOV coordinates and a lot of the other optional columns (like Hillas parameters, core location, ...) for now for testing. Maybe we'll provide a second slim version that has just the parameters needed for high-level analysis, that would then be smaller by a factor of a few (factor 3?) to download.

Ok no need to discuss this in extreme detail now. I don't know how Manuel implemented the background model creation in gammapy but I guess without DETX/DETY your script doesn't work, right? If it does work, the stacking in instrument coordinates is simply wrong (except if you recompute DETX and DETY from GEOLON, GEOLAT, RA_PNT, DEC_NT, RA, DEC and TIME which seems a bit overkill, no?)

Just make it optional. I will include them in PA event lists and will compute background models with it though.

[cdeil]

OK, I've made DETX, DETY optional in 7c7c76c (and added THETA as optional column).

There's further work to do on this spec ... I might get to it on Saturday, otherwise next week.

[joleroi]

Would it be possible to add HIL_MISS to the optional event list columns?
I need it for a study

[mimayer]

What's HIL_MISS?

[joleroi]

The the offset between the target position and the reconstructed source position (perpendicular to the Image main axis)

Thinking about it now, I guess we create to much overhead by exporting more and more Hillas parameters. In this case, I would support what @cdeil said before: No low level parameters in the event list. Maybe it's not flexible enough.

[mimayer]

Yes indeed, I guess there will always be a parameter that might be important for a certain study. The approach of @cdeil to export event list with large detail and then slim it down to relevant parameters for distribution might be the right thing to do.

[cdeil]

Here's what I think about optional header keywords / columns in general (applies to the Hillas parameters here, but also to many other optional things):

• It's useful to document them. They're optional, OK to ignore for data producers that don't want to put that info. It's helpful to have an agreed definition and name for people that want to work with that info (e.g. compare Hillas parameters between HAP and PA).
• It's bad to include too many optional things that will only be used by a few people. It's extra work to write and read the spec, and e.g. for the Hillas columns, the real work hasn't been done, i.e. give the details and formulas or link to a reference paper so that it's clear what the definition is ... different people do use different definitions for Hillas parameters, so it's not very useful as-is.

In the end, which optional info to include is a subjective choice of the people taking the time to write the spec. In this case, I'm +1 to remove the Hillas parameters, because I doubt someone will take the time to write down definitions.

And @joleroi - The HAP FITS exporter is not very flexible when it comes to adding columns for quick checks. It's easier to add columns in the ROOT event list exporter and then it's possible to read those as a Numpy array with http://www.rootpy.org/ if you want to do your studies with Python (and write to FITS if you want to use other FITS tools).

[joleroi]

+1 for not exporting Hillas stuff :)

[mimayer]

When going through the required keywords of the event list, I realised we need a better name for "TSTART_STR" and "TSTOP_STR". The reason is that header keywords should not be longer than 8 characters. I therefore propose to change to "TSTART_S" and "TSTOP_S".

[cdeil]

With #60 this is issue is now adressed.

As you can see here, the only required columns now are: EVENT_ID, TIME, RA, DEC, ENERGY