Fix point-like ``RAD_MAX`` energy binning about gamma-astro-data-formats

[jjlk]

Hi @TarekHC , @cdeil , @bkhelifi , @contrera, @jknodlseder , @cboisson ,

I'm ultra motivated to try your specifications in the FITS IRFs for point-like source analysis with CTA. I'll start playing with the IRFs in the exporter the ones exported from ROOTs files containing ON-axis IRFs)

What about a mixed solution including your comments 4 and 5. It should be easy to add a new HDU (called THETA_CUT or something like that) with keywords ENERG_LO, ENERG_HI and RAD_MAX?

In addition, it should be an optional HDU (is it allowed?) since cube-style analysis do not use it. Do you think that it is something that I could try? Any comments?

++

[TarekHC]

Hi @jjlk ,

Your ultra motivation makes me ultra happy! :)

What about a mixed solution including your comments 4 and 5. It should be easy to add a new HDU (called THETA_CUT or something like that) with keywords ENERG_LO, ENERG_HI and RAD_MAX?

Some comments regarding this solution:

• Note the energy in which you would cut would be the reconstructed energy, so the keywords should probably be ERECO_LO and ERECO_HI.
• If we add a new HDU, then we would need to remove the RAD_MAX column from the point-like IRFs.
• The only missing point here is that all point-like IRFs would need a "pointer" to those specific cuts. Perhaps, for the moment, we could impose that only one set of point-like IRFs can be included in a given fits file, all corresponding to the cuts present within this additional HDU (THETA_CUT, or however we call it).

In addition, it should be an optional HDU (is it allowed?) since cube-style analysis do not use it. Do you think that it is something that I could try? Any comments?

It would be a required HDU if any point-like IRF is present within the DL3 file. As soon as the data reader finds a HDUCLAS3 == "point-like", then it should search for the THETA_CUT HDU.

If others (@cdeil , @bkhelifi , @contrera, @jknodlseder , @cboisson) agree, we could implement this solution and see if it works for the moment. Of course, I would be in favor of applying this modification to cover a very important use case for CTA, as the current implementation is not really usable... So let me know if I can be of any help.

[cdeil]

Here's the current spec: http://gamma-astro-data-formats.readthedocs.io/en/latest/irfs/point_like/index.html

My understanding is that no-one is currently using it. The HESS & MAGIC files I've seen have a RAD_MAX header key, which isn't mentioned in the spec anymore. I think this should be reverted and allowed as a valid option again.

For the energy-dependent offset cut, I think maybe a separate HDU would be best. My reasoning is that this info is only loaded to process and select EVENTS, and event processing for full-enclosure is also decoupled from IRF processing by science tools (e.g. AEFF is loaded for exposure computation). Also it means that really full-enclosure and point-like IRFs are even more similar. It's not a strong argument, of course you can organise the same information and different cases in many different ways, but I think that would be my preference.

Concerning the question how to link to the extra HDU: that is just a general unsolved problem we have discussed many times. We could just leave that be for now. Note that we also don't have something in the spec to point e.g. from EVENTS to other HDUs or from AEFF to EDISP or whatever. In Gammapy I would be OK to have some logic for now that tries to do the right thing by default, i.e. look at the header keys and look in the same file, and probably we would also in the future add the new HDU to the HDU index table, just like we have for all other HDUs.

Maybe @TarekHC you could bring the spec in order and @jjlk would implement the corresponding CTA ROOT -> FITS exporter (see WIP at gammapy/gammapy-extra#83)?
Note that in the CTA ROOT files the energy-dependent offset is a separate histogram, so doing the same in FITS should be easy to export.

I think that's really needed before we can ask CTA to start producing these files (because it's more work to document / check them and explain what is what to users, producing them it's a ton of work). As you know, there is hesitation to add this for CTA, so if we can get the format and exporters right from the start it really helps.

[jjlk]

Hi @TarekHC , @cdeil ,

Maybe @TarekHC you could bring the spec in order and @jjlk would implement the corresponding CTA ROOT -> FITS exporter (see WIP at gammapy/gammapy-extra#83)?

It sounds like a plan. I can help with the spec if needed. @TarekHC , you tell me.

I think that's really needed before we can ask CTA to start producing these files (because it's more work to document / check them and explain what is what to users, producing them it's a ton of work). As you know, there is hesitation to add this for CTA, so if we can get the format and exporters right from the start it really helps.

I totally agree.

Question, for now the simulation and analysis working group is also providing a background rate as a function of reconstructed enregy. Should we have that in the spec (there are specs for '2D' and '3D' background)? I think it's important since we are using this a lot for simulated high level data products. What do you think?

Small comment:

Note the energy in which you would cut would be the reconstructed energy, so the keywords should probably be ERECO_LO and ERECO_HI.

It seems that the standards does not cover differences between true and reconstructed energy: http://gamma-astro-data-formats.readthedocs.io/en/latest/irfs/irf_axes/index.html

Thanks! ++

[cdeil]

For background, I think what we have is exactly what we need:
http://gamma-astro-data-formats.readthedocs.io/en/latest/irfs/full_enclosure/bkg/index.html
It already states reco energy axis.

So probably for point-like it would always be the same: a header key declaring it's point-like (what we already have now), and either a RAD_MAX header key (what we have now e.g. from HESS) or a table with the RAD_MAX info if it's energy-dependent.

Otherwise the formats are identical for full-enclosure and point-like, which is nice.

[TarekHC]

Ok. I think this discussion is easier with a pull request in hand. I'll try to have one ready as soon as possible, so Julien can start work on this subject right away.

For background, I think what we have is exactly what we need:
http://gamma-astro-data-formats.readthedocs.io/en/latest/irfs/full_enclosure/bkg/index.html
It already states reco energy axis.

Uff, I didn't remember this. I hate the fact that different quantities have identical column name... But ok... :)

[TarekHC]

Hi @cdeil and @jjlk ,

I created (relatively quickly, still needs some work) this pull request: #99

I took the liberty of calling it "RAD_MAX", but please change it if you think it is not good enough.

Tomorrow I can continue working on it, implementing your suggestions.

[TarekHC]

In principle I close this issue, which should be solved with #99.

If @jjlk sees it is still not good enough, then please re-open.

[TarekHC]

Hi @jjlk,

Quick question: did you manage to implement this on gammaPy? I mean, is it now allowed within a 1D analysis a variable theta2 cut vs E (or offset)?

Thanks!