gammapy / enrico

Enrico helps you with your Fermi data analysis

Gammapy

A Python Package for Gamma-ray Astronomy.

Gammapy is an open-source Python package for gamma-ray astronomy built on Numpy, Scipy and Astropy. It is used as core library for the Science Analysis tools of the Cherenkov Telescope Array (CTA), recommended by the H.E.S.S. collaboration to be used for Science publications, and is already widely used in the analysis of existing gamma-ray instruments, such as MAGIC, VERITAS and HAWC.

• Webpage: https://gammapy.org
• Documentation: https://docs.gammapy.org/
• Code: https://github.com/gammapy/gammapy

Contributing Code, Documentation, or Feedback

The Gammapy Project is made both by and for its users, so we welcome and encourage contributions of many kinds. Our goal is to keep this a positive, inclusive, successful, and growing community by abiding with the Gammapy Community Code of Conduct.

The Gammapy project uses a mechanism known as a Developer Certificate of Origin (DCO). The DCO is a binding statement that asserts that you are the creator of your contribution, and that you wish to allow Gammapy to use your work to cite you as contributor. More detailed information on contributing to the project or submitting feedback can be found on the Contributing page.

Licence

Gammapy is licensed under a 3-clause BSD style license - see the LICENSE.rst file.

Supporting the project

The Gammapy project is not sponsored and the development is made by the staff of the institutes supporting the project over their research time. Any contribution is then encouraged, as punctual or regular contributor.

Status shields

(mostly useful for developers)

• 
• 
• 

enrico's Issues

Software Version in outputs

Dear all,

Thanks for this very useful package.

I noticed that no information concerning Sciencetools version, Enrico version (if there are any), Data processing version, etc... are shown in outputs.

I think such information may be useful to check and compare old analyses.

Cheers,
Cyril.

gtbin run failed please help

I have this error message

Invalid parameter name in command line assignment: "hpx_ordering_scheme"
Invalid parameter name in command line assignment: "hpx_ordering_scheme"
Caught N5hoops12ApeExceptionE at the top level: Cannot open parameter file for gtbin; Ape exception code 22 (at hoops_ape.cxx: 435)
/
/
File "/home/mmohamed/anaconda/lib/python2.7/GtApp.py", line 72, in run
    raise RuntimeError, self.appName + " execution failed"
RuntimeError: gtbin execution failed

Update 2FGL -> 3FGL in Enrico (or support both)

Hi,
Any idea regarding the time of release of 3 FGL catalog? Actually now a days referees have started asking to include unmodeled sources by generating TS maps. For one or two sources, this can be done, however, if someone wants to use a large sample (say 100 blazars), then generating TS map of 100 sources will be too time and memory consuming. So is there any alternate of searching unmodeled sources from TS maps or we need to wait till 3 FGL comes up?
Thanks.

Write enrico_extended

Make a tool to fit the extension of Gaussian and disk models.
For each test extension gtlike (or BinnedAnalysis, UnbinnedAnalysis) can be run.

There are many ways to implement this, let's discuss what is most robust / useful.

Auto-generate enrico tools summary page

I'll write a script to auto-generate this page
https://enrico.readthedocs.org/en/latest/tools.html
and include the help output of each tool as a sub-page.

Support the 1FHL

download the 1FHL catalogue

allow the creation of XML model from it

qstat and qsub command not found

Dear David SANCHEZ,

First of all, thank you for your code it is very nice.
However I have some difficulties to use it.
When I run enrico the commands enrico_config and enrico_xml works fine
but when I run enrico_sed (or enrico_lc) it gives error that <</bin/sh: qstat: command not found>> and<<sh: qsub: command not found>> .

Could you please help me to solve this problem ?

Thank you very much,
Best regards,
Narek

useless config entries

Are the following entries in a conf file useful :

[space]
srcpix = 120
nlong = 120.0
nlat = 120.0

[tool]
chatter = 2
clobber = no
debug = no
gui = no
mode = ql

Create an Enrico logo

We should create an Enrico logo that can be used for the docs pages and presentations.

Everyone ... feel free to give it a try and attach an image to this issue (you can copy & paste images into the comment field on github) and then we decide in two weeks before the 0.2 release.

Write enrico_ts

enrico_ts should make TS, flux, spectral index etc maps by running BinnedAnalysis or UnbinnedAnalysis for each pixel.
This will be done by:

• converting the image to a list of pixels
• processing bunches of nearby pixels in parallel on the cluster
• reassemble the TS etc for each pixel into images.

Calculation of de-correlation energy and search of highest energy photon

Dear Sir,
Would you please elaborate that how enrico package calculates de-correlation energy for a source of interest?

Further, as a small request, is it possible to include a tool to calculate energy of highest energy photon detected from a source? I found that ASDC server (http://tools.asdc.asi.it/?&searchtype=fermi) provides a GUI version where we need to enter source's basic information (RA, DEC etc.) and it returns the highest energy photon's energy. Inclusion of such a tool in the enrico package will be highly useful while analyzing the data of bulk of sources.
Thanks.

-- Vaidehi

Use argparse

I've included argparse, we should use it throughout instead of the deprecated optparse we are using now.

Remove hardcoded constants scattered throughout the code

There are many constants, like emax = 3e5 or filenames hardcoded throughout the code.
This eventually leads to inconsistent results in some cases where people do want to change these constants e.g. in their config file.

We should at least move all constants to constants.py to simplify maintenance.

Time selection : SED and LC

I assign this to me : finish what Victor wanted to implement. see issues 37 and 40

Undefined variable errors

The static code analysis from my editor (Eclipse PyDev) claims the following errors for Enrico:

Description Resource    Path    Location    Type
Undefined variable: FitRunnerback   RunGTlike.py    /enrico/enrico  line 73 PyDev Problem
Undefined variable: UpperLimits fitmaker.py /enrico/enrico  line 273    PyDev Problem
Undefined variable: FitRunnerfront  RunGTlike.py    /enrico/enrico  line 74 PyDev Problem

I think these are real errors.
@davidsanchez Can you have a look? Or even run an analysis that uses these lines of code to test?

useless config entries

Are the following entries in a conf file useful :

[space]
srcpix = 120
nlong = 120.0
nlat = 120.0

[tool]
chatter = 2
clobber = no
debug = no
gui = no
mode = ql

PEP 8, Indentation

We can use e.g. reindent.py or some other tool to get proper code formatting (python PEP 8 standard)

There are also rules for naming, e.g. files should be lower-case, classes CamelBack notation, functions like_this().

Repetition of Model name in fitmaker.py

Hi,

I found that the PLSuperExpCutoff model to be repeated in the file, fitmaker.py. This occurs along the lines, 101-106. Should one of them be a logParabola or some other model, perhaps?

Thank you,
Venkatessh

MWL Fit

Make an example of how to do a combined fit on MWL data (chi2 Fit to HESS and Fermi data for now)

Add options for parallel execution on multicore machines

See discussion here:
https://groups.google.com/d/topic/gammapy_enrico/7ezMHqYsnEI/discussion

It should be pretty simple to execute the science tools in parallel for multiple energy or time bins and lead to almost N-time speedup on an N-core machine by calling bash instead of qsub.

Document batch submission

Currently the docs don't mention how batch submission works at as far as I can see ... @jlenain or @davidsanchez if you could add a paragraph somewhere that would be great.

Allows submission for AppLC

I'am working on it

Add a gui for changing the conf file

This is just an idea but might be great. this is of course a long term project.

enrico_setupcheck improvements

The output should be made more clear, i.e. give the user instructions which environment variables to set or commands to execute to get a fully-working setup.

Also the status of the downloaded and preprocessed data should be summarized.

Make 0.3 stable release

We should make a v0.3 stable release of Enrico (so that people can mention this for publications):

• Close all issues under the 0.3 milestone.
• Review the docs and do some manual testing that things work
• Create v0.3 tag in the git repo
• Update the tarball to PyPI: https://pypi.python.org/pypi/enrico/
• Send announcement to the enrico-users mailing list
• Ask for Enrico v0.3 to be listed here: http://fermi.gsfc.nasa.gov/ssc/data/analysis/user/

Tool for 2FGL queries and plots

We should add a command line tool to query the 2FGL and plot lightcurve / spectra with flux points / upper limits / butterflies.

Similar to what TeVCat provides, but publication quality png and pdf plots:
http://tevcat.uchicago.edu/LATCat/spectra/lat_0009.html
http://tevcat.uchicago.edu/searchLATCat.cgi?mode=2&ra=00%2006%2036%20&dec=+72%2047%2000

Additional EBin Settings Requested

It would be useful to have the Ebin computations automatically freeze everything in the model other than the normalization of the source. Currently, the full model is refit in each energy bin and nearby sources are also allowed to vary differently in each energy bin. While this may be desirable sometimes, often we want to freeze everything except the normalization for each bin.

ImportError: cannot import name MEV_TO_ERG

Hi,

After <> I get this error
<<Traceback (most recent call last):
File "/opt/enrico/bin/enrico_lc", line 4, in
import enrico.lightcurve as lc
File "/opt/enrico/enrico/lightcurve.py", line 7, in
from enrico import plotting
File "/opt/enrico/enrico/plotting.py", line 7, in
from enrico.constants import MEV_TO_ERG, ERG_TO_MEV
ImportError: cannot import name MEV_TO_ERG >>

when I use enrico_sed or enrico_lc.

should I fix something else ?
Thank you !

enrico_survey

is enrico_survey up-to-date?

Should enrico_weekly and enrico_survey use config files?

I guess we should use config files for these parts also, but not for the things that are set via environment variables (such as location of weekly, preprocessed and auxiliary data files).

The reason I haven't for now with enrico_weekly and enrico_survey is that I couldn't figure out how to combine environment variables with a default config file.

It should also be possible to submit different sets of parameters to the cluster.

Documentation needs complete rewrite

The documentation is completely outdated, doesn't correspond to the tools any more.

We should find ways to automatically generate the help output of the scripts/enrico_* and include this (as well as default config files) in the Sphinx docu.
Options are still changing a lot and any hand-written documentation would become outdated quickly.

Write enrico_summarize

Write a tool that summarizes analysis results.
Probably best done with an rst template, from which html and pdf can easily be generated.

add doc for the scripts

the scripts don't have a doc yet

New tool: enrico_getdata?

We should make a tool that gets the data for the ROI, energy range, event class the user wants from one of these locations:

• FSSC (automate http requests so that the user doesn't have to fill out the web form and download files by hand)
• Weekly data
• Preprocessed data

Making this a separate tool from enrico_like gives more flexibility for the user how to get started with the analysis, i.e. where to get the input photon and spacecraft files.

problem with flux points

Hi,

When I use enrico_sed .conf it works for all steps but as soon as it tries to calculate the flux points it stops and gives error. This is the error

# *** Preparing submission of fit into energy bins ***
# ************************************************************
('Emin = ', 100.0, ' Emax = ', 300000.0, ' Nbins = ', 5)
[Message]: Submition # 0 at energy 222
Traceback (most recent call last):
  File "/opt/enrico/bin/enrico_sed", line 58, in <module>
    sed(config,infile)
  File "/opt/enrico/bin/enrico_sed", line 13, in sed
    run(infile)
  File "/opt/enrico/enrico/RunGTlike.py", line 74, in run
    energybin.RunEbin(folder,Nbin,Fit,FitRunner)
  File "/opt/enrico/enrico/energybin.py", line 116, in RunEbin
    configfiles = PrepareEbin(Fit, FitRunner)
  File "/opt/enrico/enrico/energybin.py", line 97, in PrepareEbin
    NewFitObject = ChangeModel(Fit, ener[ibin], ener[ibin + 1], srcname, Pref[ibin] ,Gamma[ibin])
  File "/opt/enrico/enrico/energybin.py", line 31, in ChangeModel
    Fit.logLike.getSource(name).getSrcFuncs()['Spectrum'].getParam('Index').setBounds(Gamma_min,Gamma_max)
  File "/opt/Fermi/x86_64-unknown-linux-gnu-libc2.5/lib/python/pyLikelihood.py", line 1595, in setBounds
    return lib_pyLikelihood.Parameter_setBounds(*args)
RuntimeError: Attempt to set bounds that exclude the existing value.

Can you please help me to solve this problem.
Thank you in advance.
Best regards

Name conflict with more than one analysis in same directory

When doing two or more analysis (with same source but different [file]/tag) in the same directory, most of the resulting files are differentiated by the tag in the filename, but some are not and therefore conflict between analysis. A few I have found (probably to exhaustive):

• Spectrum dir
• TSmap (some files whithin have tag, most don't)
• Lightcurve_Nbins (for folded lightcurves, the N bins can be repeated for different time periods)
• Subtract_Model_cmap and Residual_Model_cmap

These file/dir names should be changed to include the tag.

Wrong defintion for PLSuperExpCutoff

The variable Index2 in the model PLSuperExpCutoff should be positive, as it reflects the exponent in the arguments of the exponential (see explanation of model and example XML model).

I will submit a pull-request fixing this in a few minutes.

Change the few scripts to normal Enrico tools

Currently we have three "scripts" (see here and here) and a lot of "tools" (see here and here).

I think we should change the "scripts" to "tools", i.e. put them into the bin folder and have their name start with enrico_. I realise they are a little bit different because they have extra dependencies, but I don't think that's a problem ... we'll simply print "you need to install X to run this tool" when it's not available.

The advantage of making the change is that it's a bit easier for the user (enrico and tab-complete shows all tools / scripts available and no need to wonder what the distinction is when reading the docs).

@davidsanchez OK if I make this change?

improve enrico_conf

some questions are missing

• binned or unbinned chain

other?

Error while installing Enrico package

Dear Sir,
I am trying to install enrico package on fedora 17 32bit OS. I followed the instructions given on webpage (http://enrico.readthedocs.org/en/latest/setup.html#setup) and started with installation of 'distribute'. I faced an error showing something about 'urllib2.py', which I could remove when I changed the 'urllib2.py' file inside $FERMI_DIR by 'urllib2.py' file from /usr/lib/ (i.e. from system's python). Further, when I ran next command "python get-pip.py", I got the following error

Traceback (most recent call last):
File "get-pip.py", line 7269, in
do_exec(entry, locals())
File "", line 1, in do_exec
File "", line 9, in
File "/tmp/unpacker-iAXDPU-scratchdir/pip/init.py", line 10, in
File "/tmp/unpacker-iAXDPU-scratchdir/pip/util.py", line 17, in
File "/tmp/unpacker-iAXDPU-scratchdir/pip/vendor/distlib/version.py", line 13, in
File "/tmp/unpacker-iAXDPU-scratchdir/pip/vendor/distlib/compat.py", line 31, in
class RemapModule(module):
ImportError: cannot import name HTTPSHandler

Unfortunately, this error I am not able to handle.

Interestingly, when I ran the same command outside FERMI environment, it ran successfully. I checked and found that the python version inside Fermi package is 2.7.2, while the python by default installed in the fedora 17 is 2.7.3. If necessary, how can I update the python version inside Fermi package?
Please help.

Thank you very much.

-- Vaidehi

Large memory leak in enrico_lc

Has anybody traced this leak?. I cannot confirm that it's really a leak, but every time I run enrico_lc the memory consumption increases without limit until the OS kills the process (this in a machine with 16gb RAM + 8gb SWAP).

The problem seems to be in the RunGTlike.py function or any of its dependencies. I tried to force the garbage collector inside the _MakeLC method for each iteration without success.

In the meantime I will try to run that part externally (using os.system), but this is more a workaround than a solution.

add the possibility use multiple conf file

I will add the possibility to run different conf file in one command e.g. :
enrico_XXX Myconf1.conf Myconf2.conf

or

enrico_XXX list

where list is an ascii file with the list of conf file

`enrico/RunGtlike.py` should not be a command line tool

It's cleaner if library code and command line scripts are separated.

Therefore I suggest to remove the if __name__ == '__main__': part from enrico/RunGTlike.py.

If I see correctly the only place where RunGTlike.py is used as a command line tool is in enrico.energybin.RunEbin?

In that case we could simply call enrico.RunGTlike.run() there instead of calling RunGTlike.py as a command line tool?

@davidsanchez If you agree I think I could quickly make the change.

Add enrico_test

I'll try and add some high-level tests to Enrico.

They should process super-small datasets (a few MB max, probably < 1 MB is OK) and run Enrico analyses that are super-fast (a few minutes max, proabably < 1 min).

We can put reference results for these analyses and automatically check that everything works.

I'll write a script to do this (or extend enrico_setupcheck) and then we can tell users to run this to check their installation and probably even run it on https://travis-ci.org/ automatically for every commit.

Arbitrary energy binning for Ebin

At the IMPRS tutorial we were asked if enrico could compute spectral flux points with different binning than a log-uniform binning. In the same way we are discussing for lightcurves, they could be read from an (Emin,Emax) pair list file.

An advanced iterative version of this would be to start with many small bins and combine bins untill all are above a threshold TS (this could take a looong time because of livetime recalculations, but would definitely be interesting, as the number of bins would be given by the quality of the data).

Add ptsrc=no option to config file

Faissal is producing light curves with 100s of bins, resulting in many 100 GBs of files.

The question is how the file sizes can be reduced.

One option we were thinking of was passing ptsrc=no to gtsrcmaps, which produces the larges files (200 MB per time bin).

Or maybe add an option to remove intermediate FITS files once a time bin is finished processing?

@zblz @davidsanchez What do you think?

Update Enrico for `v9r32p5` and re-processed PASS 7

@davidsanchez The v9r32p5 version of the ScienceTools and re-processed PASS 7 data is out.
We should update Enrico (e.g. the diffuse emission files), make the 0.2 release and then get it listed here soon.
@ellisowen and I have some time to work on Enrico next week.

verbosity in lightcurve generation

Currently enrico sets verbose=no for the calls to gtlike for each of the bins of a lightcurve. That results in no likelihood output to the logfile. Given that this calls are done in the background, if the likelihood output is not saved to the logfile the user has no idea what has happened during the fitting of that bin. Since sometimes there are problems with the flux error determination because of hitting the parameter limits, I think this should be set to verbose=yes (or not modified from the initial config) so there is more info in the logfile.

fitting SED with broken-power law

Hallo Christoph and David

I want ask about possibility of fitting SED with Broken power-law, what i have to do, do i have to choose generic-model, and define broken-power-law in the xml file and then run the script

or there another way.??

2- Does the script enrico_testmodel, support testing also the broken power-law and will tell me which is the better model to fit the data, or the script as i think support only power-law and log-parapola.

I already posted this in google group, but there was no answer

Thanks

Should enrico have a setup.py?

I'm not sure if it makes sense to have a setup.py for enrico, i.e. to install it into site-packages of the Fermi Python.

Here is an example of a set of modules and scripts that doesn't to this:
https://github.com/mraue/pyfact

At the moment we have a setup.py, but it only caused me trouble so far because I was working on enrico in my git repo, but accidentally imported old versions I had previously installed via python setup.py install

I'm leaning towards removing the setup.py.
David, what do you think?