Coder Social home page Coder Social logo

icecube / flarestack Goto Github PK

View Code? Open in Web Editor NEW
8.0 4.0 6.0 48.86 MB

Unbinned likelihood analysis code for astroparticle physics datasets

Home Page: https://flarestack.readthedocs.io/en/latest/?badge=latest

License: MIT License

Python 42.95% Jupyter Notebook 57.05%
astroparticle-physics unbinned-likelihood-analysis physics-analysis icecube icecube-datasets

flarestack's Introduction

flarestack

Documentation Status CI PyPI version Coverage Status Binder DOI

Code for unbinned likelihood analysis of astroparticle physics data, created by @robertdstein.

Both time-dependent and time-independent analyses can be performed, as well as a "flare-search" algorithm to find event clustering in time as well as space.

Performs single point source analyses, as well as the stacking of sources according to predefined weighting. Also performs stacking analyses where the signal strength of each source is fit individually.

Getting started

The easiest way to start using flarestack is to play with the introductory ipython notebooks, which can be opened with the following link:

Binder

The notebooks themselves are found under examples/ipython_notebooks/.

The "Binder" provides a pre-built Docker image containing all necessary dependencies, so you can simply click and play. It avoids the need for local installation, and should provide understanding of how the code works.

Installation instructions

OPTION A: I only want to do an analysis, and trust the under-the-hood code

We recommend installing flarestack using poetry

In that case:

poetry add flarestack

This will resolve your environment and install all dependencies.

Another option is to use pip

pip install flarestack

OPTION B: Actually, I want to see the backend code myself. Maybe I want to contribute to it!

First clone the git repository:

git clone [email protected]:icecube/flarestack.git

We again recommend using poetry to install the code in your working tree. For this, go to the directory you cloned the repo into (i.e. ~/flarestack) and execute:

poetry install

This will install all packages as documented in the poetry.lock file.

If you want to also build the documentation locally, you should install the respective dependencies:

poetry install --with docs

You can still use pip if you want to:

pip install -e flarestack/

Either way will give you the very latest copy of the code, update the installed version if you git pull or modify scripts yourself, and still enable you to import flarestack.

If you do want to contribute to flarestack, you can check out some guidelines here.

Using KDE PDFs

If you want to use the KDE spatial PDFs you need to install photospline using conda. flarestack ships with a conda_env.yml file that pins the correct photospline version. Find your flarestack directory (either in your site-packages directory if you followed Option A or in your git clone if you followed Option B) and execute:

conda env create -f conda_env.yml

Alternatively, You can try and install the latest photospline version with as described here.

Right, anyway, I've now downloaded flarestack. Can I use it right away?

You can get started with flarestack immediately using public IceCube datasets provided as part of the code. You can simply run scripts such as those under /flarestack/analyses/, and do your science!

You can optionally set custom directorioes for datasets, and for storing data calculated with the code.

Setting up the dataset directory

If you are running on WIPAC or DESY, you do not need to specify a dataset directory, as IceCube data will be found automatically. Otherwise, you can add:

export FLARESTACK_DATASET_DIR=/path/to/datasets

to point the code to local copies of Icecube datasets.

Setting up directory for storing data

flarestack will produce many files that do not need to be version-controlled. The principle is that everything within this directory can be reproduced by the code, so does not need to be backed up. By default, these files will be saved in a separate within the user home directory, but it might be preferrable to save them elsewhere, such as a scratch directory. You can specify the parent directory:

export FLARESTACK_SCRATCH_DIR=/path/to/scratch

A folder flarestack__data will be created in that directory. This is where you will find plots, pickle files and other files produced by the code.

Building documentation

flarestack documentation is available at Read the Docs, however you may want to build it locally (especially if you are contributing to the project). To do this, run sphinx-build from the root directory of the repository:

sphinx-build -b html docs/source/ docs/build/html

Testing flarestack

Is flarestack actually working? You can check the functionality of flarestack with unit tests. There are a suite of unit tests to cover flarestack functionality, which can be run from the base flarestack directory with:

python -m unittest discover tests/

flarestack runs with Github Actions, a Continuous Integration Service. After each commit, the suite of tests is run, to ensure that the commit did not break anything. You can see the results of these tests at:

CI

If you want to contribute to flarestack, please remember to add new tests! The fraction of code presently covered by tests is measured using Coveralls (https://coveralls.io/). As a rule of thumb, at least 80% of the core code should be covered, but >90% would be even better. The current code coverage is:

Coverage Status

Using flarestack for IceCube analysis

flarestack is currently used for internal IceCube analysis using unpublished Monte Carlo simulations, as outlined in analysis READMEs. Additional analysis of public IceCube data using effective areas would be possible with this code, but this feature has not been tested or fully developed. Any use of this code for public data is done without the endorsement of the IceCube collaboration.

Citing flarestack

If you use flarestack for analysis, please cite it! A DOI is provided by Zenodo, which can reference both the code repository, or specific releases of Flarestack.

DOI

Contributors

flarestack's People

Contributors

dependabot[bot] avatar fbradascio avatar github-actions[bot] avatar jannisne avatar mlincett avatar msackerm avatar robertdstein avatar sathanas31 avatar simonegarrappa avatar

Stargazers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar

Watchers

 avatar  avatar  avatar  avatar

flarestack's Issues

Replace run_desy_cluster

Is your feature request related to a problem? Please describe.
run_desy_cluster and run_desy_cluster_job_id do nearly identical things

Describe the solution you'd like
run_desy_cluster should be replaced by the content of run_desy_cluster_job_id, with default behaviour that is the same as the current implementation. The name run_desy_cluster should be maintained.

Describe alternatives you've considered
Not doing this. But then you have unnecessary duplication.

Unexpected behaviour for crosscheck with skylab

Describe the bug
When stacking more than about 10 sources, flarestack's results deviate from expectations as well as from the reference skylab results.

To Reproduce
Steps to reproduce the behavior:

  1. Run 'flarestack/analyses/skylab_crosscheck/calculate_skylab_sensitivity.py'
  2. Look at the produced plots

Expected behavior
The expected relation of sensitivity flux and number of stacked sources lies in between F proportional so sqrt(N) and F=const.

sens_nsources_gamma=2.0.pdf
sens_nsources_gamma=2.0_persource.pdf

Pickle files not saved correctly when running on the cluster using issue #10 feature

Describe the bug
When submitting jobs using the feature of Issue#10, all pickle files are saved in one folder (corresponding to the max scale, i.e. the last folder to be created)

To Reproduce
Steps to reproduce the behavior:

  1. Run the following loop:
            scale_loop = np.linspace(0, 3 * mh.guess_scale(), 15)
            for scale in scale_loop:
                  mh_dict["fixed_scale"] = scale
                  analyse(mh_dict, cluster=True, n_cpu=1, n_jobs=n_jobs)

Expected behavior
15 folders should be created (one for each step) and 100 pickle files should be saved in each one of them. Instead all 150 pickle files are saved in the 15th folder.

Additional context

tailor default memory to dataset

Is your feature request related to a problem? Please describe.
I'm always frustrated when I start a job on the desy cluster, then it dies because the memory requested is insufficient. This is not a hard problem, to first order it scales with the size of the MC.

Describe the solution you'd like
A method to adjust the default memory request depending on the dataset MC. Maybe by checking the dataset MC file size without loading?

Describe alternatives you've considered
Leave it as is. It's not too annoying, you figure it out manually once per analysis.

update readme

Is your feature request related to a problem? Please describe.
I would like to remove the redundancy in requirements.txt/setup.py, and have a clearer setup/install process in the readme

Describe the solution you'd like
pip install -e . for option b

Describe alternatives you've considered
Not doing that. But it wouldn't be as clear

Error for dumping flare injection values

Describe the bug
Discovered bug that the flare cluster search method still uses the old deprecated method for accessing injectors.

To Reproduce
Steps to reproduce the behavior:

  1. Run any analysis using mh_name = flare
  2. See:
    File "/afs/ifh.de/user/s/steinrob/flarestack-dev/flarestack/core/minimisation.py", line 1562, in return_injected_parameters for inj in self.injectors.values(): AttributeError: 'FlareMinimisationHandler' object has no attribute 'injectors'

Expected behavior
No error message

Additional context
Problem was not caught before, despite there being a unit test for the flare search, because the function return_injected_parameters is not covered by present unit tests. That should also be remedied.

Clear log cache for DESY cluster scripts

Is your feature request related to a problem? Please describe.
I'm always frustrated when I send something to the desy cluster, want to check the logs, and discover 10000000 old logs there too. I would like the old logs to be cleared when new jobs are submitted.

Describe the solution you'd like
The pre-submitter status quo ante should be restored. On cluster submission, old logs (maybe by time of creation or by job number) should be deleted.

Describe alternatives you've considered
Deleting manually. But rm -rf .... is bound to go wrong eventually.

Exception handling in `ic_season.get_published_sens_ref_dir()`

It is not clear to me how the exception handling is supposed to work in the following:

def get_published_sens_ref_dir():
try:
return ref_dir_7yr, ref_10yr
except NameError:
logger.error(
"No reference sensitivity directory found. "
"Please create one at {0}".format(
icecube_dataset_dir + "mirror-7year-PS-sens/"
)
)
raise

From the rest of the module, I see no way a NameError can be caught there, but ref_dir_7yr, ref_10yr will rather have None values.

Once we agree on the desired behaviour I am happy to follow-up on the issue.

"No Background TS Distribution found" when looping over catalogues in Unblinder Class

Describe the bug
"No Background TS Distribution found" when using the Unblinder Class for the AGN cores stacking analysis.

To Reproduce
Steps to reproduce the behavior:

  1. Run 'flarestack/analyses/agn_cores/scripts_for_unblinding/unblind_agn_cores.py
  2. Use setting 'mock_unblinding=True'
  3. See error

Expected behavior
Obtain upper limits and plots for unblinded results.

Additional context
The problem is related to the fact the code originally loops over injection properties, while in this case should loop over sub-catalogues (classified on the number of sources).

Corrupted cached band mask

Describe the bug
Cached band mask seems to be corrupted

To Reproduce
Steps to reproduce the behavior:

  1. Run any analysis on the cluster with a new catalogue before running it locally
  2. It will crush
  3. Run it again locally and you will see the same error

Expected behavior
The band mask should be produced also on the cluster

Additional context
ERROR MESSAGE:
inj = self.mh.get_injector(season)
File "/afs/ifh.de/user/b/bradascf/flarestack/flarestack/core/minimisation.py", line 273, in get_injector
self._injectors[season_name] = self.add_injector(self.seasons[season_name], self.sources)
File "/afs/ifh.de/user/b/bradascf/flarestack/flarestack/core/minimisation.py", line 1004, in add_injector
return season.make_injector(sources, **self.inj_dict)
File "/afs/ifh.de/user/b/bradascf/flarestack/flarestack/data/init.py", line 272, in make_injector
return MCInjector.create(self, sources, **inj_kwargs)
File "/afs/ifh.de/user/b/bradascf/flarestack/flarestack/core/injector.py", line 201, in create
return cls.subclasses[inj_name](season, sources, **inj_dict)
File "/afs/ifh.de/user/b/bradascf/flarestack/flarestack/core/injector.py", line 407, in init
MCInjector.init(self, season, sources, **kwargs)
File "/afs/ifh.de/user/b/bradascf/flarestack/flarestack/core/injector.py", line 231, in init
self.n_exp = self.calculate_n_exp()
File "/afs/ifh.de/user/b/bradascf/flarestack/flarestack/core/injector.py", line 430, in calculate_n_exp
self.n_exp[i]["n_exp"] = self.calculate_n_exp_single(source)
File "/afs/ifh.de/user/b/bradascf/flarestack/flarestack/core/injector.py", line 283, in calculate_n_exp_single
return np.sum(self.calculate_single_source(source, 1.)["ow"])
File "/afs/ifh.de/user/b/bradascf/flarestack/flarestack/core/injector.py", line 275, in calculate_single_source
source_mc, omega, band_mask = self.select_mc_band(source)
File "/afs/ifh.de/user/b/bradascf/flarestack/flarestack/core/injector.py", line 250, in select_mc_band
band_mask = self.get_band_mask(source, min_dec, max_dec)
File "/afs/ifh.de/user/b/bradascf/flarestack/flarestack/core/injector.py", line 474, in get_band_mask
self.load_band_mask(mask_index[0])
File "/afs/ifh.de/user/b/bradascf/flarestack/flarestack/core/injector.py", line 463, in load_band_mask
self.band_mask_cache = sparse.load_npz(path)
File "/afs/ifh.de/user/b/bradascf/.local/lib/python3.6/site-packages/scipy/sparse/_matrix_io.py", line 133, in load_npz
matrix_format = loaded['format']
File "/afs/ifh.de/user/b/bradascf/.local/lib/python3.6/site-packages/numpy/lib/npyio.py", line 255, in getitem
bytes = self.zip.open(key)
File "/cvmfs/icecube.opensciencegrid.org/py3-v4/RHEL_7_x86_64/lib/python3.6/zipfile.py", line 1373, in open
raise BadZipFile("Bad magic number for file header")
zipfile.BadZipFile: Bad magic number for file header

Problem in MCInjector (ref_fluxes)

Try to run MinimisationHandler and iterate_run function. File: benchmark/benchmark_point_source_sensitivity.py

Traceback (most recent call last): File "flarestack/analyses/agn_cores/benchmark_point_source_sensitivity.py", line 86, in <module> mh.iterate_run(mh_dict["scale"], n_steps=10, n_trials=mh_dict["n_trials"]) File "/afs/ifh.de/user/b/bradascf/flarestack/flarestack/core/minimisation.py", line 241, in iterate_run self.run(n_trials*10, scale=0.0) File "/afs/ifh.de/user/b/bradascf/flarestack/flarestack/core/minimisation.py", line 526, in run for val in inj.ref_fluxes[scale_shortener(scale)].values(): AttributeError: 'MCInjector' object has no attribute 'ref_fluxes'

Unit test coverage for cluster submission

Is your feature request related to a problem? Please describe.
I'm always frustrated when I discover a minor bug in the cluster submission process. These are currently not checked. This is related to #32.

Describe the solution you'd like
We cannot actually submit stuff to the cluster with unit tests. But we can at least try running the specific execution commands locally. i.e using python to execute bash commands that run flarestack. Of course, to de-desyify the the submission script, #35 would need to be resolved first.

Describe alternatives you've considered
Hoping for the best. It might work.

At least one local run required

Describe the bug
Sometimes if only run on the cluster, there seems to be a problem with dealing with sparse matrices. One local run can circumvent that

To Reproduce
Steps to reproduce the behavior:

  1. Run 'flare stack/analyses/skylab_crosscheck/calculate_skylab_sensitivity.py'
  2. Set sin(dec) to 0.25 and run in the cluster
  3. See error:
    57164053_stderr.txt

Expected behavior
No error.

Cover unblinding/upper limits in unit tests

Is your feature request related to a problem? Please describe.
I'm always frustrated when bugs appear for unblinding/upper limits, since these are not currently covered by unit tests.

Describe the solution you'd like
Run a mock unblinding, and set upper limits, in test_full_analysis_chain.py after the simulation step.

Describe alternatives you've considered
Leaving it to chance, and not testing. But no, that's dumb.

Documentation for core/energy_pdf.py

Is your feature request related to a problem? Please describe.
I'm always frustrated when there is no documentation for code.

Describe the solution you'd like
I want documented code. More so probably people from outside using Flarestack.

Describe alternatives you've considered
Not adding documentation. Makes it harder for everyone though.

Additional context
This is necessary if a JOSS paper is the goal.

TS type for Unblinder Class

Describe the bug

To Reproduce
Steps to reproduce the behavior:

  1. Run 'flarestack/analyses/ccsn/necker_2019/unblind_ccsn_update.py'
  2. Make sure mock_unblinding=True in line 52!
  3. See "AttributeError: 'Unblinder' object has no attribute 'ts_type'"

Expected behavior
The ideal behaviour would be that you try fixed-weights, check the fit quality, and if not then try fit weights instead.

Sumbit cluster jobs with fixed scale

Is your feature request related to a problem? Please describe.
I'm always frustrated when I can't run w/ O(1e3) on the cluster

Describe the solution you'd like
Being able to split more the jobs

Describe alternatives you've considered
Kill myself.

Additional context

Use energy proxy mapping from TXS 0506+056 data release to build energy PDfs

Is your feature request related to a problem? Please describe.
Currently, it is not possible to convert true energy into reconstructed energy proxy for public data.

Describe the solution you'd like
A mapping between true energy vs energy proxy was recently published by IceCube (Oct 22nd, 2019 at https://icecube.wisc.edu/science/data/TXS0506_point_source). This can be incorporated into the code.

Describe alternatives you've considered
Assuming energy proxy = lognormal centered on true energy, or exactly equals true energy. But that's dumb/oversimplified, and does not recreate PDFs properly.

Reorganise llh.py

Is your feature request related to a problem? Please describe.
I'm always frustrated when coding in llh.py, because many things are nearly identical/redundant.

Describe the solution you'd like
A rationalisation, where the standard matrix method is used as default. Ideally, the 'standard' likelihood method would be removed entirely, but then things won't play nice with fitting_weights and flares. So it'd take some work.

Describe alternatives you've considered
Partially renaming things but maintaining separate implementations. It's easier in the short term, but more cumbersome to maintain.

Additional context
Flarestack will give nonsensical results if sources overlap under the default 'standard', but works fine with matrices. This is dangerous default behaviour. Changing names would probably lead to a change where old configs do not work anymore, or do different things.

Azimuth Spatial PDFs

Is your feature request related to a problem? Please describe.
I'm always frustrated by the fact that IceCube is like an old sci fi movie-You can see all the strings! Background events preferentially align with string axes, and you could use that to reject background.

Describe the solution you'd like
Events to be weighted based on 2D azimuth+sin(dec) splines. But, it should be noted that the signal azimuth dependence is energy-dependent. Interpolation a la energy? Care should be taken for injection (with azimuth, by shuffling/sampling from MC without needing it to align with time?). Also, in theory, would lead to signal oscillations with period 24/6 hours. Probably negligible?

See:
azimuth_mc.pdf

Describe alternatives you've considered
Leaving things as is. Easier, but might be missing option to reject some extra background.

Additional context
Could be particularly useful in the southern sky. See background distribution:

azimuth.pdf

Mismatched number of trials when running on the DESY cluster

When submitting a job to the DESY cluster, for large number of sources, the number of trials set in the minimisation dictionary does not match the one shown in INFO:flarestack.core.results, which leads to a failure of the DP/sensitivity analysis.

Can anyone please help?
Screenshot 2022-03-15 at 11 01 33

Drop Python 3.7

Is your feature request related to a problem? Please describe.
I'm always frustrated when I realise flarestack cannot work with astropy v5.0

Describe the solution you'd like
Drop suport for python 3.7

Describe alternatives you've considered
Stay on the old astropy v4.3 forever

bias plots not working with fitting weights

Describe the bug
Bias plots don't show the expected relation between injected and retrieved number of neutrinos.
It looks like by summing up the injected numbers of all transients, the data would lie on the red dashed line. Might indicate that actually the number of retrieved neutrinos for a single event is plotted against the total number of injected neutrinos.

To Reproduce
Steps to reproduce the behavior:

  1. Run 'flarestack/analyses/ccsn/necker_2019/calculate_sensitivity.py'
  2. look at the bias plots
    bias_n_s (MASTER OT J120451.50+265946.6).pdf

Wrong measure of number of injected neutrinos

The number of injected neutrinos seems always to be measured as zero.

To Reproduce
Steps to reproduce the behavior:

  1. Run 'flarestack/analyses/ccsn/stasik/calculate_sensitivity_reproduce_stasik.py'
  2. look at the bias n_s plots of the individual sources

Expected behavior
n_s should rough;y be equal to n_injected

bias_n_s (b'SN2008S').pdf

Trouble Running Flarestack on Ubuntu 20.04

Describe the bug
When I run

python -m unittest discover tests/

In the flarestack folder in my terminal, it throws me me errors saying there is no module named numpy, even though I ran sudo pip3 install -U numpy

It also tells me that there is invalid syntax in some places. What can I do?
To Reproduce
Steps to reproduce the behavior:

  1. Run 'python -m unittest discover tests/'
  2. Use Ubuntu 20.04
  3. See error:
advait@advait-dhingra:~/Documents/Coding/flarestack$ python -m unittest discover tests/
EEEEEEEEEEEEEEEEEEEEE
======================================================================
ERROR: test_likelihood_spatial (unittest.loader.ModuleImportFailure)
----------------------------------------------------------------------
ImportError: Failed to import test module: test_likelihood_spatial
Traceback (most recent call last):
  File "/usr/lib/python2.7/unittest/loader.py", line 254, in _find_tests
    module = self._get_module_from_name(name)
  File "/usr/lib/python2.7/unittest/loader.py", line 232, in _get_module_from_name
    __import__(name)
  File "/home/advait/Documents/Coding/flarestack/tests/test_likelihood_spatial.py", line 2, in <module>
    from flarestack.data.public import icecube_ps_3_year
  File "flarestack/__init__.py", line 1, in <module>
    from flarestack.core.minimisation import MinimisationHandler
  File "flarestack/core/minimisation.py", line 73
    raise Exception(f"MinimisationHandler dictionary contained both 'fixed_scale' key for "
                                                                                          ^
SyntaxError: invalid syntax


======================================================================
ERROR: test_util_custom_dataset (unittest.loader.ModuleImportFailure)
----------------------------------------------------------------------
ImportError: Failed to import test module: test_util_custom_dataset
Traceback (most recent call last):
  File "/usr/lib/python2.7/unittest/loader.py", line 254, in _find_tests
    module = self._get_module_from_name(name)
  File "/usr/lib/python2.7/unittest/loader.py", line 232, in _get_module_from_name
    __import__(name)
  File "/home/advait/Documents/Coding/flarestack/tests/test_util_custom_dataset.py", line 6, in <module>
    from flarestack.data.public import icecube_ps_3_year
  File "flarestack/__init__.py", line 1, in <module>
    from flarestack.core.minimisation import MinimisationHandler
  File "flarestack/core/minimisation.py", line 73
    raise Exception(f"MinimisationHandler dictionary contained both 'fixed_scale' key for "
                                                                                          ^
SyntaxError: invalid syntax


======================================================================
ERROR: test_likelihood_standard_matrix (unittest.loader.ModuleImportFailure)
----------------------------------------------------------------------
ImportError: Failed to import test module: test_likelihood_standard_matrix
Traceback (most recent call last):
  File "/usr/lib/python2.7/unittest/loader.py", line 254, in _find_tests
    module = self._get_module_from_name(name)
  File "/usr/lib/python2.7/unittest/loader.py", line 232, in _get_module_from_name
    __import__(name)
  File "/home/advait/Documents/Coding/flarestack/tests/test_likelihood_standard_matrix.py", line 6, in <module>
    from flarestack.data.public import icecube_ps_3_year
  File "flarestack/__init__.py", line 1, in <module>
    from flarestack.core.minimisation import MinimisationHandler
  File "flarestack/core/minimisation.py", line 73
    raise Exception(f"MinimisationHandler dictionary contained both 'fixed_scale' key for "
                                                                                          ^
SyntaxError: invalid syntax


======================================================================
ERROR: test_asimov_estimator (unittest.loader.ModuleImportFailure)
----------------------------------------------------------------------
ImportError: Failed to import test module: test_asimov_estimator
Traceback (most recent call last):
  File "/usr/lib/python2.7/unittest/loader.py", line 254, in _find_tests
    module = self._get_module_from_name(name)
  File "/usr/lib/python2.7/unittest/loader.py", line 232, in _get_module_from_name
    __import__(name)
  File "/home/advait/Documents/Coding/flarestack/tests/test_asimov_estimator.py", line 5, in <module>
    import numpy as np
ImportError: No module named numpy


======================================================================
ERROR: test_mh_fit_weights (unittest.loader.ModuleImportFailure)
----------------------------------------------------------------------
ImportError: Failed to import test module: test_mh_fit_weights
Traceback (most recent call last):
  File "/usr/lib/python2.7/unittest/loader.py", line 254, in _find_tests
    module = self._get_module_from_name(name)
  File "/usr/lib/python2.7/unittest/loader.py", line 232, in _get_module_from_name
    __import__(name)
  File "/home/advait/Documents/Coding/flarestack/tests/test_mh_fit_weights.py", line 6, in <module>
    from flarestack.data.public import icecube_ps_3_year
  File "flarestack/__init__.py", line 1, in <module>
    from flarestack.core.minimisation import MinimisationHandler
  File "flarestack/core/minimisation.py", line 73
    raise Exception(f"MinimisationHandler dictionary contained both 'fixed_scale' key for "
                                                                                          ^
SyntaxError: invalid syntax


======================================================================
ERROR: test_simulate_catalogue (unittest.loader.ModuleImportFailure)
----------------------------------------------------------------------
ImportError: Failed to import test module: test_simulate_catalogue
Traceback (most recent call last):
  File "/usr/lib/python2.7/unittest/loader.py", line 254, in _find_tests
    module = self._get_module_from_name(name)
  File "/usr/lib/python2.7/unittest/loader.py", line 232, in _get_module_from_name
    __import__(name)
  File "/home/advait/Documents/Coding/flarestack/tests/test_simulate_catalogue.py", line 6, in <module>
    from flarestack.utils import load_catalogue
  File "flarestack/__init__.py", line 1, in <module>
    from flarestack.core.minimisation import MinimisationHandler
  File "flarestack/core/minimisation.py", line 73
    raise Exception(f"MinimisationHandler dictionary contained both 'fixed_scale' key for "
                                                                                          ^
SyntaxError: invalid syntax


======================================================================
ERROR: test_util_astronomy (unittest.loader.ModuleImportFailure)
----------------------------------------------------------------------
ImportError: Failed to import test module: test_util_astronomy
Traceback (most recent call last):
  File "/usr/lib/python2.7/unittest/loader.py", line 254, in _find_tests
    module = self._get_module_from_name(name)
  File "/usr/lib/python2.7/unittest/loader.py", line 232, in _get_module_from_name
    __import__(name)
  File "/home/advait/Documents/Coding/flarestack/tests/test_util_astronomy.py", line 6, in <module>
    from flarestack.analyses.tde.shared_TDE import tde_catalogue_name
  File "flarestack/__init__.py", line 1, in <module>
    from flarestack.core.minimisation import MinimisationHandler
  File "flarestack/core/minimisation.py", line 73
    raise Exception(f"MinimisationHandler dictionary contained both 'fixed_scale' key for "
                                                                                          ^
SyntaxError: invalid syntax


======================================================================
ERROR: test_mh_flare (unittest.loader.ModuleImportFailure)
----------------------------------------------------------------------
ImportError: Failed to import test module: test_mh_flare
Traceback (most recent call last):
  File "/usr/lib/python2.7/unittest/loader.py", line 254, in _find_tests
    module = self._get_module_from_name(name)
  File "/usr/lib/python2.7/unittest/loader.py", line 232, in _get_module_from_name
    __import__(name)
  File "/home/advait/Documents/Coding/flarestack/tests/test_mh_flare.py", line 6, in <module>
    from flarestack import MinimisationHandler, analyse
  File "flarestack/__init__.py", line 1, in <module>
    from flarestack.core.minimisation import MinimisationHandler
  File "flarestack/core/minimisation.py", line 73
    raise Exception(f"MinimisationHandler dictionary contained both 'fixed_scale' key for "
                                                                                          ^
SyntaxError: invalid syntax


======================================================================
ERROR: test_cosmo_rates (unittest.loader.ModuleImportFailure)
----------------------------------------------------------------------
ImportError: Failed to import test module: test_cosmo_rates
Traceback (most recent call last):
  File "/usr/lib/python2.7/unittest/loader.py", line 254, in _find_tests
    module = self._get_module_from_name(name)
  File "/usr/lib/python2.7/unittest/loader.py", line 232, in _get_module_from_name
    __import__(name)
  File "/home/advait/Documents/Coding/flarestack/tests/test_cosmo_rates.py", line 3, in <module>
    import numpy as np
ImportError: No module named numpy


======================================================================
ERROR: test_full_analysis_chain (unittest.loader.ModuleImportFailure)
----------------------------------------------------------------------
ImportError: Failed to import test module: test_full_analysis_chain
Traceback (most recent call last):
  File "/usr/lib/python2.7/unittest/loader.py", line 254, in _find_tests
    module = self._get_module_from_name(name)
  File "/usr/lib/python2.7/unittest/loader.py", line 232, in _get_module_from_name
    __import__(name)
  File "/home/advait/Documents/Coding/flarestack/tests/test_full_analysis_chain.py", line 58
    "name": f"{base_name}/{gamma}/",
                                  ^
SyntaxError: invalid syntax


======================================================================
ERROR: test_submitter (unittest.loader.ModuleImportFailure)
----------------------------------------------------------------------
ImportError: Failed to import test module: test_submitter
Traceback (most recent call last):
  File "/usr/lib/python2.7/unittest/loader.py", line 254, in _find_tests
    module = self._get_module_from_name(name)
  File "/usr/lib/python2.7/unittest/loader.py", line 232, in _get_module_from_name
    __import__(name)
  File "/home/advait/Documents/Coding/flarestack/tests/test_submitter.py", line 5, in <module>
    from flarestack.shared import flux_to_k
  File "flarestack/__init__.py", line 1, in <module>
    from flarestack.core.minimisation import MinimisationHandler
  File "flarestack/core/minimisation.py", line 73
    raise Exception(f"MinimisationHandler dictionary contained both 'fixed_scale' key for "
                                                                                          ^
SyntaxError: invalid syntax


======================================================================
ERROR: test_likelihood_standard (unittest.loader.ModuleImportFailure)
----------------------------------------------------------------------
ImportError: Failed to import test module: test_likelihood_standard
Traceback (most recent call last):
  File "/usr/lib/python2.7/unittest/loader.py", line 254, in _find_tests
    module = self._get_module_from_name(name)
  File "/usr/lib/python2.7/unittest/loader.py", line 232, in _get_module_from_name
    __import__(name)
  File "/home/advait/Documents/Coding/flarestack/tests/test_likelihood_standard.py", line 6, in <module>
    import numpy as np
ImportError: No module named numpy


======================================================================
ERROR: test_likelihood_standard_overlapping (unittest.loader.ModuleImportFailure)
----------------------------------------------------------------------
ImportError: Failed to import test module: test_likelihood_standard_overlapping
Traceback (most recent call last):
  File "/usr/lib/python2.7/unittest/loader.py", line 254, in _find_tests
    module = self._get_module_from_name(name)
  File "/usr/lib/python2.7/unittest/loader.py", line 232, in _get_module_from_name
    __import__(name)
  File "/home/advait/Documents/Coding/flarestack/tests/test_likelihood_standard_overlapping.py", line 6, in <module>
    from flarestack.data.public import icecube_ps_3_year
  File "flarestack/__init__.py", line 1, in <module>
    from flarestack.core.minimisation import MinimisationHandler
  File "flarestack/core/minimisation.py", line 73
    raise Exception(f"MinimisationHandler dictionary contained both 'fixed_scale' key for "
                                                                                          ^
SyntaxError: invalid syntax


======================================================================
ERROR: test_mh_fixed_weights (unittest.loader.ModuleImportFailure)
----------------------------------------------------------------------
ImportError: Failed to import test module: test_mh_fixed_weights
Traceback (most recent call last):
  File "/usr/lib/python2.7/unittest/loader.py", line 254, in _find_tests
    module = self._get_module_from_name(name)
  File "/usr/lib/python2.7/unittest/loader.py", line 232, in _get_module_from_name
    __import__(name)
  File "/home/advait/Documents/Coding/flarestack/tests/test_mh_fixed_weights.py", line 6, in <module>
    from flarestack.data.public import icecube_ps_3_year
  File "flarestack/__init__.py", line 1, in <module>
    from flarestack.core.minimisation import MinimisationHandler
  File "flarestack/core/minimisation.py", line 73
    raise Exception(f"MinimisationHandler dictionary contained both 'fixed_scale' key for "
                                                                                          ^
SyntaxError: invalid syntax


======================================================================
ERROR: test_time_pdfs (unittest.loader.ModuleImportFailure)
----------------------------------------------------------------------
ImportError: Failed to import test module: test_time_pdfs
Traceback (most recent call last):
  File "/usr/lib/python2.7/unittest/loader.py", line 254, in _find_tests
    module = self._get_module_from_name(name)
  File "/usr/lib/python2.7/unittest/loader.py", line 232, in _get_module_from_name
    __import__(name)
  File "/home/advait/Documents/Coding/flarestack/tests/test_time_pdfs.py", line 6, in <module>
    from flarestack.data.public import icecube_ps_3_year
  File "flarestack/__init__.py", line 1, in <module>
    from flarestack.core.minimisation import MinimisationHandler
  File "flarestack/core/minimisation.py", line 73
    raise Exception(f"MinimisationHandler dictionary contained both 'fixed_scale' key for "
                                                                                          ^
SyntaxError: invalid syntax


======================================================================
ERROR: test_cosmo (unittest.loader.ModuleImportFailure)
----------------------------------------------------------------------
ImportError: Failed to import test module: test_cosmo
Traceback (most recent call last):
  File "/usr/lib/python2.7/unittest/loader.py", line 254, in _find_tests
    module = self._get_module_from_name(name)
  File "/usr/lib/python2.7/unittest/loader.py", line 232, in _get_module_from_name
    __import__(name)
  File "/home/advait/Documents/Coding/flarestack/tests/test_cosmo.py", line 6, in <module>
    from flarestack.cosmo import get_diffuse_flux_at_1GeV, get_diffuse_flux_at_100TeV, \
  File "flarestack/__init__.py", line 1, in <module>
    from flarestack.core.minimisation import MinimisationHandler
  File "flarestack/core/minimisation.py", line 73
    raise Exception(f"MinimisationHandler dictionary contained both 'fixed_scale' key for "
                                                                                          ^
SyntaxError: invalid syntax


======================================================================
ERROR: test_mh_large_catalogue (unittest.loader.ModuleImportFailure)
----------------------------------------------------------------------
ImportError: Failed to import test module: test_mh_large_catalogue
Traceback (most recent call last):
  File "/usr/lib/python2.7/unittest/loader.py", line 254, in _find_tests
    module = self._get_module_from_name(name)
  File "/usr/lib/python2.7/unittest/loader.py", line 232, in _get_module_from_name
    __import__(name)
  File "/home/advait/Documents/Coding/flarestack/tests/test_mh_large_catalogue.py", line 6, in <module>
    from flarestack.data.public import icecube_ps_3_year
  File "flarestack/__init__.py", line 1, in <module>
    from flarestack.core.minimisation import MinimisationHandler
  File "flarestack/core/minimisation.py", line 73
    raise Exception(f"MinimisationHandler dictionary contained both 'fixed_scale' key for "
                                                                                          ^
SyntaxError: invalid syntax


======================================================================
ERROR: test_energy_pdfs (unittest.loader.ModuleImportFailure)
----------------------------------------------------------------------
ImportError: Failed to import test module: test_energy_pdfs
Traceback (most recent call last):
  File "/usr/lib/python2.7/unittest/loader.py", line 254, in _find_tests
    module = self._get_module_from_name(name)
  File "/usr/lib/python2.7/unittest/loader.py", line 232, in _get_module_from_name
    __import__(name)
  File "/home/advait/Documents/Coding/flarestack/tests/test_energy_pdfs.py", line 7, in <module>
    import numpy as np
ImportError: No module named numpy


======================================================================
ERROR: test_negative_ns (unittest.loader.ModuleImportFailure)
----------------------------------------------------------------------
ImportError: Failed to import test module: test_negative_ns
Traceback (most recent call last):
  File "/usr/lib/python2.7/unittest/loader.py", line 254, in _find_tests
    module = self._get_module_from_name(name)
  File "/usr/lib/python2.7/unittest/loader.py", line 232, in _get_module_from_name
    __import__(name)
  File "/home/advait/Documents/Coding/flarestack/tests/test_negative_ns.py", line 6, in <module>
    import numpy as np
ImportError: No module named numpy


======================================================================
ERROR: test_simcube (unittest.loader.ModuleImportFailure)
----------------------------------------------------------------------
ImportError: Failed to import test module: test_simcube
Traceback (most recent call last):
  File "/usr/lib/python2.7/unittest/loader.py", line 254, in _find_tests
    module = self._get_module_from_name(name)
  File "/usr/lib/python2.7/unittest/loader.py", line 232, in _get_module_from_name
    __import__(name)
  File "/home/advait/Documents/Coding/flarestack/tests/test_simcube.py", line 5, in <module>
    import numpy as np
ImportError: No module named numpy


======================================================================
ERROR: test_likelihood_fixed_energy (unittest.loader.ModuleImportFailure)
----------------------------------------------------------------------
ImportError: Failed to import test module: test_likelihood_fixed_energy
Traceback (most recent call last):
  File "/usr/lib/python2.7/unittest/loader.py", line 254, in _find_tests
    module = self._get_module_from_name(name)
  File "/usr/lib/python2.7/unittest/loader.py", line 232, in _get_module_from_name
    __import__(name)
  File "/home/advait/Documents/Coding/flarestack/tests/test_likelihood_fixed_energy.py", line 53
    "name": f"tests/test_fixed_energy/{sindec:.2f}/",
                                                   ^
SyntaxError: invalid syntax


----------------------------------------------------------------------
Ran 21 tests in 0.001s

FAILED (errors=21)

Expected behavior
I would expect the tests to succeed

ModuleNotFoundError Notebook examples not working

Describe the bug
The example notebooks provided in the docs are inconsistent with the current version of flarestack.

To Reproduce
Steps to reproduce the behavior:

  1. Run '0-Setting up Flarestack.ipynb' on binder or locally
  2. Run Cells
  3. Module error for flarestack.precompute

Expected behavior
precompute.py has been removed in the latest release v1.2.2

Additional context

ModuleNotFoundError Traceback (most recent call last)
in
1 from pathlib import Path
2 home = str(Path.home())
----> 3 from flarestack.precompute import run_precompute, set_scratch_directory
4 set_scratch_directory(home)

ModuleNotFoundError: No module named 'flarestack.precompute'

Reduce memory consumption of input dictionaries

Is your feature request related to a problem? Please describe.
Disk quota exceeded when creating a new dictionary in flarestack/input/analysis/

Describe the solution you'd like
To reduce the memory consumption of each dict.

Describe alternatives you've considered
To delete manually old dictionaries.

Additional context

Update to include new Icecube public data

Is your feature request related to a problem? Please describe.
I'm always frustrated when I'm using data from 2013, even though Icecube has released more complete data since.

Describe the solution you'd like
Include datasets listed in https://arxiv.org/abs/2101.09836

Describe alternatives you've considered
Leave things as they are. You would just be needlessly missing new data.

Toy MC for unit tests

Is your feature request related to a problem? Please describe.
I'm always frustrated when bugs turn up involving MC, as these are not currently covered by unit tests.

Describe the solution you'd like
A small toy MC dataset, with the same structure as the proprietary IceCube MC, should be made and be included as package data. Unit tests should cover these cases.

Describe alternatives you've considered
Not doing that. "Never making a mistake ever in coding" is an alternative solution, but the preferred one may be easier.

Change default behaviour of the Unblinder when calculating limits

Describe the bug
The injection time is needed to multiply it with the mean luminosity to get the mean energy emitted in neutrinos. The Unblinder calculates this injection time using TimePDF.raw_injection_time() which gives the overlap of the source with the dataset.

Expected behavior
This is an at least controversial way of doing it since for example for a box time pdf we would rather want the box length and not the overlap of the sources with the dataset.

Solution
We could define a property of the TimePDF`` called maybe length``` that return the injection length. This could then be used by the Unblinder.

Estimate sensitivity scale

I'm always frustrated when I have to guess the injection scale where I assume the sensitivity to be and run the injection steps to find out I was wrong.

This could be avoided by first estimating the scale using quick injections with a low number of trials.

That does however require the background TS distribution which would mean that the background TS calculation and the signal injections would have to be submitted in two subsequent steps and not, as it is now, at the same time.

sensitivity.pdf

Confusing number of expected neutrinos from LowMemoryInjector

Describe the bug
The LowMemoryInjector seems to give a number of expected neutrinos for any given scale that can vastly differ from the MCInjector. The difference depends on the injection_declination_bandwidth. For values lower than 1.5 degrees the LowMemoryInjector gets a neutrino number much larger than the one calculated with the MCInjector. Vice versa for higher values.

To Reproduce
The notebook analyses/ccsn/necker_2019/check_injections-compare_flarestack_injections.ipynb illustrates that behaviour. Run with different values for injection_bandwidth and see the resulting figures. Examples are attached.
1 degree: northern_2.00.pdf
2 degrees: northern_2.00.pdf

Expected behavior
The Injectors should give the same estimate for the neutrino number.

Eliminate shared.py

Is your feature request related to a problem? Please describe.
I'm always frustrated when I realise shared.py is a redundant pseudo-init file

Describe the solution you'd like
Everything from shared should be moved to init.py.

Describe alternatives you've considered
Leaving as is. Less time, but less readable for people who aren't me.

Add error estimate to sensitivity/discovery potential

Is your feature request related to a problem? Please describe.
I'm always frustrated when you calculate a sensitivity, but have to judge by eye how reliable the estimate is.

Describe the solution you'd like
Use sqrt(n) as an assumption for the error, curve fit, and propagate through to get a range of sensitivities. Error on thresholds themselves would be more difficult, but possible.

Describe alternatives you've considered
Leaving things as they are. Not wrong, but less informative.

Unbiased fits for ns for large catalogues

Describe the bug
Currently ns bias plots indicate fitted parameters are not well-recovered for large (O(1-10k)) catalogues.

To Reproduce
Steps to reproduce the behavior:

  1. Run e.g flarestack/analyses/agn_cores/test_completeness.py
  2. See bias plot

Expected behavior
Bias-free fit.

Curl warning

Describe the bug
When running the wait_for_cluster() function, the following error appears:

curl: option --negotiate: the installed libcurl version doesn't support this curl: try 'curl --help' or 'curl --manual' for more information error: commlib error: can't set CA chain file (/afs/ifh.de/user/s/steinrob/.sge/cert.pem) error: commlib error: ssl error ([ID=336245134] in module "SSL routines": "ca md too weak") error: unable to send message from qstat to qmaster using port 6444 on host "ate.zeuthen.desy.de": can't set CA chain file

To Reproduce
Steps to reproduce the behavior:

  1. Runwait_for_cluster()
  2. See error

Expected behavior
should see no warning

Move cluster script creation to `flarestack__data`

Is your feature request related to a problem? Please describe.
I'm always frustrated when I see the random cluster scripts appearing in the flarestack package directory. They are "disposable" self-generated scripts, so belong in flarestack__data with everything else.

Describe the solution you'd like
Move the output directory to flarestack__data/cluster

Describe alternatives you've considered
Not doing that. No huge disadvantage, but less confusing.

Smarter python path setup for cluster

Is your feature request related to a problem? Please describe.
I'm always frustrated when I have to limit flarestack dependencies to those covered by cvmfs/icecube.... In case we ever wanted new dependencies, or more up to date software versions, things would break. In any case, we shouldn't be forcing a particular method of installing flarestack.

Describe the solution you'd like
Whichever python path used to execute a given flarestack submission script should then be directly written into the generated bash script, and that same python path can then be used by the cluster. That way, we are neutral to which method of flarestack installation is applied, whether it is conda, virtual environment or just "cvmfs&export pythonpath". The PYTHONPATH exporting can continue as normal, as it doesn't harm things. The user is responsible for sorting out their own dependencies, and then the cluster submission can be consistently personalised to each user.

Describe alternatives you've considered
Leaving things as is. There will be an overhead because the submitter module will need to be regularly updated to use the most recent cvmfs python iteration. Also restricts from ever exploring new dependencies.

Reference sensitivity not found when running in Madison

Describe the bug
When trying to load the reference discovery potential for the 7yr Point Source Analysis the file can not be found when running in Madison.

To Reproduce
Steps to reproduce the behavior:

  1. Run 'flarestack/analyses/benchmarks/benchmark_point_source_sensitivity_7yr.py' on a Madison Cobalt machine.
  2. See error

Expected behavior
No bug.

Additional context
It seems that the file /data/user/steinrob/mirror-7year-PS-sens/disc.npy is missing.

Binder no longer works

Describe the bug
Using Binder, and then running the example notebooks, no longer works. The "package data" is not installed for some reason. Perhaps it can be fixed via e.g requirements.txt, with a command to force install package data.

To Reproduce
Go to binder.
Go to examples, ipython_notebooks, and then 1-Introducing the public IceCube dataset.
Run cells
FileNotFoundError: [Errno 2] No such file or directory: '/srv/conda/envs/notebook/lib/python3.7/site-packages/flarestack/data/public/icecube/all_sky_point_source/raw_data/3year-data-release.zip'

Expected behavior
The scripts should run without this error.

Code coverage above >90%

Is your feature request related to a problem? Please describe.
I'm always frustrated when I see that red coveralls badge, telling us we have less than 80% code coverage. We should really aim for >90%.

Describe the solution you'd like
More unit tests. This would also require a resolution of issue #22 .

Describe alternatives you've considered
Not doing that. But then we'll never get rid of the red box, will we?

Python 3.10 support

Is your feature request related to a problem? Please describe.
Python 3.10 is here. We should test if it works with flarestack. If so, we should add it to setup.py and github actions.

CI GitHub actions runs for ever

Describe the bug
When triggering the CI when pushing the unit tests run and finish with no errors.
Then the code somehow does not progress.

To Reproduce
Re-run e.g. this and see the code output.

Expected behavior
If unit tests finish, the action should progress.

Make contributing guide

Is your feature request related to a problem? Please describe.
I'm always frustrated when I realise there is no guide informing users how best to contribute,

Describe the solution you'd like
Creating a CONTRIBUTING.md file, along the lines of this, to guide users. Could detail rules for commiting, build bots, deploying releases, code coverage, travis etc. etc.

Describe alternatives you've considered
Not doing that. But then how will newbies know what to do?

Add `.gitignore`

The git-friendliness of the repository could be improved by implementing a .gitignore file.

Adding:

  • *.pyc
  • flarestack.egg-info
    gives quite a clean sheet.

Will submit a PR.

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    ๐Ÿ–– Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. ๐Ÿ“Š๐Ÿ“ˆ๐ŸŽ‰

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google โค๏ธ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.