gammapy / gammapy-benchmarks

Performance benchmarks for Gammapy

Gammapy

A Python Package for Gamma-ray Astronomy.

Gammapy is an open-source Python package for gamma-ray astronomy built on Numpy, Scipy and Astropy. It is used as core library for the Science Analysis tools of the Cherenkov Telescope Array (CTA), recommended by the H.E.S.S. collaboration to be used for Science publications, and is already widely used in the analysis of existing gamma-ray instruments, such as MAGIC, VERITAS and HAWC.

• Webpage: https://gammapy.org
• Documentation: https://docs.gammapy.org/
• Code: https://github.com/gammapy/gammapy

Contributing Code, Documentation, or Feedback

The Gammapy Project is made both by and for its users, so we welcome and encourage contributions of many kinds. Our goal is to keep this a positive, inclusive, successful, and growing community by abiding with the Gammapy Community Code of Conduct.

The Gammapy project uses a mechanism known as a Developer Certificate of Origin (DCO). The DCO is a binding statement that asserts that you are the creator of your contribution, and that you wish to allow Gammapy to use your work to cite you as contributor. More detailed information on contributing to the project or submitting feedback can be found on the Contributing page.

Licence

Gammapy is licensed under a 3-clause BSD style license - see the LICENSE.rst file.

Supporting the project

The Gammapy project is not sponsored and the development is made by the staff of the institutes supporting the project over their research time. Any contribution is then encouraged, as punctual or regular contributor.

Status shields

(mostly useful for developers)

• 
• 
• 

gammapy-benchmarks's Issues

Location of provenance extraction

We discussed this morning about having some provenance information for each run of the validation as is currently done for benchmark.
I assume that this is something that is the responsibility of the general make.py, not of each individual validation project (fermi, CTA 1DC, joint-crab etc). So it does not need to be reimplemented in each of them. Right @Bultako ?

Issues with v1.2 validation

In the preparation of Gammapy v1.2 release, an issue appeared in the validation hess-dl3-dr1. The memory usage goes to very large values and the action hangs.
The issue appears during the msh1552 fit.

benchmarks not running with gammapy-0.18.2

my setup:

curl -O https://gammapy.org/download/install/gammapy-0.18.2-environment.yml
conda env create -f gammapy-0.18.2-environment.yml
conda activate gammapy-0.18.2
pip install psrecord
gammapy download datasets

gammapy info

System:

	python_executable      : /home/hpc/unrz/unrz009h/.conda/envs/gammapy-0.18.2/bin/python 
	python_version         : 3.7.0      
	machine                : x86_64     
	system                 : Linux      


Gammapy package:

	version                : 0.18.2     
	path                   : /home/hpc/unrz/unrz009h/.conda/envs/gammapy-0.18.2/lib/python3.7/site-packages/gammapy 


Other packages:

	numpy                  : 1.19.4     
	scipy                  : 1.4.1      
	astropy                : 4.1        
	regions                : 0.4        
	click                  : 7.0        
	yaml                   : 5.2        
	IPython                : 7.10.1     
	jupyterlab             : 2.0.1      
	matplotlib             : 3.2.1      
	pandas                 : 0.25.3     
	healpy                 : 1.13.0     
	iminuit                : 1.3.8      
	sherpa                 : 4.12.0     
	naima                  : not installed 
	emcee                  : 2.2.1      
	corner                 : 2.0.1      
	parfive                : 1.0.2      


Gammapy environment variables:

	GAMMAPY_DATA           : /home/vault/unrz/unrz009h/gammapy-datasets 

io is working

INFO:__main__:Executing command: python /home/vault/unrz/unrz009h/gammapy-benchmarks-master/benchmarks/io.py
io.py:46: Running io.py with N_OBS = 10
io.py:47: cwd = /tmp/tmpbsabd9cb
Process finished (6.28 seconds)
INFO:__main__:Writing results/io/results.txt
INFO:__main__:Writing results/io/results.png
INFO:__main__:Writing results/io/provenance.yaml
INFO:__main__:Writing results/results.yaml

analysis_3d not working

python make.py run-benchmark analysis_3d
INFO:__main__:Executing command: python /home/vault/unrz/unrz009h/gammapy-benchmarks-master/benchmarks/analysis_3d.py
Invalid unit found in background table! Assuming (s-1 MeV-1 sr-1)
Invalid unit found in background table! Assuming (s-1 MeV-1 sr-1)
Invalid unit found in background table! Assuming (s-1 MeV-1 sr-1)
/home/hpc/unrz/unrz009h/.conda/envs/gammapy-0.18.2/lib/python3.7/site-packages/astropy/units/quantity.py:477: RuntimeWarning: invalid value encountered in true_divide
  result = super().__array_ufunc__(function, method, *arrays, **kwargs)
  result = super().__array_ufunc__(function, method, *arrays, **kwargs)
Traceback (most recent call last):
  File "/home/vault/unrz/unrz009h/gammapy-benchmarks-master/benchmarks/analysis_3d.py", line 118, in <module>
    run_benchmark()
  File "/home/vault/unrz/unrz009h/gammapy-benchmarks-master/benchmarks/analysis_3d.py", line 107, in run_benchmark
    data_fit(stacked)
  File "/home/vault/unrz/unrz009h/gammapy-benchmarks-master/benchmarks/analysis_3d.py", line 79, in data_fit
    fit = Fit(optimize_opts={"print_level": 1})
TypeError: __init__() got an unexpected keyword argument 'optimize_opts'
Process finished (60.44 seconds)
Traceback (most recent call last):
  File "make.py", line 146, in <module>
    cli()
  File "/home/hpc/unrz/unrz009h/.conda/envs/gammapy-0.18.2/lib/python3.7/site-packages/click/core.py", line 764, in __call__
    return self.main(*args, **kwargs)
  File "/home/hpc/unrz/unrz009h/.conda/envs/gammapy-0.18.2/lib/python3.7/site-packages/click/core.py", line 717, in main
    rv = self.invoke(ctx)
  File "/home/hpc/unrz/unrz009h/.conda/envs/gammapy-0.18.2/lib/python3.7/site-packages/click/core.py", line 1137, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "/home/hpc/unrz/unrz009h/.conda/envs/gammapy-0.18.2/lib/python3.7/site-packages/click/core.py", line 956, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/home/hpc/unrz/unrz009h/.conda/envs/gammapy-0.18.2/lib/python3.7/site-packages/click/core.py", line 555, in invoke
    return callback(*args, **kwargs)
  File "make.py", line 108, in run_benchmarks
    **MONITOR_OPTIONS,
  File "make.py", line 142, in run_single_benchmark
    shutil.copyfile(Path(path) / "bench.yaml", f"results/{benchmark}/bench.yaml")
  File "/home/hpc/unrz/unrz009h/.conda/envs/gammapy-0.18.2/lib/python3.7/shutil.py", line 120, in copyfile
    with open(src, 'rb') as fsrc:
FileNotFoundError: [Errno 2] No such file or directory: '/tmp/tmpchtlunfm/bench.yaml'

analysis_3d_joint

python make.py run-benchmark analysis_3d_joint
INFO:__main__:Executing command: python /home/vault/unrz/unrz009h/gammapy-benchmarks-master/benchmarks/analysis_3d_joint.py
Invalid unit found in background table! Assuming (s-1 MeV-1 sr-1)
Traceback (most recent call last):
  File "/home/vault/unrz/unrz009h/gammapy-benchmarks-master/benchmarks/analysis_3d_joint.py", line 125, in <module>
    run_benchmark()
  File "/home/vault/unrz/unrz009h/gammapy-benchmarks-master/benchmarks/analysis_3d_joint.py", line 114, in run_benchmark
    data_fit(datasets)
  File "/home/vault/unrz/unrz009h/gammapy-benchmarks-master/benchmarks/analysis_3d_joint.py", line 86, in data_fit
    fit = Fit()
TypeError: __init__() missing 1 required positional argument: 'datasets'
Process finished (128.22 seconds)
Traceback (most recent call last):
  File "make.py", line 146, in <module>
    cli()
  File "/home/hpc/unrz/unrz009h/.conda/envs/gammapy-0.18.2/lib/python3.7/site-packages/click/core.py", line 764, in __call__
    return self.main(*args, **kwargs)
  File "/home/hpc/unrz/unrz009h/.conda/envs/gammapy-0.18.2/lib/python3.7/site-packages/click/core.py", line 717, in main
    rv = self.invoke(ctx)
  File "/home/hpc/unrz/unrz009h/.conda/envs/gammapy-0.18.2/lib/python3.7/site-packages/click/core.py", line 1137, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "/home/hpc/unrz/unrz009h/.conda/envs/gammapy-0.18.2/lib/python3.7/site-packages/click/core.py", line 956, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/home/hpc/unrz/unrz009h/.conda/envs/gammapy-0.18.2/lib/python3.7/site-packages/click/core.py", line 555, in invoke
    return callback(*args, **kwargs)
  File "make.py", line 108, in run_benchmarks
    **MONITOR_OPTIONS,
  File "make.py", line 142, in run_single_benchmark
    shutil.copyfile(Path(path) / "bench.yaml", f"results/{benchmark}/bench.yaml")
  File "/home/hpc/unrz/unrz009h/.conda/envs/gammapy-0.18.2/lib/python3.7/shutil.py", line 120, in copyfile
    with open(src, 'rb') as fsrc:
FileNotFoundError: [Errno 2] No such file or directory: '/tmp/tmpokzbe12p/bench.yaml'

lightcurve_1d

python make.py run-benchmark lightcurve_1d
INFO:__main__:Executing command: python /home/vault/unrz/unrz009h/gammapy-benchmarks-master/benchmarks/lightcurve_1d.py
Invalid unit found in background table! Assuming (s-1 MeV-1 sr-1)
/home/hpc/unrz/unrz009h/.conda/envs/gammapy-0.18.2/lib/python3.7/site-packages/astropy/units/quantity.py:477: RuntimeWarning: overflow encountered in exp
  result = super().__array_ufunc__(function, method, *arrays, **kwargs)
/home/hpc/unrz/unrz009h/.conda/envs/gammapy-0.18.2/lib/python3.7/site-packages/astropy/units/quantity.py:477: RuntimeWarning: invalid value encountered in subtract
  result = super().__array_ufunc__(function, method, *arrays, **kwargs)
Traceback (most recent call last):
  File "/home/vault/unrz/unrz009h/gammapy-benchmarks-master/benchmarks/lightcurve_1d.py", line 136, in <module>
    run_benchmark()
  File "/home/vault/unrz/unrz009h/gammapy-benchmarks-master/benchmarks/lightcurve_1d.py", line 120, in run_benchmark
    datasets = simulate()
  File "/home/vault/unrz/unrz009h/gammapy-benchmarks-master/benchmarks/lightcurve_1d.py", line 72, in simulate
    name=f"dataset_{i}",
TypeError: create() got an unexpected keyword argument 'geom'
Process finished (4.16 seconds)
Traceback (most recent call last):
  File "make.py", line 146, in <module>
    cli()
  File "/home/hpc/unrz/unrz009h/.conda/envs/gammapy-0.18.2/lib/python3.7/site-packages/click/core.py", line 764, in __call__
    return self.main(*args, **kwargs)
  File "/home/hpc/unrz/unrz009h/.conda/envs/gammapy-0.18.2/lib/python3.7/site-packages/click/core.py", line 717, in main
    rv = self.invoke(ctx)
  File "/home/hpc/unrz/unrz009h/.conda/envs/gammapy-0.18.2/lib/python3.7/site-packages/click/core.py", line 1137, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "/home/hpc/unrz/unrz009h/.conda/envs/gammapy-0.18.2/lib/python3.7/site-packages/click/core.py", line 956, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/home/hpc/unrz/unrz009h/.conda/envs/gammapy-0.18.2/lib/python3.7/site-packages/click/core.py", line 555, in invoke
    return callback(*args, **kwargs)
  File "make.py", line 108, in run_benchmarks
    **MONITOR_OPTIONS,
  File "make.py", line 142, in run_single_benchmark
    shutil.copyfile(Path(path) / "bench.yaml", f"results/{benchmark}/bench.yaml")
  File "/home/hpc/unrz/unrz009h/.conda/envs/gammapy-0.18.2/lib/python3.7/shutil.py", line 120, in copyfile
    with open(src, 'rb') as fsrc:
FileNotFoundError: [Errno 2] No such file or directory: '/scratch/1545732.eadm/tmp4rd9g4ed/bench.yaml'

lightcurve_3d

python make.py run-benchmark lightcurve_3d
INFO:__main__:Executing command: python /home/vault/unrz/unrz009h/gammapy-benchmarks-master/benchmarks/lightcurve_3d.py
Invalid unit found in background table! Assuming (s-1 MeV-1 sr-1)
/home/hpc/unrz/unrz009h/.conda/envs/gammapy-0.18.2/lib/python3.7/site-packages/astropy/units/quantity.py:477: RuntimeWarning: overflow encountered in exp
  result = super().__array_ufunc__(function, method, *arrays, **kwargs)
/home/hpc/unrz/unrz009h/.conda/envs/gammapy-0.18.2/lib/python3.7/site-packages/astropy/units/quantity.py:477: RuntimeWarning: invalid value encountered in subtract
  result = super().__array_ufunc__(function, method, *arrays, **kwargs)
Traceback (most recent call last):
  File "/home/vault/unrz/unrz009h/gammapy-benchmarks-master/benchmarks/lightcurve_3d.py", line 157, in <module>
    run_benchmark()
  File "/home/vault/unrz/unrz009h/gammapy-benchmarks-master/benchmarks/lightcurve_3d.py", line 145, in run_benchmark
    get_lc(datasets)
  File "/home/vault/unrz/unrz009h/gammapy-benchmarks-master/benchmarks/lightcurve_3d.py", line 105, in get_lc
    print(lc.to_table(format="lightcurve", sed_type="flux")["flux"])
AttributeError: 'LightCurve' object has no attribute 'to_table'
Process finished (134.81 seconds)
Traceback (most recent call last):
  File "make.py", line 146, in <module>
    cli()
  File "/home/hpc/unrz/unrz009h/.conda/envs/gammapy-0.18.2/lib/python3.7/site-packages/click/core.py", line 764, in __call__
    return self.main(*args, **kwargs)
  File "/home/hpc/unrz/unrz009h/.conda/envs/gammapy-0.18.2/lib/python3.7/site-packages/click/core.py", line 717, in main
    rv = self.invoke(ctx)
  File "/home/hpc/unrz/unrz009h/.conda/envs/gammapy-0.18.2/lib/python3.7/site-packages/click/core.py", line 1137, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "/home/hpc/unrz/unrz009h/.conda/envs/gammapy-0.18.2/lib/python3.7/site-packages/click/core.py", line 956, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/home/hpc/unrz/unrz009h/.conda/envs/gammapy-0.18.2/lib/python3.7/site-packages/click/core.py", line 555, in invoke
    return callback(*args, **kwargs)
  File "make.py", line 108, in run_benchmarks
    **MONITOR_OPTIONS,
  File "make.py", line 142, in run_single_benchmark
    shutil.copyfile(Path(path) / "bench.yaml", f"results/{benchmark}/bench.yaml")
  File "/home/hpc/unrz/unrz009h/.conda/envs/gammapy-0.18.2/lib/python3.7/shutil.py", line 120, in copyfile
    with open(src, 'rb') as fsrc:
FileNotFoundError: [Errno 2] No such file or directory: '/tmp/tmpzfnuy02c/bench.yaml'

spectrum_1d

python make.py run-benchmark spectrum_1d
INFO:__main__:Executing command: python /home/vault/unrz/unrz009h/gammapy-benchmarks-master/benchmarks/spectrum_1d.py
Traceback (most recent call last):
  File "/home/vault/unrz/unrz009h/gammapy-benchmarks-master/benchmarks/spectrum_1d.py", line 143, in <module>
    run_benchmark()
  File "/home/vault/unrz/unrz009h/gammapy-benchmarks-master/benchmarks/spectrum_1d.py", line 120, in run_benchmark
    stacked = data_prep()
  File "/home/vault/unrz/unrz009h/gammapy-benchmarks-master/benchmarks/spectrum_1d.py", line 55, in data_prep
    stacked = SpectrumDatasetOnOff.create(geom=geom, energy_axis_true=e_true, name="stacked")
TypeError: create() got an unexpected keyword argument 'geom'
Process finished (4.18 seconds)
Traceback (most recent call last):
  File "make.py", line 146, in <module>
    cli()
  File "/home/hpc/unrz/unrz009h/.conda/envs/gammapy-0.18.2/lib/python3.7/site-packages/click/core.py", line 764, in __call__
    return self.main(*args, **kwargs)
  File "/home/hpc/unrz/unrz009h/.conda/envs/gammapy-0.18.2/lib/python3.7/site-packages/click/core.py", line 717, in main
    rv = self.invoke(ctx)
  File "/home/hpc/unrz/unrz009h/.conda/envs/gammapy-0.18.2/lib/python3.7/site-packages/click/core.py", line 1137, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "/home/hpc/unrz/unrz009h/.conda/envs/gammapy-0.18.2/lib/python3.7/site-packages/click/core.py", line 956, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/home/hpc/unrz/unrz009h/.conda/envs/gammapy-0.18.2/lib/python3.7/site-packages/click/core.py", line 555, in invoke
    return callback(*args, **kwargs)
  File "make.py", line 108, in run_benchmarks
    **MONITOR_OPTIONS,
  File "make.py", line 142, in run_single_benchmark
    shutil.copyfile(Path(path) / "bench.yaml", f"results/{benchmark}/bench.yaml")
  File "/home/hpc/unrz/unrz009h/.conda/envs/gammapy-0.18.2/lib/python3.7/shutil.py", line 120, in copyfile
    with open(src, 'rb') as fsrc:
FileNotFoundError: [Errno 2] No such file or directory: '/tmp/tmp3m7smbxv/bench.yaml'

spectrum_1d_joint

python make.py run-benchmark spectrum_1d_joint
INFO:__main__:Executing command: python /home/vault/unrz/unrz009h/gammapy-benchmarks-master/benchmarks/spectrum_1d_joint.py
Traceback (most recent call last):
  File "/home/vault/unrz/unrz009h/gammapy-benchmarks-master/benchmarks/spectrum_1d_joint.py", line 140, in <module>
    run_benchmark()
  File "/home/vault/unrz/unrz009h/gammapy-benchmarks-master/benchmarks/spectrum_1d_joint.py", line 117, in run_benchmark
    datasets = data_prep()
  File "/home/vault/unrz/unrz009h/gammapy-benchmarks-master/benchmarks/spectrum_1d_joint.py", line 54, in data_prep
    empty = SpectrumDatasetOnOff.create(geom, energy_axis_true=e_true)
TypeError: create() got an unexpected keyword argument 'energy_axis_true'
Process finished (3.65 seconds)
Traceback (most recent call last):
  File "make.py", line 146, in <module>
    cli()
  File "/home/hpc/unrz/unrz009h/.conda/envs/gammapy-0.18.2/lib/python3.7/site-packages/click/core.py", line 764, in __call__
    return self.main(*args, **kwargs)
  File "/home/hpc/unrz/unrz009h/.conda/envs/gammapy-0.18.2/lib/python3.7/site-packages/click/core.py", line 717, in main
    rv = self.invoke(ctx)
  File "/home/hpc/unrz/unrz009h/.conda/envs/gammapy-0.18.2/lib/python3.7/site-packages/click/core.py", line 1137, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "/home/hpc/unrz/unrz009h/.conda/envs/gammapy-0.18.2/lib/python3.7/site-packages/click/core.py", line 956, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/home/hpc/unrz/unrz009h/.conda/envs/gammapy-0.18.2/lib/python3.7/site-packages/click/core.py", line 555, in invoke
    return callback(*args, **kwargs)
  File "make.py", line 108, in run_benchmarks
    **MONITOR_OPTIONS,
  File "make.py", line 142, in run_single_benchmark
    shutil.copyfile(Path(path) / "bench.yaml", f"results/{benchmark}/bench.yaml")
  File "/home/hpc/unrz/unrz009h/.conda/envs/gammapy-0.18.2/lib/python3.7/shutil.py", line 120, in copyfile
    with open(src, 'rb') as fsrc:
FileNotFoundError: [Errno 2] No such file or directory: '/tmp/tmpo4w58b54/bench.yaml'

tsmap_estimator

python make.py run-benchmark tsmap_estimator
INFO:__main__:Executing command: python /home/vault/unrz/unrz009h/gammapy-benchmarks-master/benchmarks/tsmap_estimator.py
WARNING: FITSFixedWarning: 'datfix' made the change 'Set DATEREF to '2001-01-01T00:01:04.184' from MJDREF.
Set MJD-OBS to 54682.655283 from DATE-OBS.
Set MJD-END to 57236.967546 from DATE-END'. [astropy.wcs.wcs]
Traceback (most recent call last):
  File "/home/vault/unrz/unrz009h/gammapy-benchmarks-master/benchmarks/tsmap_estimator.py", line 89, in <module>
    run_benchmark()
  File "/home/vault/unrz/unrz009h/gammapy-benchmarks-master/benchmarks/tsmap_estimator.py", line 72, in run_benchmark
    data = data_prep()
  File "/home/vault/unrz/unrz009h/gammapy-benchmarks-master/benchmarks/tsmap_estimator.py", line 30, in data_prep
    "$GAMMAPY_DATA/fermi-3fhl-gc/fermi-3fhl-gc-psf-cube.fits.gz", format="gtpsf"
TypeError: read() got an unexpected keyword argument 'format'
Process finished (15.14 seconds)
Traceback (most recent call last):
  File "make.py", line 146, in <module>
    cli()
  File "/home/hpc/unrz/unrz009h/.conda/envs/gammapy-0.18.2/lib/python3.7/site-packages/click/core.py", line 764, in __call__
    return self.main(*args, **kwargs)
  File "/home/hpc/unrz/unrz009h/.conda/envs/gammapy-0.18.2/lib/python3.7/site-packages/click/core.py", line 717, in main
    rv = self.invoke(ctx)
  File "/home/hpc/unrz/unrz009h/.conda/envs/gammapy-0.18.2/lib/python3.7/site-packages/click/core.py", line 1137, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "/home/hpc/unrz/unrz009h/.conda/envs/gammapy-0.18.2/lib/python3.7/site-packages/click/core.py", line 956, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/home/hpc/unrz/unrz009h/.conda/envs/gammapy-0.18.2/lib/python3.7/site-packages/click/core.py", line 555, in invoke
    return callback(*args, **kwargs)
  File "make.py", line 108, in run_benchmarks
    **MONITOR_OPTIONS,
  File "make.py", line 142, in run_single_benchmark
    shutil.copyfile(Path(path) / "bench.yaml", f"results/{benchmark}/bench.yaml")
  File "/home/hpc/unrz/unrz009h/.conda/envs/gammapy-0.18.2/lib/python3.7/shutil.py", line 120, in copyfile
    with open(src, 'rb') as fsrc:
FileNotFoundError: [Errno 2] No such file or directory: '/tmp/tmpqv60pod_/bench.yaml'

ring_background_estimator

python make.py run-benchmark ring_background_estimator
INFO:__main__:Executing command: python /home/vault/unrz/unrz009h/gammapy-benchmarks-master/benchmarks/ring_background_estimator.py
Traceback (most recent call last):
  File "/home/vault/unrz/unrz009h/gammapy-benchmarks-master/benchmarks/ring_background_estimator.py", line 139, in <module>
    run_benchmark()
  File "/home/vault/unrz/unrz009h/gammapy-benchmarks-master/benchmarks/ring_background_estimator.py", line 120, in run_benchmark
    analysis = data_prep()
  File "/home/vault/unrz/unrz009h/gammapy-benchmarks-master/benchmarks/ring_background_estimator.py", line 33, in data_prep
    config.datasets.geom.wcs.width = {"width": "3 deg", "height": "3 deg"}
  File "pydantic/main.py", line 281, in pydantic.main.BaseModel.__setattr__
ValueError: "WcsConfig" object has no field "width"
Process finished (4.17 seconds)
Traceback (most recent call last):
  File "make.py", line 146, in <module>
    cli()
  File "/home/hpc/unrz/unrz009h/.conda/envs/gammapy-0.18.2/lib/python3.7/site-packages/click/core.py", line 764, in __call__
    return self.main(*args, **kwargs)
  File "/home/hpc/unrz/unrz009h/.conda/envs/gammapy-0.18.2/lib/python3.7/site-packages/click/core.py", line 717, in main
    rv = self.invoke(ctx)
  File "/home/hpc/unrz/unrz009h/.conda/envs/gammapy-0.18.2/lib/python3.7/site-packages/click/core.py", line 1137, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "/home/hpc/unrz/unrz009h/.conda/envs/gammapy-0.18.2/lib/python3.7/site-packages/click/core.py", line 956, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/home/hpc/unrz/unrz009h/.conda/envs/gammapy-0.18.2/lib/python3.7/site-packages/click/core.py", line 555, in invoke
    return callback(*args, **kwargs)
  File "make.py", line 108, in run_benchmarks
    **MONITOR_OPTIONS,
  File "make.py", line 142, in run_single_benchmark
    shutil.copyfile(Path(path) / "bench.yaml", f"results/{benchmark}/bench.yaml")
  File "/home/hpc/unrz/unrz009h/.conda/envs/gammapy-0.18.2/lib/python3.7/shutil.py", line 120, in copyfile
    with open(src, 'rb') as fsrc:
FileNotFoundError: [Errno 2] No such file or directory: '/tmp/tmpffh2wrnd/bench.yaml'

Any advice on how to fix the issue?

Better tracking of benchmark performances

The current output of benchmarks allows to follow to total time and memory usage over time, but does not really indicate the evolution of sub-part of the benchmark scripts.
In some cases, this might completely hide some trend. For instance, the spectrum_1d CPU time is dominated by the data reduction step, but the fit or flux points parts can evolve very significantly.

We need a system able to track the various sub-tasks independently and able to provide a easy reading of results.

Additionally we might want to add some profiling information in order to better track performances degradations.

We have discussed using asv in the past. We might reconsider it.

@QRemy has already tested some profiling solutions and might want to comment. Opinions @adonath , @Bultako , @bkhelifi ?

Update validation for v1.0rc release

This is a reminder issue to update validation scripts before v1.0rc.

Add validation for RADMAX_2D MAGIC data

With the addition of energy dependent ON region, this new spectral extraction technique must be validated. The two MAGIC Crab runs can be used to do that.

Use separate true energy axis for 3D benchmarks

I noticed that we don't specify a separate true energy axis e.g. here https://github.com/gammapy/gammapy-benchmarks/blob/master/benchmarks/analysis_3d_joint.py#L42. To get a more realistic analysis scenario and be able to assess changes like gammapy/gammapy#3349 we should add this.

Store analysis results / debug info for benchmarks

To be able to compare benchmarks we should make sure the analysis results stay the same. E.g. to understand whether a change in performance comes from improved code or just a different number of iterations for a fit. String the results in YAML somewhere and just compare visually with git diff should be fine for now.