cta-observatory / pyhessio

CTA Python wrapper for hessio event format that is used in output of simtel_array. [DEPRECATED see pyeventio instead]

pyhessio

CTA Python wrapper for hessio event format that is used in output of simtel_array.

• Code: https://github.com/cta-observatory/pyhessio
• Docs: https://cta-observatory.github.io/pyhessio/

This is a temporaly solution for testing ctapipe with real CTA MC data.

• 

Quick Start

Users of the Anaconda python distribution should follow the instructions for Anaconda python distribution.

Dependencies

python3
setuptools
numpy

build and install

Installation for an anaconda env

Binary installation:

conda install -c cta-observatory pyhessio

Building from source:

conda create -n cta python=3.5
source activate cta
git clone https://github.com/cta-observatory/pyhessio
cd pyhessio
python setup.py install

Datasets

To keep the pyhessio code repo small, we have decided to put the example files used for unit tests in a separate repo: https://github.com/cta-observatory/pyhessio-extra

git submodule init
git submodule update

For pyhessio developers

python setup.py develop

Using pyhessio

Warnings:

1. only ONE simtelarray file could be open at once.

2. This wrapper uses ugly global static variables for development simplicity reason, and because it is a temporally solution until CTA MC data format exists.

3. PROD3 MC use udge amount of memory, and memory is only free when close_file is executed. To force close_file execution, we force to use context manager to instantiate object:

with pyhessio.open('pyhessio-extra/datasets/gamma_test.simtel.gz') as f:
    f.fill_next_event()

pyhessio's Issues

Stochastic read errors on macos

Using the current release version, under certain circumstances, pyhessio reads wrong values, and that stochastically.

See for example this failing test on travis: https://travis-ci.org/cta-observatory/ctapipe/jobs/477729111

I could also reproduce this locally, running ctapipe test:

pytest ctapipe/io/tests/test_simteleventsource.py::test_compare_event_hessio_and_simtel

It fails at different points from time to time because pyhessio yields different values.

Return an array from get_mc_number_photon_electron

Can get_mc_number_photon_electron be changed to return a np array of the pe_count for the entire camera, removing the pixel_id argument?

Currently it does return an array, however this is an array of just the pixel_id requested.

I attempted to make the changes myself, but I'm not sure how to test those changes without them being integrated into the github first.

trouble using pyhessio

Hello,

this is very likely a trivial question: I've installed pyhessio as described in the README using Anaconda. Do I have to make additions to PYTHONPATH or LD_LIBRARY_PATH to use pyhessio? I am getting import errors when I try to use it in ctapipe ('ImportError: No module named 'hessio'').

Cheers.

Gernot

Indices of telescope dictionary for photo-electrons shifted by 1

I tried to compare the camera response to the number of photo-electrons from a noiseless shower sample I generated myself. I noticed that the enumeration of the telescope IDs between the two containers is shifted by 1: The photo-electron image for "telescope 1" actually belongs to telescope 2 (see attached plot).

Please keep in mind that only triggered telescopes are stored in the file so that this problem is not easily fixed down the analysis chain by simply offsetting one index.

e.g. If only telescope 6 would be hit, the photo-electrons of telescope 5 are saved (which are empty) but not the ones of telescope 6; this information is lost.

update to latest hessio library

The latest hessioxxx library is needed for reading the latest sims, so we should update the version inside pyhessio.

lib pyhessioc.cpython not found when current directory is pyhessio

jacquem>cd pyhessio
jacquem>python
Python 3.5.cd pyt0 |Anaconda 2.4.0 (64-bit)| (default, Oct 19 2015, 21:57:25)
[GCC 4.4.7 20120313 (Red Hat 4.4.7-1)] on linux
Type "help", "copyright", "credits" or "license" for more information.

import pyhessio
Traceback (most recent call last):
File "", line 1, in
File "/home/jacquem/workspace/CTA/pyhessio/pyhessio/init.py", line 27, in
lib = np.ctypeslib.load_library('pyhessioc', _path)
File "/home/jacquem/anaconda3/lib/python3.5/site-packages/numpy/ctypeslib.py", line 128, in load_library
raise OSError("no file with expected extension")
OSError: no file with expected extension

jacquem>cd /tmp/
jacquem>python
Python 3.5.0 |Anaconda 2.4.0 (64-bit)| (default, Oct 19 2015, 21:57:25)
[GCC 4.4.7 20120313 (Red Hat 4.4.7-1)] on linux
Type "help", "copyright", "credits" or "license" for more information.
import pyhessio

PyHESSIO module move_to_next_event not reading the correct event ID for triggered events

When calling the move_to_next_event module in order to generate the triggered event stream from sim_telarray output produced with its last version (2018/11/07), the event ID associated to each event is apparently not read but set to a sequential value up to the number of triggers. In other words, instead of reading e.g. this:

EVTNUM
1 2401
2 11100
3 13006
4 21910
5 22316
6 23405
...

I am reading this:

EVTNUM
1 1
2 2
3 3
4 4
5 5
6 6
…

Apparently, the Python function is not reading any triggered event ID (therefore leaving the values of col. EVTNUM to their initializations). I have already verified that the datum is not missing from the sim_telarray output thanks to the ROOT converter EventDisplay. Note that this problem does not hold when trying instead to read MC events through move_to_next_event(event_type = 2021), which are recorded with the correct event ID (i.e. evt_num = 100*n + p, n being the shower number and p the telescope random position).

Note that, in order to have PyHESSIO reading the output of last sim_telarray version, I was forced to update the HESSIOxxx folder with the current one distributed by Konrad Bernloehr along with the latest CoRSiKa+sim_tel distribution, otherwise not being able to do it. Could the reported issue derive from some kind of mismatch between PyHESSIO and the latest HESSIOxxx?

cannot get pyhessio version in standard way

Python packages should expose a __version__ attribute at the top-level of the package. This is needed to be able to check the version is correct at runtime. It seems to be missing from pyhessio:

>>> import pyhessio
>>> import ctapipe
>>> ctapipe.__version__
>>> pyhessio.__version__

AttributeError: module 'pyhessio' has no attribute '__version__'

Simply setting __version__ in __init__.py should work, though there are other mechanisms to get the version automatically (if in init, it needs to be updated by hand when the version is increased)

Number of photoelectrons at each pixel

Extract MCEvent.mc_pe_list[tel].pe_count[pix] into a mc-event object.

MC photo electron image is zero for all pixels and telescopes

When looking at the mc information read by HESSIOEventSource I realized that photo_electron_image is an array of zeros for all telescopes:

source = EventSourceFactory.produce(input_url=input_file,
                                    max_events=2)
for event in source:
    for tel_id in event.r0.tels_with_data:
        print("Sum of photo_electron_image: ", sum(event.mc.tel[tel_id].photo_electron_image))

results in Sum of photo_electron_image: 0 for all telescopes and events. Am I missing something here or is there a bug in get_mc_number_photon_electron?

I'm using pyhessio 2.1.1

Problems with installation

I just had problems installing pyhessio when creating the cta-dev environment:

______________________________________________________________ ERROR collecting ctapipe/io/tests/test_serializer.py _______________________________________________________________
ctapipe/io/tests/test_serializer.py:29: in <module>
    input_containers = generate_input_containers()
ctapipe/io/tests/test_serializer.py:22: in generate_input_containers
    gen = hessio_event_source(input_filename, max_events=3)
/lustre/fs19/group/cta/users/kpfrang/anaconda3/envs/cta-dev/lib/python3.6/site-packages/astropy/utils/decorators.py:120: in deprecated_func
    return func(*args, **kwargs)
ctapipe/io/hessio.py:45: in hessio_event_source
    reader = HESSIOEventSource(input_url=url, **kwargs)
ctapipe/io/hessioeventsource.py:26: in __init__
    import pyhessio
/lustre/fs19/group/cta/users/kpfrang/anaconda3/envs/cta-dev/lib/python3.6/site-packages/pyhessio/__init__.py:61: in <module>
    lib_close = np.ctypeslib.load_library('pyhessioc', _path)
/lustre/fs19/group/cta/users/kpfrang/anaconda3/envs/cta-dev/lib/python3.6/site-packages/numpy/ctypeslib.py:150: in load_library
    return ctypes.cdll[libpath]
/lustre/fs19/group/cta/users/kpfrang/anaconda3/envs/cta-dev/lib/python3.6/ctypes/__init__.py:423: in __getitem__
    return getattr(self, name)
/lustre/fs19/group/cta/users/kpfrang/anaconda3/envs/cta-dev/lib/python3.6/ctypes/__init__.py:418: in __getattr__
    dll = self._dlltype(name)
/lustre/fs19/group/cta/users/kpfrang/anaconda3/envs/cta-dev/lib/python3.6/ctypes/__init__.py:348: in __init__
    self._handle = _dlopen(self._name, mode)
E   OSError: /lib64/libc.so.6: version `GLIBC_2.14' not found (required by /lustre/fs19/group/cta/users/kpfrang/anaconda3/envs/cta-dev/lib/python3.6/site-packages/pyhessio/pyhessioc.cpython-36m-x86_64-linux-gnu.so)

The same error occurred when reinstalling it via

conda install -c cta-observatory pyhessio

Only as I build it from source python setup.py develop, it didn't throw me this error anymore and pyhessio seems to work fine now. Does someone know what's the problem here?

Calibration events in NSB only simulations skipped

When reading NSB simulation file it correctly recognizes that calibration events and prints RDLR: CALIBEVENT!. However, I cannot access these events which makes me think that they are skipped.

---

Example for gamma_20deg_180deg_run4___cta-prod3-demo_desert-2150m-Paranal-baseline-5x.simtel.gz

As you can see I don't have any event read with event_source for the NSB file, even though 100 events should be contained in the file (which equals the number for times RDLR: CALIBEVENT! is printed out).

Should they not be readable in exactly the same way as other simtel files as well?

Anaconda install not available for python 3.5

Hi all,

Currently, within the ctapipe "installation for developers", the version that is stated is python 3.5. But now I'm trying to install pyhessio using conda (which is also the method described) and it is not working.

>conda install pyhessio
Fetching package metadata ...........
Solving package specifications: .


PackageNotFoundError: Package not found: '' Package missing in current linux-64 channels: 
  - pyhessio

You can search for packages on anaconda.org with

    anaconda search -t conda pyhessio

I tested, and the installation works using python 3.4.5.

I know that I can install it via gitHub clone and "python setup.py install", but then documentation should be fixed.

So is there a problem with python 3.5? Or is this easily fixed?

Thanks in advance!

Tarek

second gain channel missing

For cameras with two gain channels (e.g. ASTRI) the second one is missing (all entries are zero):

In [1]: from ctapipe.io.hessio import hessio_event_source

In [2]: source = hessio_event_source("/home/ichanmich/Data/cta/ASTRI9/gamma/run1001.simtel.gz", max_events=10, allowe
   ...: d_tels=[10])

In [3]: for event in source:
   ...:     signal = event.dl0.tel[10].adc_sums
   ...:     print(signal)
   ...:     print()
   ...:     
WARNING: ErfaWarning: ERFA function "taiutc" yielded 1 of "dubious year (Note 4)" [astropy._erfa.core]
[[1214 1161 1226 ..., 1116 1210 1170]
 [   0    0    0 ...,    0    0    0]]

[[1054 1146 1115 ..., 1112 1127 1209]
 [   0    0    0 ...,    0    0    0]]

[[ 995 1244 1113 ..., 1151 1164 1181]
 [   0    0    0 ...,    0    0    0]]

I/O block extended by 173452 to 1173452 bytes
I/O block extended by 490012 to 1663464 bytes
[[1113 1294 1191 ..., 1110 1075 1155]
 [   0    0    0 ...,    0    0    0]]

[[1135 1054 1316 ..., 1040 1164 1011]
 [   0    0    0 ...,    0    0    0]]

[[ 984 1074 1018 ..., 1192 1173 1075]
 [   0    0    0 ...,    0    0    0]]

[[ 975 1179 1189 ..., 1157 1044 1323]
 [   0    0    0 ...,    0    0    0]]

[[1130 1081 1274 ..., 1031 1028 1023]
 [   0    0    0 ...,    0    0    0]]

[[1290  929 1046 ..., 1197 1145 1102]
 [   0    0    0 ...,    0    0    0]]

[[1157 1020 1167 ..., 1098 1196 1095]
 [   0    0    0 ...,    0    0    0]]

incorrect Pedestal values

Dear all,

we encountered a worrisome feature in the pedestal values of the ASTRI telescopes. The ASTRI-PMTs operate in a two-gain mode but the corresponding pedestals seem to be associated to consecutive PMTs instead of taking the two channels into account (please find plots and further explanation in the forwarded mail below).

We are now asking if this peculiar feature is taken into account in the simulation and read-out chain and how we have to treat these cameras -- or if there is indeed an issue somewhere along the chain that should be taken care of.

It appears that the signal from the two gain-channels can be read out without problem (see attached plot showing the photo-electrons vs. the measured ADC value);
but it seams that the pedestal values are

1. shuffled together from the two channels
2. missing a smearing to account for electronic noise.

On a related note we are wondering if there is already a recommendation when to switch between the different gain channels.
The first channel seems to be saturating at a PE level of about 80.

Best regards,
Tino and the CEA group

problem with installation

Hi,

we followed this few steps:

conda create -n cta python=3.5
source activate cta
git clone https://github.com/cta-observatory/pyhessio
cd pyhessio
python setup.py install

but get this:

/home/bigongiari/Software/miniconda3/envs/pyhessio/compiler_compat/ld: cannot find -lpthread
/home/bigongiari/Software/miniconda3/envs/pyhessio/compiler_compat/ld: cannot find -lc
collect2: error: ld returned 1 exit status
error: command 'gcc' failed with exit status 1

Read somewhere I could add a line to setup.py:

extra_link_args=['-L/usr/lib/x86_64-linux-gnu/']

Could this fix the problem?
Where exactly should I put that line in setup.py?

Thx

hessio AdcData.AdcData type is uint32_t

pyhessio self.lib.get_adc_sum.argtypes is declare with a ndpointer of int32 as third argument, but in hessio library AdcData.AdcData type is uint32_t.

So this third argument must be declare as a uint32.

Install problem with anaconda

Dear all,

I'm trying to install the pyhessio module to start working with ctapipe. I'm following these instructions (https://cta-observatory.github.io/ctapipe/getting_started/index.html):
git clone https://github.com/cta-observatory/pyhessio
conda build pyhessio
conda install --use-local pyhessio
The last command returns an error linked to conflict packages:

(cta)[julien@touques] ctasoft $ conda install --use-local pyhessio
Using Anaconda Cloud api site https://api.anaconda.org
Fetching package metadata: ......
Solving package specifications: .......
Error: Unsatisfiable package specifications.
Generating hint:
[ COMPLETE ]|#################################################################################################################################| 100%

Hint: the following packages conflict with each other:

• pyhessio
• python 3.5*

Use 'conda info pyhessio' etc. to see the dependencies for each package.

Do you have any idea how to fix this ? For informations, I'm using Mac OS X 10.11.3 and the latest version of anaconda (for python2.7 but I created a virtual environment as suggested in the previous link). You can find the log of the build command in the attached file log_build.txt.

Best regards,

Julien

Reading input cards from simtel files

Hello pyhessio,

We were wondering if pyhessio supports the reading of the history block from simtel files (-h option in, e.g., read_cta in hessio). We would like to add these infos as metadata to our data files for machine learning, mainly for provenance purposes. Our dumper code makes use of ctapipe & pyhessio.

Ideally, if this feature existed in pyhessio, it would be great if there was a function in ctapipe.io.hessio to retrieve these metadata (e.g. hessio_get_history), but this discussion will come later and will deserve a separate issue in the proper project, I guess...

Cheers,

Daniel

better file open/close functions

I would suggest renaming (or adding an alias):

file_open() should be open_file(), to follow naming conventions (start with verb).

There is already close_file(), for example.

Also, it would be good to provide a context manager interface for open_file():

with pyhessio.open_file(filename) as hessiofile:
     # do things

# file is automatically closed

see https://docs.python.org/3/library/contextlib.html

conda build pyhessio failed on OSX without XCode

Package: pyhessio-0.2-1
source tree in: /Applications/anaconda/conda-bld/work

• cmake .
  -DCMAKE_INSTALL_PREFIX=/Applications/anaconda/envs/_build_placehold_placehold_placehold_placehold_place
  -- The C compiler identification is AppleClang 7.0.0.7000176
  -- The CXX compiler identification is AppleClang 7.0.0.7000176
  CMake Error at
  /usr/local/Cellar/cmake/3.2.2/share/cmake/Modules/Platform/Darwin.cmake:76
  (message):
  CMAKE_OSX_DEPLOYMENT_TARGET is '10.6' but CMAKE_OSX_SYSROOT:

  ""

  is not set to a MacOSX SDK with a recognized version. Either set
  CMAKE_OSX_SYSROOT to a valid SDK or set CMAKE_OSX_DEPLOYMENT_TARGET to
  empty.
  Call Stack (most recent call first):

/usr/local/Cellar/cmake/3.2.2/share/cmake/Modules/CMakeSystemSpecificInformation.cmake:36
(include)
CMakeLists.txt:2 (project)

-- Configuring incomplete, errors occurred!
See also
"/Applications/anaconda/conda-bld/work/CMakeFiles/CMakeOutput.log".
Command failed: /bin/bash -x -e
/Users/alisonmitchell/ctapipe/pyhessio/build.sh

Use python logging in the c extension

This would make the output handling a lot better.

See here for how it can be done:
https://gist.github.com/hensing/0db3f8e3a99590006368

HessioExceptions should inherit from a base HessioException

This would enable to catch a generic hessio exception.

Closing files not working properly

Closing the files with pyhessio.close_file() seems not to work properly, as
when opening another file or the same file again, it segfaults.

Read pixel modules

It might be useful for ctapipe to have also the pixel modules. Those are available in sim_telarray files and are referred to as "drawers" in hessioxxx.

Access to sim_telarray histograms

In order to produce some of the high level performance plots with ctapipe, it is necessary to access some information sin the sim_telarray file stored only as histograms, such as the energy distribution of events simulated in a MC run. Is there a simple way to allow access to these histograms?

Shared Library Linking problem on OSX

(summarized and migrated this issue from the old repo)

On OSX when installing via conda, the package cannot be imported due to an incorrect _rpath_in pyhessioc.so or in libhessio.dylib. The problem is that libhessio.dylib cannot be found, though it is installed in anaconda/lib.

OSError: dlopen(/Users/x/anaconda/lib/python3.4/site-packages/pyhessio-0.2-py3.4-macosx-10.5-x86_64.egg/pyhessio/pyhessioc.so, 6):
 Library not loaded: @rpath/./libhessio.dylib
  Referenced from: /Users/x/anaconda/lib/python3.4/site-packages/pyhessio-0.2-py3.4-macosx-10.5-x86_64.egg/pyhessio/pyhessioc.so
  Reason: image not found

Apparently, anaconda handles it's virtual environments by updating the internal linking path of all shared libraries (see e.g. conda/conda-build#279).

Having an @rpath directive should work, but for some reason it does not here. This may be related to the extra ./ in the path, or the warning given during the build:

CMake Warning (dev):
  Policy CMP0042 is not set: MACOSX_RPATH is enabled by default.  Run "cmake
  --help-policy CMP0042" for policy details.  Use the cmake_policy command to
  set the policy and suppress this warning.

  MACOSX_RPATH is not specified for the following targets:

   hessio

only allow one file open at a time

pyhessio.file_open should throw an exception if a file is already open, to prevent code crashes when two functions try to open a file simultaneously.

add sphinx documentation

It would be good to document the pyhessio API using sphinx so the list of functions and descriptions appear on a web page.

The API (list of C and python functions and return values) is being proposed as a standard for CTA data readers/writers, so it would be very useful to have documentation.

In principle that just means making sure there are docstrings everywhere and running sphinx-quickstart to setup sphinx.

get_calibration: low gain data and high gain data are interlaced

get_calibration returns numpy array (num_gain dimention), where low gain data and high gain data are interlaced

Telescope pointing az and ze

What functions shall I use to read telescope azimuth and zenith?
I found get_azimuth_raw and get_altitude_raw that seem suitable, but I'm not sure they return correct results, can somebody confirm?

Memory leak when open/read/close file

After opening a file, reading event inside and closing it, a memory leak appears.