mnovak42 / g4hepem

Geant4 EM physics simulation R&D project looking for solutions to reduce the computing performance bottleneck experienced by HEP detector simulation applications.

The G4HepEm R&D Project

The G4HepEm R&D project was initiated by the Electromagnetic Physics Working Group of the Geant4 collaboration as part of looking for solutions to reduce the computing performance bottleneck experienced by the High Energy Physics (HEP) detector simulation applications.

Description

The project (initially) targets the most performance critical part of the HEP detector simulation applications, i.e. the EM shower generation covering and particle transport. By

• starting from the EM physics modelling part, the project isolates and extracts all the data and functionalities required for EM shower simulation in HEP detectors in a clear, compact and well documented form

• investigates the possible computing performance benefits of replacing the current general particle transport simulation stepping-loop of Geant4 by alternatives, highly specialised for particle types and tailored for HEP detector simulation applications

• by providing a clear, compact and well documented environment for EM shower generation (already connected to Geant4 applications), the project also provides an excellent domain for further related R&D activities.

• especially, the projects facilitates R&D activities targeting EM shower simulation on GPU by providing:

  • functionalities of making all required physics data available on the main device memory, together with example kernels for their run-time usage.

  • isolated, self-contained (i.e. "single function") implementations of both the physics related parts of the stepping loop and all required physics interactions

All details can be found in the documentation.

Requirements

G4HepEm is one of the ongoing developments, carried out within the EM physics working group of the Geant4 collaboration. It has been made available in order to facilitate and catalyse correlated R&D activities by providing and sharing the related expertise and specific knowledge. Therefore, G4HepEm is tightly connected to and depends on the Geant4 simulation toolkit.

The only requirement of G4HepEm is a recent Geant4 version to be available on the system. The recommended version, Geant4-10.7.p01 is available at the corresponding Geant4 Download area. All information regarding the Prerequisites of Geant4 as well as the detailed Geant4 Installation Instructions can be found in the related sections of the Geant4 Installation Guide.

Quick start

More detailed instructions can be found in the Build and Install section of the documentation.

It is assumed in the followings, that the required version of the Geant4 toolkit is installed on the system and the corresponding Geant4Config.cmake CMake configuration file is located under the G4_INSTALL directory. Then building and installing G4HepEm (under the G4HepEm_INSTALL directory) can be done by simple:

$ git clone https://github.com/mnovak42/g4hepem.git    
$ cd g4hepem/
$ mkdir build
$ cd build/
$ cmake ../ -DGeant4_DIR=G4_INSTALL -DCMAKE_INSTALL_PREFIX=G4HepEm_INSTALL
$ make install

Example: build and execute an example application

After building and installing G4HepEm under the G4HepEm_INSTALL directory, the g4hepem/apps/examples/TestEm3 (general) simplified sampling calorimeter example application can be built and executed as:

$ cd g4hepem/apps/examples/TestEm3/
$ mkdir build
$ cd build/
$ cmake ../ -DGeant4_DIR=G4_INSTALL -DG4HepEm_DIR=G4HepEm_INSTALL/lib/cmake/G4HepEm/  
$ make
$ ./TestEm3 -m ../ATLASbar.mac

Execute the application as ./TestEm3 --help for more information and see the g4hepem/apps/examples/TestEm3/ATLASbar.mac example input macro file for more details.

g4hepem's Issues

Persisting G4HepEmParameters together with G4HepEmData

As discussed in meetings and in apt-sim/AdePT#70, we need to "connect up" Geant4 and AdePT/Vecgeom materials/solids/physics. The G4HepEmData and GDML sides are taken care of or in progress, one thing I noticed as missing though is G4HepEmParameters. This is used both for data table init and for runtime calculations, so use of G4HepEmData will require the corresponding G4HepEmParameters. In the hybrid G4+GPU case that's fine, but for the split serialize/deserialize use case, we should also serialize G4HepEmParameters.

There are several ways to achieve this, the neatest I've seen being @hahnjo's creation of a G4HepEmState struct

struct G4HepEmState {
  G4HepEmData data;
  G4HepEmParameters parameters;
};

which would be easy to persist and has the required consistency. Should we introduce this type in G4HepEmData and the corresponding (de)serialization?

How to cite this project?

I would like to cite this project in an upcoming publication. So far I have put together this bibtex entry:

@misc{G4HepEm,
	author = {Mihaly Novak and Jonas Hahnfeld and Ben Morgan and Stephan Hageböck},
	title = {The G4HepEm R\&D Project},
	year = {2022},
	publisher = {GitHub},
	howpublished = {\url{https://github.com/mnovak42/g4hepem}}
}

Could you please tell me if this is appropriate for you?

In the long run, it may be beneficial to add a CITATION.cff file to your repository, so people can find this information faster.

Thank you!

license

@mnovak42 I think it would be good to add a license to the project to avoid any confusion...

Originally posted by @hahnjo in #8 (comment)

High-res logos

Hi!
I would like to place the g4hepem logo into a presentation I want to give at ACAT22. Do you have higher-resolution logos than the ones in https://github.com/mnovak42/g4hepem/tree/master/docs/source/logos?
Thx!

Formats for persisting G4HepEmData

Hi @mnovak42, before I fully start to implement the persistency for G4HepEmData and the data it holds, do you have any preferences/opinions on what format(s) to use? The data is pretty straightforward, so that helps, and of course we would keep the de/serialization code decoupled from the core G4HepEmData. There are a variety of options I can think of in rough order of increasing dependency:

• Plaintext (but at the cost of writing an deserialisation parser)
• JSON (using nlohmann_json or similar)
• HDF5
• ROOT

I'm tempted to start with plaintext/json just to confirm data I/O and understand the structure, but if there are strong objections to this I can start with the preferred system!

Incorrect dynamic array sizes for some data structures after construction

In testing serialization/deserialization reads in the same data that was written, I think I've tripped across a bug in how a few sizes for dynamic arrays are set in G4HepEmElectronData and G4HepEmGammaData. In the cases I've found, the symptom is that the data member giving the size of the array is 0, but the pointer to the array is not nullptr. Consequently nothing is serialised for that array (I'll add additional error checking for that case!) and so round-tripping to/from file fails comparison.

The affected (size, pointer) pairs are:

• G4HepEmGammaData
  • fElemSelectorConvNumData, fElemSelectorConvData
• G4HepEmElectronData
  • fElemSelectorIoniNumData, fElemSelectorIoniData
  • fElemSelectorBremSBNumData, fElemSelectorBremSBData
  • fElemSelectorBremRBNumData, fElemSelectorBremRBData

This doesn't appear to be related to the changes in #39 as the same behaviour is observed both with and without those changes. I'm happy to do further checks and discuss/submit fixes, but wanted to report/discuss first as whilst the places where the sizes/arrays are handled are pretty isolated, the code around them touches a lot of data so the more eyes/advice the better!

Checking required optimization and other flags

Briefly discussed offline, so just following up here with some investigations. The main issue is that @mnovak42 found some performance degradation in TestEm3 after some of the recent CMake changes. We suspected a missing optimization flag or similar, but needed some further checks

Checking the compile of the current master of g4hepem, I've confirmed the optimization flag (from CMAKE_BUILD_TYPE) is passed through properly. So that part is fine, and the issue must come from not passing through one or more of the Geant4 flags (Geant4_CXX_FLAGS). This could well be the -ftls-model option, but not sure.

@mnovak42, could you let me know what the contents of the Geant4_CXX_FLAGS are for the install you are working with please? Also, if there's a simple run of the tests that will show the slowdown, let me know and I'll try and reproduce locally.

Investigate the possibilities of reusing some of the host side functionalities on the device

Some of the run-time functionalities (G4HepEmRun), already implemented for the host side simulation, might be reused on the device side. The first targets are actually those, that relies only on the the data structures, i.e. the run-time interpolation of the macroscopic cross sections, stopping power, range, etc..

This would bring great advantage : reduce the code duplications i.e. having implementations both on the
host and device side for these functions.

Jonas (@hahnjo) has already started to work on this:
https://github.com/hahnjo/g4hepem/tree/header-implementations

Upcoming development items

Upcoming development items

This issue is for summarising the next couple of steps in the development for providing all fine-grained functionalities required by a complete integration into the ATLASand CMSdetector simulation environment that maximise the corresponding performance gain.

Required

Required for providing the same functionalities, configuration options given by the native part of the Geant4 toolkit and used today by ATLASand CMSin their detector simulation:

• configuration per detector region: provide the possibility of applying different simulation parameters/algorithms/settings etc. in different detector region. Most importantly, different MSC stepping algorithm, MSC stepping algorithm parameters, energy loss fluctuation settings, etc.

• refactor some of the data structures: most importantly the cross section data structures, that currently stores data in plane arrays. The idea is to simply introduce more structured data that makes easier the further manipulations and extensions of these cross section data. (note: this is not strictly required but makes the following developments as well as the maintenance significantly easier)

• add the missing gamma- and lepto-nuclear interactions: according to the original plan, these interactions will be included only by their cross sections, i.e. G4HepEm will only determine if nuclear interaction needs to be performed but won't include the interaction description (final state generation). This can be done easily by the corresponding native Geant4 processes in the connection layer. The motivation is that, since these nuclear interaction cross sections are significantly smaller compared to the electromagnetic (and relevant at all only at higher energies), the frequency of these kind of interactions is very low (while the complete final state production would require complex hadronic components)

Further (performance) improvements:

Further items, that are not strictly required for the simulation, but provides significant performance benefits for ATLAS and CMS in their detector simulations. Therefore, these are necessary for making available all known related computing performance improvements for ATLASand CMSdetector simulations through G4HepEm:

• "Woodcock-tracking" of $\gamma$ photons in a given detector region (see more details here or here)

• combined MSC with transportation for $e^-/e^+$ (see more details here or here)
  (done: #79)

Additional items:

Some functionalities, that might be necessary to provide if one of the HEP experiments requires to use them:

• implement an $e^-/e^+$ MSC stepping algorithm that corresponds to the most accurate "error-free" stepping provided by Goudsmit-Saunderson MSC model of Geant4 (and currently utilised by CMS in the HGCAL)