Identify gaps in current technology preventing OpenMP support about dagmc

[gonuke]

2011-Aug-04 12:40:33 by sjackson:

MCNP5 requires its OpenMP compiler to support both the THREADPRIVATE and EQUIVALENCE attributes for the same variable. Our current fortran compiler (gfortran 4.3.2) does not. It produces errors such as:

Error: EQUIVALENCE attribute conflicts with THREADPRIVATE attribute in 'xxx' at (1)

(=xxx= being one of the many variables that participates in EQUIVALENCE statements.)

A gfortran bug was filed against this issue in 2009 here. It does not appear to have been fixed yet, although a patch does exist that allows the code to compile. We could apply this patch if we were willing to build a custom version of gfortran. The alternative is to work with a different compiler (which would probably entail using a different compiler system for the dag-mcnp5 glue code and MOAB, and possibly sorting out a different scheme for calling C functions from Fortran).

[gonuke]

2011-Aug-04 12:46:42 by sjackson:

Once we have compiler support for building OpenMP MCNP5, it remains to be determined whether MOAB's DagMC object can support multithreaded access. In principle, it should, since the queries made by MCNP5 are theoretically read-only. TimTautges mentioned that MOAB does some internal caching to speed up queries, and this caching may not be thread-safe. He thought it wouldn't be terribly difficult to work around this, and it might even work correctly (possibly slowly) without any intervention.

The current version of the dag-mcnp5 glue code is not thread-safe: it uses a handful of static variables that would need to become thread-local. Need to investigate the best way to do this-- whether through OpenMP directives or pthread calls or, on the right compiler, the =__thread= keyword. (Or, on a very modern compiler supporting c++0x, the =thread_local= keyword.)

[gonuke]

2011-Aug-04 13:17:26 by sjackson:

There's a pgf90 installation at /afs/engr.wisc.edu/apps/bin/pgf90. It does compile mcnp5 with OpenMP enabled, however, it stubbornly refuses to generate 64-bit code, and in 32-bit mode I can't coax it to get past the linking step when building dagmc. (It has trouble finding a 32-bit =crtbegin.o=, probably a gcc configuration issue, probably not the only issue we'd run into following this path. Anyway, we'd want our OpenMP version to be 64-bit, so fixing the 32-bit build isn't the right path.)

[makeclean]

Whilst doing some other testing, I gave this issue some thought, in addition to the changes required mentioned by Steve, the calls to MOAB will have to be made thread safe, or indeed store the required "cookies" as TimT calls them in some thread safe form, in its current implementation we are not getting thread safe behavior,

The code snippet from commit 05b8c91 is simple ray_fire driver code, first finding which volume the point (0,0,0) is,
and then firing a ray from 0,0,0 along 1,0,0 and thus return the distance,

14 Distance = 0 Surface = 0
21 Distance = 10 Surface = 12682136550675316779
0 Distance = 10 Surface = 12682136550675316779
5 Distance = 10 Surface = 12682136550675316779
22 Distance = 10 Surface = 12682136550675316779
15 Distance = 10 Surface = 12682136550675316779
23 Distance = 10 Surface = 12682136550675316779
12 Distance = 10 Surface = 12682136550675316779
2 Distance = 0 Surface = 0
7 Distance = 10 Surface = 12682136550675316779
18 Distance = 10 Surface = 12682136550675316779
8 Distance = 10 Surface = 12682136550675316779
13 Distance = 0 Surface = 0
9 Distance = 0 Surface = 0
20 Distance = 10 Surface = 12682136550675316779
1 Distance = 10 Surface = 12682136550675316779
16 Distance = 10 Surface = 12682136550675316779
3 Distance = 10 Surface = 12682136550675316779
10 Distance = 0 Surface = 0
4 Distance = 10 Surface = 12682136550675316779
19 Distance = 0 Surface = 0
11 Distance = 0 Surface = 0
17 Distance = 10 Surface = 12682136550675316779
6 Distance = 10 Surface = 12682136550675316779

So as well as knowing the thread_id from MCNP, we will need to pass back or determine within MOAB the thread

[makeclean]

To appropriately test this issue we can define two tests which will show/not show the correct functioning of an OMP enabled DAGMC.

1. The first geometry should be N volumes arranged in a row, particles are 'born' in the first volume and stream in the direction of the remaining volumes, the #pragma omp parallel construct should be placed such that each thread will make calls to point_in_volume and ray_fire in the appropriate order such that each the particle that belongs to each thread shows the correct information regarding its location, next volume etc.
2. The second test is similar to the first, with the main difference being that there are now Nxnum_thread volumes, arranged similiarly as in 1, however in this cases the volumes are arranged such that the particle created by each thread will not have the same history as other particles, i.e an array of volumes where each threads ray starts in different volumes and the next ray_fire would also take them to new, non shared volumes.

I will attach figures to clarify.

[pshriwise]

In terms of the DAGMC/MOAB I think we have proof now that the DAGMC interface is threadable.

I'm adding relevant code changes that were necessary to make that happen here:

GQT Threading Updates

#561 - DAGMC Threading Interface