PUMGen fails with MPI abort error about pumgen

[Thomas-Ulrich]

Hi, This is a problem of simmodeler, not of pumgen. I guess the parallel algorithm of simmodeler are less used and therefore less tested. How many mpi ranks and threads do you use? Not really useful to use more than 20 cpus or so for regular size meshes. But using several nodes is useful for the available ram. Usually, changing slightly the mesh parameters can help fixing the problem. Thomas. Le 6 septembre 2020 13:07:09 GMT+02:00, vyasjc <[email protected]> a écrit :

…

Sun Sep 06 14:01:48, Warn: SimModeler: [433] Code: 60 String: General error Rank 433 [Sun Sep 6 14:01:48 2020] [c0-3c0s13n3] application called MPI_Abort(MPI_COMM_WORLD, 1) - process 433 Sun Sep 06 14:01:48, Warn: SimModeler: [966] Code: 60 String: General error Rank 966 [Sun Sep 6 14:01:48 2020] [c3-3c1s4n3] application called MPI_Abort(MPI_COMM_WORLD, 1) - process 966 Sun Sep 06 14:01:48, Warn: SimModeler: [984] Code: 60 String: General error Rank 984 [Sun Sep 6 14:01:48 2020] [c3-3c1s9n1] application called MPI_Abort(MPI_COMM_WORLD, 1) - process 984 Sun Sep 06 14:01:48, Warn: SimModeler: [688] Code: 60 String: General error Rank 688 [Sun Sep 6 14:01:48 2020] [c1-3c2s5n3] application called MPI_Abort(MPI_COMM_WORLD, 1) - process 688 Sun Sep 06 14:01:48, Warn: SimModeler: [856] Code: 60 String: General error Rank 856 [Sun Sep 6 14:01:48 2020] [c2-3c2s5n0] application called MPI_Abort(MPI_COMM_WORLD, 1) - process 856 Sun Sep 06 14:01:48, Warn: SimModeler: [675] Code: 60 String: General error Rank 675 [Sun Sep 6 14:01:48 2020] [c1-3c2s2n2] application called MPI_Abort(MPI_COMM_WORLD, 1) - process 675 Sun Sep 06 14:01:48, Warn: SimModeler: [592] Code: 60 String: General error Rank 592 [Sun Sep 6 14:01:48 2020] [c1-3c0s10n3] application called MPI_Abort(MPI_COMM_WORLD, 1) - process 592 Sun Sep 06 14:01:48, Warn: SimModeler: [650] Code: 60 String: General error Rank 650 [Sun Sep 6 14:01:48 2020] [c1-3c1s11n1] application called MPI_Abort(MPI_COMM_WORLD, 1) - process 650 Sun Sep 06 14:01:48, Warn: SimModeler: [658] Code: 60 String: General error Rank 658 [Sun Sep 6 14:01:48 2020] [c1-3c1s13n1] application called MPI_Abort(MPI_COMM_WORLD, 1) - process 658 Sun Sep 06 14:01:48, Warn: SimModeler: [655] Code: 60 String: General error Rank 655 [Sun Sep 6 14:01:48 2020] [c1-3c1s12n2] application called MPI_Abort(MPI_COMM_WORLD, 1) - process 655 Sun Sep 06 14:01:48, Warn: SimModeler: [668] Code: 60 String: General error Rank 668 [Sun Sep 6 14:01:48 2020] [c1-3c1s15n3] application called MPI_Abort(MPI_COMM_WORLD, 1) - process 668 Sun Sep 06 14:01:48, Warn: SimModeler: [667] Code: 60 String: General error Rank 667 [Sun Sep 6 14:01:48 2020] [c1-3c1s15n2] application called MPI_Abort(MPI_COMM_WORLD, 1) - process 667 Sun Sep 06 14:01:48, Warn: SimModeler: [664] Code: 60 String: General error Rank 664 [Sun Sep 6 14:01:48 2020] [c1-3c1s14n3] application called MPI_Abort(MPI_COMM_WORLD, 1) - process 664 Sun Sep 06 14:01:48, Warn: SimModeler: [682] Code: 60 String: General error Rank 682 [Sun Sep 6 14:01:48 2020] [c1-3c2s4n1] application called MPI_Abort(MPI_COMM_WORLD, 1) - process 682 Sun Sep 06 14:01:48, Warn: SimModeler: [842] Code: 60 String: General error Rank 842 [Sun Sep 6 14:01:48 2020] [c2-3c2s1n2] application called MPI_Abort(MPI_COMM_WORLD, 1) - process 842 Sun Sep 06 14:01:48, Warn: SimModeler: [853] Code: 60 String: General error Rank 853 [Sun Sep 6 14:01:48 2020] [c2-3c2s4n1] application called MPI_Abort(MPI_COMM_WORLD, 1) - process 853 Sun Sep 06 14:01:48, Warn: SimModeler: [605] Code: 60 String: General error Sun Sep 06 14:01:48, Warn: SimModeler: [817] Code: 60 String: General error -- You are receiving this because you are subscribed to this thread. Reply to this email directly or view it on GitHub: #13

[vyasjc]

Hii Thomas,

Thanks for the suggestions. You are quite right that its simmodeler error as you can see below in the warning. I am using 64 nodes and 1 task per node and 1 cpu per task. Actually, my domain size is big. Therefore, when i use grid size of 800m on fault as well as in domain it creates mesh but when i use 200m on fault and 800m in domain it fails. I am thinking this is because of memory issue but not quite sure how to fix it.

---

Sep 06 22:33:52, Debug: SimModeler: MeshCubeRefine
16% [=========== ] ^MSun Sep 06 22:33:52, Debug: Creating volume mesh 1 0 0 0
16% [=========== ] |^MSun Sep 06 22:33:52, Debug: Creating volume mesh 1 0 0 0
16% [=========== ] /^MSun Sep 06 22:33:52, Debug: Creating volume mesh 1 0 0 0
16% [=========== ] -^MSun Sep 06 22:33:52, Debug: Creating volume mesh 1 0 0 0
16% [=========== ] ^MSun Sep 06 22:33:52, Debug: Creating volume mesh 1 0 0 0
16% [=========== ] |^MSun Sep 06 22:33:52, Debug: Creating volume mesh 1 0 0 0
Sun Sep 06 22:34:01, Warn: SimModeler: [49] Code: 60 String: General error
Rank 49 [Sun Sep 6 22:34:01 2020] [c6-2c2s9n1] application called MPI_Abort(MPI_COMM_WORLD, 1) - process 49
srun: error: nid05157: task 49: Aborted
srun: Terminating job step 15578354.0
slurmstepd: error: *** STEP 15578354.0 ON nid05104 CANCELLED AT 2020-09-06T22:34:01 ***
srun: error: nid05127: task 19: Terminated
srun: error: nid05162: task 54: Terminated
srun: error: nid05109: task 5: Terminated
srun: error: nid05140: task 32: Terminated

[Thomas-Ulrich]

Hi, I would need to see the mesh parameters, but with 64 nodes, you have a huge amount of RAM on shaheen I think, so this should not be the problem.
What the size (number of cells) of the mesh with 800m everywhere?

[vyasjc]

Below is the attached input file (.xml file). With 800m on fault and in domain grid size, the mesh size is 44 Million (successful run case). Now when I have 200m on fault and 800m in domain, not sure on how to estimate mesh size in advance (ofcourse for uniform grid spacing it is straight forward).

TP-3D-final_withXML.xml.pdf

[Thomas-Ulrich]

Hi,

I would not recommend using too many SmoothingLevel (time to mesh, usefulness?) and too small aspect ratio (less fit of the target geometry to match the AR criteria).
Here is what I usually use:

<area_AspectRatio value="6"/>
<vol_AspectRatio value="12"/>
<SurfaceMeshing SmoothingLevel="1" SmoothingType="Gradient" DiscreteAngle="5.0" Snap="0"></SurfaceMeshing>
<VolumeMeshing  SmoothingLevel="1" SmoothingType="Gradient" SetOptimisation="1"></VolumeMeshing>

Apart from that, I think if you can mesh with 800m on fault you should be able to mesh with 200m, because the mesh wont be much larger.
You should launch a set of runs with slightly varying parameters until simmodeler can mesh (e.g. gradation rate 0.14,0.15,0.16).
That's what I do usually and it works.
(we should report the problem also to simmodeler at some point so that it get fixed).

[vyasjc]

Hii Thomas,

Thanks for your suggestions. I used your suggested parameters for area and volume aspect ratio and also tried different gradation rate (from 0.1 to 0.3) but still getting the similar error. I will try very large aspect ratio to check if that works.

[Thomas-Ulrich]

can you share the folder on shaheen? I can try to mesh it on supermucNG with an older (and therefore more stable) simModelerLib.

[vyasjc]

Thanks, here is the folder: /project/k1488/mesh/simmodeler-mesh-final-800m

[Thomas-Ulrich]

Hi,
So I could generate a mesh with your files, on supermuc NG test queue, with:

#SBATCH --nodes=3
#SBATCH --ntasks-per-node=8

and the mesh is not significantly larger than with fault mesh size=800:

Tue Sep 08 09:33:55, Info:  AR max: 11.9975 
Tue Sep 08 09:33:55, Info:  AR (target: < ~10): 
Tue Sep 08 09:33:55, Info:    [ 0.00 , 2.00 ): 882814 
Tue Sep 08 09:33:55, Info:    [ 2.00 , 4.00 ): 45818488 
Tue Sep 08 09:33:55, Info:    [ 4.00 , 6.00 ): 1174844 
Tue Sep 08 09:33:55, Info:    [ 6.00 , 10.00 ): 544527 
Tue Sep 08 09:33:55, Info:    [ 10.00 , 20.00 ): 5304 
Tue Sep 08 09:33:55, Info:    [ 20.00 , 40.00 ): 0 
Tue Sep 08 09:33:55, Info:    [ 40.00 , 100.00 ): 0 
Tue Sep 08 09:33:55, Info:    [ 100.00 ,inf): 0 
Tue Sep 08 09:33:55, Info:  Converting mesh to APF 
Tue Sep 08 09:34:46, Info:  Mesh size: 48425977 
Tue Sep 08 09:34:48, Info:  Minimum insphere found: 10.1935 

my pumgen is compiled against simmodeler: 14.0-190921, which is slightly more recent than your 14.0-200719.
Maybe that's one of the problems.
(also note that I use much less nodes than you).

[vyasjc]

Hii Thomas,

When I use only 4 nodes, I am also able to create mesh with my simmodeler (so old one). This is really strange and bizarre, is it that the work load balance was creating whole problem? Is that the old simmodeler has some work partitioning issue that you are aware of? And thanks for your suggestions, because without experience you can't find out such tiny detail.

[Thomas-Ulrich]

Good,

Maybe they do not have access to large HPC infrastructures, or it is not much tested because there are not many HPC users.