202204 release - exited on signal 11 (Segmentation fault) about shield_physics

[kaiyuan-cheng]

@StevePny It turns out that the default stack size, 8 MB, is insufficient to hold the large one-dimension variable, lbms. The solution is to set an unlimited stack size.

[lharris4]

Yes, the stacksize error strikes again. 'limit stacksize unlimited' should be run on all machines by default. We may want to make an FAQ item to this extent. Lucas

…

On Thu, Sep 8, 2022 at 6:15 PM kaiyuan-cheng ***@***.***> wrote: Closed #11 <#11> as completed. — Reply to this email directly, view it on GitHub <#11 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AMUQRVDDCGODRLHIXNRXIDLV5JQPVANCNFSM5VPZFXLA> . You are receiving this because you are subscribed to this thread.Message ID: ***@***.***>

[lharris4]

Hi, Steve. That is great---glad to hear it runs correctly now. This is useful background for other users of the container. Lucas

…

On Thu, Dec 1, 2022 at 10:48 AM Steve Penny ***@***.***> wrote: To provide a clarifying detail - The docker container does not inherit the system stack limit by default. The ulimit can be set on the command line when running the docker container, but 'unlimited' is not a permitted option. In order to specify an unlimited stack size in the docker container, one can add this option: --ulimit stack=-1 With this setting I can run the regional_Laura_test case on an AWS c6g.8xlarge ec2 instance. Note - to be safe, I also set the stack size in the ec2 instance with: ulimit -s unlimited — Reply to this email directly, view it on GitHub <#11 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AMUQRVC75IUN63OAUZEXSBTWLDCGJANCNFSM5VPZFXLA> . You are receiving this because you commented.Message ID: ***@***.***>

[laurenchilutti]

@StevePny What test are you running that results in this seg fault failure? Is it one of the tests in the CI directory?

[laurenchilutti]

@StevePny To expand upon my previous comment: Is it one of the tests in the SHiELD_build repository CI directory?

[StevePny]

@laurenchilutti
The latter case is the regional_Laura case distributed with the SHiELD-in-a-box:
https://www.gfdl.noaa.gov/shield/shield-in-a-box/
https://zenodo.org/record/5090124/files/regional_Laura.zip

I was able to install SHiELD and run these example cases (regional_Laura and global_nest_Laura) prior to the 202204 release.

[kaiyuan-cheng]

@StevePny
I have tested the latest SHiELD code with the regional Laura case. When I built SHiELD natively on an NOAA HPC, the Laura case works fine. However, it does not work with the containerized SHiELD, which is very strange.

[kaiyuan-cheng]

Looks like it is the NCEP library causing the crash. Segmentation fault occurs at

SHiELD_physics/gsmphys/sfcsub.F

Line 2757 in 8c46d4f

call getgb(lugb,lugi,kdata,lskip,jpds,jgds,ndata,lskip,

However, I still don't understand why it is the case. Before this line, another NCEP library, getgbh(), works just fine. Also, the same compiler flags and arguments worked previously.

[StevePny]

@kaiyuan-cheng just checking in - has any progress been made on clearing up this issue, or should we continue with the pre-202204 version?

[kaiyuan-cheng]

I don't have a solution at this moment, unfortunately.

…

On Mon, Jul 18, 2022 at 7:04 PM Steve Penny ***@***.***> wrote: @kaiyuan-cheng <https://github.com/kaiyuan-cheng> just checking in - has any progress been made on clearing up this issue, or should we continue with the pre-202204 version? — Reply to this email directly, view it on GitHub <#11 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AR2VX65KW5VABFKC5XN3A5TVUXPGLANCNFSM5VPZFXLA> . You are receiving this because you were mentioned.Message ID: ***@***.***>

[StevePny]

To provide a clarifying detail -
The docker container does not inherit the system stack limit by default. The ulimit can be set on the command line when running the docker container, but 'unlimited' is not a permitted option. In order to specify an unlimited stack size in the docker container, one can add this option:

--ulimit stack=-1

With this setting I can run the regional_Laura_test case on an AWS c6g.8xlarge ec2 instance.

Note - to be safe, I also set the stack size in the ec2 instance with:
ulimit -s unlimited