arporter / psycloned_nemo Goto Github PK

This Issue provides a place to capture planning (and possibly results) for the 2nd NEMO GPU hackathon. The intention is for this hackathon to focus on getting NEMO running on multiple GPUs using MPI.

• Hardware. STFC's workstation has 2 GPUs. Isambard hardware configuration to be checked.
• NEMO configuration. Currently we use ORCA1-SI3-G08 but we will need a larger domain when running on multiple GPUs.
• Use PSyclone to process existing configuration with MPI enabled. Validate.
• Investigate performance of this 'vanilla' MPI version with ORCA1 configuration.
• Optimise using [GPUDirect?]
• Investigate performance on larger domain.
• Investigate performance on Ampere??

I've created the profiling_new_api branch which has profiling regions inserted. To compile this you'll need to build the PSyclone profiling wrapper library. This can be found in PSyclone/lib/profiling/nvidia/. Once you've built it, you need to tweak the NEMO build system to tell it to ignore the associated module use statements and to link with the wrapper library and the nvtx library itself. I've added the following to my arch file (edit as appropriate for your system):

%PROFILE_HOME        ${HOME}/PSyclone/lib/profiling/nvidia
%PROFILE_LIB         -L%PROFILE_HOME -lnvtx_prof -L${CUDA_DIR}/lib64 -lnvToolsExt
%PROFILE_INC         -I%PROFILE_HOME

and then extended the list of include and link flags:

%USER_INC            %PROFILE_INC %XIOS_INC %OASIS_INC %NCDF_INC
%USER_LIB            %PROFILE_LIB %XIOS_LIB %OASIS_LIB %NCDF_LIB

I also edited dev_r10037_GPU/mk/bldxag.cfg and added the following line:

bld::excl_dep        use::profile_psy_data_mod

(this tells FCM it doesn't need to try and build the profile_psy_data_mod module.)

I've created the no_ice_prof branch which is the same as the profiling branch but built with makenemo del_key 'key_si3' (all other build arguments remain the same).