Predictive Science Inc.
www.predsci.com
POT3D
is a Fortran code that computes potential field solutions to approximate the solar coronal magnetic field using observed photospheric magnetic fields as a boundary condition. It can be used to generate potential field source surface (PFSS), potential field current sheet (PFCS), and open field (OF) models. It has been (and continues to be) used for numerous studies of coronal structure and dynamics. The code is highly parallelized using MPI and is GPU-accelerated using MPI+OpenACC. The HDF5 file format is used for input/output.
POT3D
is the potential field solver for the WSA model in the CORHEL software suite publicly hosted at the Community Coordinated Modeling Center (CCMC).
A version of POT3D
that includes GPU-acceleration with both MPI+OpenACC and MPI+OpenMP was released as part of the Standard Performance Evaluation Corporation's (SPEC) beta version of the SPEChpc(TM) 2021 benchmark suites.
Details of the POT3D
code can be found in the following publications:
- Variations in Finite Difference Potential Fields.
Caplan, R.M., Downs, C., Linker, J.A., and Mikic, Z. Submitted to Ap.J. (2021) - From MPI to MPI+OpenACC: Conversion of a legacy FORTRAN PCG solver for the spherical Laplace equation.
Caplan, R.M., Mikic, Z., and Linker, J.L. arXiv:1709.01126 (2017)
Modify the file build.sh
to set the HDF5
library paths/flags and compiler flags compatible with your system environment.
Then, run ./build.sh
.
See comments in build.sh
for more details.
After building the code, you can test it is working by running ./validate.sh
.
This will perform 2 runs of a small case using 1 and 2 MPI ranks respectively.
The runs are performed in testsuite/validation/run/
and the second run overwrites the first.
Each result will be checked against a reference solution (in /runs/validation/validation
) and a PASS/FAIL message will be displayed.
POT3D uses a namelist in an input text file called pot3d.dat
to set all parameters of a run. See the provided pot3d_input_documentation.txt
file for details on the various parameter options. For any run, an input 2D data set in HDF5 format is required for the lower radial magnetic field (Br
) boundary condition. Examples of this file are contained in the examples
and testsuite
folders.
To run POT3D
, set the desired run parameters in a pot3d.dat
text file, then copy or link the pot3d
executable into the same directory as pot3d.dat
and run the command:
<MPI_LAUNCHER> -np <N> ./pot3d
where <N>
is the total number of MPI ranks to use (typically equal to the number of CPU cores) and <MPI_LAUNCHER>
is your MPI run command (e.g. mpiexec
,mpirun
, ibrun
, srun
, etc).
For example: mpiexec -np 1024 ./pot3d
Important!
For CPU runs, make sure ifprec=2
is set in the pot3d.dat
input file.
For GPU runs, make sure ifprec=1
is set in the pot3d.dat
input file.
For standard cases, one should launch the code such that the number of MPI ranks per node is equal to the number of GPUs per node
e.g.
mpiexec -np <N> --ntasks-per-node 4 ./pot3d
or
mpiexec -np <N> --npersocket 2 ./pot3d
Note! To run efficiently, it is critical that ifprec=1
is set in pot3d.dat
.
To estimate how much memory (RAM) is needed for a run, compute:
memory-needed = nr*nt*np*8*15/1024/1000/1000 GB
where nr
, nt
, and np
are the chosen problem sizes in the r
, theta
, and phi
dimension.
Note that this estimate is when using ifprec=1
. If using ifprec=2
, the required memory is ~2x higher.
Depending on the input parameters, POT3D
can have various outputs. Typically, the three components of the potential magnetic field is output as HDF5
files. In every run, the following two text files are output:
pot3d.out
An output log showing grid information and magnetic energy diagnostics.timing.out
Time profile information of the run.
In the examples
folder, we provide ready-to-run examples of three use cases of POT3D
in the following folders:
/potential_field_source_surface
A standard PFSS run with a source surface radii of 2.5 Rsun./potential_field_current_sheet
A standard PFCS run using the outer boundary of the PFSS example as its inner boundary condition, with a domain that extends to 30 Rsun. The magnetic field solution produced is unsigned./open_field
An example of computing the "open field" model from the solar surface out to 30 Rsun using the same input surface Br as the PFSS example. The magnetic field solution produced is unsigned.
In the testsuite
folder, we provide test cases of various sizes that can be used to validate and test the performance of POT3D
.
Each test case contains an input
folder with the run input files, a run
folder used to run the test, and a validation
folder containing the output diagnotics used to validate the test, as well as a text file named validation_run_information.txt
containing information on how the validation run was computed (system, compiler, number of ranks, etc.) with performance details. Note that all tests use ifprec=1
so that they can validate GPU runs, therefore CPU performance will not be optimal.
To run a test, use the included script run_test.sh
as:
run_test.sh <TEST> <NP>
where <TEST>
is the test folder name and <NP>
is the number of MPI ranks to use. The test will run and then use the included script scripts/pot3d_validate.sh
that takes two pot3d.out
files and compares their magnetic energy values in order to validate the run results.
The following is a list of the included tests, and their problem size and memory requirements:
validation
Grid size: 63x91x225 = 1.28 million cells
Memory (RAM) needed: ~1 GBsmall
Grid size: 133x361x901 = 43.26 million cells
Memory (RAM) needed: ~6 GBmedium
Grid size: 267x721x1801 = 346.7 million cells
Memory (RAM) needed: ~41 GBlarge
Grid size: 535x1441x3601 = 2.78 billion cells
Memory (RAM) needed: ~330 GB
Note that these tests will not output the 3D magnetic field results of the run, so no extra disk space is needed.
Instead, the validation is done with the magnetic energy diagnostics in the pot3d.out
file.