weiming-hu / renewablesimulator

evergreen -- our pursuit of a more sustainable future

Hi there 👋

My name is Weiming Hu. I'm a researcher in ⚡ Renewable Energy ⚡ and ☁️ Weather Forecasting ☁️ .

I'm also an enthusiast for free software and the open-source movement.

• 🚧 I'm working on 🚧

  • Developing R and Python packages for the scientific community
  • Becoming a Bangle.JS contributor.

• 💬 Ask me about 💬

  • R, Python, C++, Supercomputers

• 📫 How to reach me 📫

  • My personal website

renewablesimulator's Issues

Parallel efficiency

This is the time profiling from 4 nodes, 72 processes on each node:

****************** Simple Clock Summary ******************
Total wall time: 5807.40820813179s
Reading file: 2727.614988088608s
Sun position calculation: 658.7170560359955s
Scenario 00000 simulation: 1468.309131860733s
Scenario 00000 simulation for analogs: 596.2754669189453s
Scenario 00000 simulation: 140.38229155540466s
Scenario 00000 simulation for forecasts: 45.32916855812073s
Scenario 00000 simulation: 133.69862699508667s
Scenario 00000 simulation for analysis: 37.081478118896484s
*************** End of Simple Clock Summary **************

This is the time profiling from 2 nodes, 72 processes on each node:

****************** Simple Clock Summary ******************
Total wall time: 7323.613418340683s
Reading file: 1980.4284970760345s
Sun position calculation: 1295.2253940105438s
Scenario 00000 simulation with analogs: 3146.107565164566s
Scenario 00000 writing simulation for analogs: 274.17069029808044s
Scenario 00000 simulation with forecasts: 273.43497824668884s
Scenario 00000 writing simulation for forecasts: 51.14760994911194s
Scenario 00000 simulation with analysis: 278.81663608551025s
Scenario 00000 writing simulation for analysis: 24.282047510147095s
*************** End of Simple Clock Summary **************

The overall wall time is improved when more processes are created. And the computation parts (e.g. various stages for simulation) show good parallelization efficiency. But the file I/O overhead is significant.

The significant file I/O could be related to parallel access to NetCDF files. All processes are accessing the same file which could slow down the process.

A better model for parallel file I/O would be a hierarchical structure where only a handful of processes carry out the file I/O.

Simulation on HPC

XSEDE Stampede2

To prepare the evergreen with MPI capability on Stampede2, we need to enable several modules and build netCDF4 with the parallel enabled NetCDF package.

On a login node, the following code prepares the environment.

# Clear all pre-loaded modules
module purge

# Load required modules
module load intel/18.0.2 impi/18.0.2 pnetcdf/1.11.0 python3/3.7.0 phdf5/1.10.4 parallel-netcdf/4.6.2

# Initialize virtual environment with python3
virtualenv -p python3 $HOME/venv

# Start the virtual environment
source $HOME/venv/bin/activate

# Install python modules with no extra setup.
# netCDF4 is installed later separately.
#
pip install pandas numpy pvlib progress Cython mpi4py pyyaml pyinstrument line_profiler tqdm

# Install netCDF4 with extra setup to support parallel I/O
# There are some steps involved.
#

# Download source code
wget https://codeload.github.com/Unidata/netcdf4-python/zip/master -O netCDF4.tar.gz

# Define environment variables
export HDF5_DIR=$TACC_HDF5_DIR
export MPI_INCDIR=$TACC_IMPI_INC

# Manual installation
pip install netCDF4.tar.gz

# Remove the source files
rm netCDF4.tar.gz

On a compute node, the following code tests the environment.

# Start the environment properly
module purge
module load intel/18.0.2 impi/18.0.2 pnetcdf/1.11.0 python3/3.7.0 phdf5/1.10.4 parallel-netcdf/4.6.2
source $HOME/venv/bin/activate

# Run a small test case
ibrun -n 10 python evergreen.py --nc ~/scratch/AnEn/4000_analogs.nc --profile --stations 20

Linking problem in build tree on MacOS

RPATH linking problem persists on MacOS in the build tree. When I try to run the built executable, I get the following error:

$ ./RenewableSimulator 
dyld: Library not loaded: ssc.dylib
  Referenced from: /Users/wuh20/github/RenewableSimulator/output/bin/./RenewableSimulator
  Reason: image not found

I used otool to look into the linked libraries. The library scc.dylib is clearly wrong. I've been trying to use different parameters and setting in CMakeLists.txt file but in vein.

$ otool -L RenewableSimulator 
RenewableSimulator:
	ssc.dylib (compatibility version 0.0.0, current version 0.0.0)
	/usr/local/opt/gcc/lib/gcc/8/libstdc++.6.dylib (compatibility version 7.0.0, current version 7.25.0)
	/usr/lib/libSystem.B.dylib (compatibility version 1.0.0, current version 1252.50.4)
	/usr/local/lib/gcc/8/libgcc_s.1.dylib (compatibility version 1.0.0, current version 1.0.0)