microhh / microhh2

Deprecated repository: use master branch of `microhh` for new 2.0 version.

MicroHH

MicroHH is a computational fluid dynamics code made for Direct Numerical Simulation (DNS) and Large-Eddy Simulation of turbulent flows in the atmospheric boundary layer. The code is written in C++.

MicroHH is hosted on GitHub (http://github.com/microhh). Here, the latest version of the source code can be found, as well as all releases. Bug notifications and fixes are always welcome.

MicroHH is described in detail in Van Heerwaarden et al. (2017). In case you decide to use MicroHH for your own research, the developers would appreciate to be notified and kindly request to cite their reference paper. The version described in the reference paper has been assigned a DOI via Zenodo.

Requirements

In order to compile MicroHH you need:

• C++ compiler
• FFTW3 libraries
• Boost libraries
• NetCDF4
• CMake
• MPI2/3 implementation (optional for MPI support)
• CUDA (optional for GPU support)
• Python + numpy + python-netcdf4 (optional for running example cases)
• Ipython + python-netcdf4 + matplotlib (optional for plotting results example cases)

Compilation of the code

First, enter the config directory:

cd config

Here, you find a potential series of settings with the extension .cmake for different systems. Check whether your system is there. If not, create a file with the correct compiler settings and the proper location for all libraries. Then, copy your system file to default.cmake. Let us assume your system is Ubuntu:

cp ubuntu.cmake default.cmake

Then, go back to the main directory and create a subdirectory with an arbitrary name in which you will compile the code. Let us assume this directory is called "build":

cd ..  
mkdir build  
cd build   

From this directory, run cmake with the suffix .. to point to the parent directory where the CMakeLists.txt is found. This builds the model without Message Passing Interface (MPI) and CUDA support.

cmake ..

In case you prefer to enable either MPI or CUDA, run INSTEAD of the previous command:

cmake .. -DUSEMPI=TRUE

or

cmake .. -DUSECUDA=TRUE

(Note that once the build has been configured and you wish to change the USECUDA or USEMPI setting, you must delete the build directory or create an additional empty directory from which cmake is run.)

With the previous command you have triggered the build system and created the make files, if the default.cmake file contains the correct settings. Now, you can start the compilation of the code and create the microhh executable with:

make -j

Your directory should contain a file named microhh now. This is the main executable.

Running an example case

To start one of the included test cases, go back to the main directory and open the directory cases. Here, a collection of test cases has been included. In this example, we start the drycblles case, a simple large-eddy simulation of a dry convective boundary layer.

cd cases/drycblles

First, we have to create the vertical profiles for our prognostic variables:

python drycblles_input.py

Then, we have to copy or link the microhh executable to the current directory. Here we assume the executable is in the build directory that we have created before.

cp ../../build/microhh .

Now, we can start microhh in initialization mode to create the initial fields:

./microhh init drycblles

If everything works out properly, a series of files has been created. The model can be started now following:

./microhh run drycblles

This will take some time. Now, a statistics file called drycblles.default.0000000.nc has been created. You can open this file with your favorite plotting tool, or run some example plots using the provided plotting script that uses Python and matplotlib. This is most easily done in interactive python:

ipython  
run drycbllesstats

This should show you a set of basic plots. Congratulations, you have just completed your first run of MicroHH.

Happy MicroHHing!

microhh2's Issues

Missing density in effective radius calculation

double rel_value = cld_mask_liq({icol, ilay}) * fac * std::pow(3.*(clwp_subset({icol, ilay})/layer_mass) / four_pi_Nc0_rho_w, (1./3.)); This line does not provide proper units.

io in its own miniclass

`dnum` is not yet calculated/output in CUDA version

Add budget statistscs

Initialisation cross-sections

There is a lot of duplicate code like:

https://github.com/microhh/microhh2/blob/thermo/src/thermo_moist.cxx#L1038
https://github.com/microhh/microhh2/blob/thermo/src/thermo_dry.cxx#L580

I think cross.cxx should handle this; i.e. all classes pass a list of available cross-sections to cross, and cross returns a list with matching cross-sections.

"Mass" is a useless quantity on standard output

Mass only gives the average value of whatever the first scalar may be (e.g., nr, in the case of RICO).

Split column out of stat

Column takes up very little CPU time, but frequent column outputting is limited by waiting for the stats.

`exec_all_stats` is not implemented in gcss radiation

Dycoms doesn't work anymore as exec_all_stats is not implemented in the GCSS radiation

rho that is included in the stats is not the correct one in thermo_vapor

In thermo_vapor the rho that is stored in the stats is the fields.rhoref which is incorrect in case swupdatebasestate is enabled.

Fix satadjust crash on one MPI task

If satadjust fails on just one MPI task, the code keeps running. This has already been fixed in MicroHH1.6, still has to be added to 2.0

Automatic test scripts

loc in Fields3d

Sullivan2010 case misses rotation

Our implementation of the Sullivan and Patton (2010) case misses rotation and the small large-scale pressure force.

Revised NetCDF I/O

Create surface_bulk

diff_smag2.cu has a hardcoded z0m

See prepare_device(). Probably left over from testing/development....

`umodel` and `vmodel` in budget routines contain model mean rather than mask mean

Move some cuda setup to some kind of constants file

Currently, every cuda call is preceeded by some preparation statemebts, that probably can be done just once in a seperate class

Switch 4m for advec causes undefined behavior on GPU tests

It seems that swadvec=4m is not correctly implemented on GPU. I get random crashes in restart tests, and setting it to 4 removes the problems.

Check all compiler warnings

Generate multiple outputs in "column"

struct for background state

Simple radiation

A simple radiation scheme that can easily ported to the GPU

Immersed Boundary Methods

First only for DNS, as was in 1.6; later modify for LES

No output in diffusive fluxes in DNS stats, erroneous output in subgrid fluxes in LES stat.

The output of the diffusive fluxes is incorrect in the new stats module. If I plot the *flux variables, none of them gives a linear flux profile in quasi steady state. The mosercase gives zero fields for diffusive fluxes.

Buoyancy correction surface

This fix: 24e57a4 is not implemented on the GPU.

Init for new stats module with same vector of strings as exec

It would be nice to have a simplified init for stats, in line with the exec function.

Add rainrate and rwp to the statistics

Bug in writing restart file times

With the MicroHH2 master I get:

Saving "time.0010800" ... OK
Saving "qt.0010801" ... OK
Saving "thl.0010801" ... OK

This is the relevant piece of code:

                // Save the data for restarts.
                if (timeloop->do_save())
                {
                    #ifdef USECUDA
                    if (!cpu_up_to_date)
                    {
                        #pragma omp taskwait
                        cpu_up_to_date = true;
                        fields  ->backward_device();
                        boundary->backward_device();
                        thermo  ->backward_device();
                    }
                    #endif
                    // Save data to disk.
                    #pragma omp task default(shared)
                    {
                        timeloop->save(timeloop->get_iotime());
                        fields  ->save(timeloop->get_iotime());
                    }
                }

Not sure what the best solution would be; save the iotime inside the block with pragma omp?taskwait?

Simulations not bitwise identical

proper half level mean for masked fluxes etc.

Merge all branches/features back into master

4th order covariances

Stats `add_profs` does not crash if name already exists

Registering the same name twice does not throw a stats error.

Convert microhh 1 to microhh 2 structure

OpenMP stats

Microphysics at GPU

Port microphysics to the GPU; possibly including icemicro

Add Couvreux-scalar conditional sampling

Decaying passive scalars

Interpolate masked stat for u and v

GPU Stats slow for many cond.samplings/large domain

Cuda-memcheck everything

Make MicroHH leak-free again!

Column ql with GPU

Column at the edge (x=0, y=0) has incorrect ql and b in CUDA code, works with MPI. Column in the middle seems to be okay.

Blacklist crash with .*

Calculate covariances for stat

Overall cleanup

*) Remove superfluous includes
*) Make sure that parameters are const/restrict, where possible
*) make sure that all numbers are of type TF

Tmp field dispatch system

Create a piece of code that can reserve and release tmp fields, to avoid having to manually manage the numbering of tmp fields.

Make logical stats groups

Groups can be confusing with prognostic thermo fields in default group and diagnostic fields in thermo group.

Allowing spatial fields of z0 in surface layer solver

Clean up stats

Make sure that masks are current in the stats class and have stats routines that calculate most standard statistics all at once