adokter / vol2bird

Generates vertical profiles of birds from weather radar volume scans

Vertical profiling of biological scatterers

vol2bird is an algorithm that calculates vertical profiles of birds and other biological scatterers from weather radar data. vol2bird is written in C.

See these publications for a description of the methodology:

Bird migration flight altitudes studied by a network of operational weather radars
Dokter AM, Liechti F, Stark H, Delobbe L, Tabary P, Holleman I
J. R. Soc. Interface, 8, 30–43, 2011, DOI 10.1098/rsif.2010.0116

Recent algorithm extensions and integration with the bioRad R package are described here:

bioRad: biological analysis and visualization of weather radar data
Dokter AM, Desmet P, Spaaks JH, van Hoey S, Veen L, Verlinden L, Nilsson C, Haase G, Leijnse H, Farnsworth A, Bouten W, Shamoun-Baranes J.
Ecography, 42, 852-860, 2019, DOI 10.1111/ecog.04028

installation

Use the vol2bird installer to install vol2bird and all its dependencies:

It currently supports:

• Ubuntu 22.04
• Ubuntu 21.04
• Ubuntu 18.04 & 18.10
• CentOS 8
• RedHat 8 (see RedHat Prerequisities below)
• Mac OS Intel

repository contents

Browse source code at:
https://github.com/adokter/vol2bird

Report a bug at: https://github.com/adokter/vol2bird/issues

• data contains two polar volume radar files for testing
• doc documentation on ODIM hdf5 data format
• docs html pages of this documentation
• docker scripts for generating and running a Docker container for vol2bird
• doxygen doxygen configuration for generating this documentation
• etc contains the user configuration file options.conf. Put your own modified copy in your working directory to run vol2bird with non-default settings.
• lib contains the main library
• pgfplugin product generation framework (PGF) plugin for the BALTRAD system
• pyvol2bird is a python wrapper for the library
• src contains main executables for vol2bird (main program) and rsl2odim (converts NEXRAD to ODIM data format)
• tests unit tests for pgfplugin

Copyright 2010-2022 Adriaan M. Dokter (Cornell lab of ornithology, University of Amsterdam) & Netherlands eScience Centre

Contributors: Anders Henja, Jurriaan H. Spaaks (NLeSC), Lourens Veen (NLeSC), Iwan Holleman (KNMI & Radboud University)

vol2bird's Issues

JSON output not complient with ODIM

Function exportBirdProfileAsJSON needs to be updated to new ODIM standard

add license

LGPL3 - versie Baltrad Anders
Apache 2 - github

static cluttermap not implemented

add libconfuse auto-detection to configure script

case with intense migration is filtered out

case provided by Baptiste Schmid for the Montancy radar in France. File frmtc_pvol_20151009T2200Z.h5 illustrates the problem. This case has a too low radial velocity standard deviation, causing it to be identified as rain.

check whether azimuth bin size correction is correct

see https://github.com/adokter/baltrad/blob/40c1c07e3efc17e5ec5dad6fa30c4012c9fc07f4/libvol2bird.c#L895

installing RAVE on Mac OSX using node-installer requires tweaking of install scripts

When installing RAVE I had to make several tweaks to the install scripts, in order for the library to compile. Anders, advise on how to simplify this process would be very welcome. I decided to abandon the use of node-installer, and install from git directly instead, see issue #37

Installed RAVE on Mac OSX using the command ./setup --prefix=/opt/blt --jdkhome=/Library/Java/JavaVirtualMachines/jdk1.8.0_25.jdk/Contents/Home/ --with-psql=/Applications/Postgres.app/Contents/MacOS/include,/Applications/Postgres.app/Contents/MacOS/lib --tomcatpwd=baltrad --with-rave-gmap --with-bropo install

undetects are not identified as missing value

improve removal of triple-PRT dealiasing errors in French volume data

Maybe include a threshold in the percentage of outliers that should be removed, and loop more than 3 times in VVP fit to remove outliers

constructorUchar assigns NAN

NAN can only be used for floats and doubles

rain filtering no longer working adequately since refactoring

profile 2 is no longer needed

can be removed in order to improve performance

add dual-pol rain cell finding

was implemented in Adriaan's reference code

write python wrapper for baltrad

calculation beam height does not account for earth's curvature and diffraction

Not very urgent, at 25 km range the correction is only 50 m, so considerably less than our standard altitude bin width of 200 m.

no control over dataset ordering in odim hdf5 output

After saving the RAVE vertical profile object to ODIM hdf5 file, the ordering of the dataset1/dataX folders is different then the order in which I assigned them to the RAVE vertical profile. Anders, could you advise what determines this order?

crash on turkish odim file

HDF5-DIAG: Error detected in HDF5 (1.8.13) thread 0:
  #000: H5A.c line 1094 in H5Aread(): unable to read attribute
    major: Attribute
    minor: Read failed
  #001: H5A.c line 1154 in H5A_read(): unable to convert between src and dst datatypes
    major: Attribute
    minor: Feature is unsupported
  #002: H5T.c line 4535 in H5T_path_find(): no appropriate function for conversion path
    major: Datatype
    minor: Unable to initialize object

Segmentation fault: 11

libvol2bird functions cannot be called from pyvol2bird.c

This constructor of a PyVol2Bird object compiles without complaints, however when running python an exception occurs:

Python 2.7.10 (default, Aug 26 2015, 18:17:29) 
[GCC 4.2.1 Compatible Apple LLVM 6.0 (clang-600.0.56)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> import _vol2bird, _raveio
>>> Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
ImportError: dlopen(./_vol2bird.so, 2): Symbol not found: _vol2birdCalcProfiles
  Referenced from: ./_vol2bird.so
  Expected in: flat namespace
 in ./_vol2bird.so

This is because the symbol _vol2birdCalcProfiles is not found, which is called here as follows

int initSuccessful = vol2birdSetUp(volume, alldata) == 0;

vol2birdSetUp is a function of our libvol2bird library, of which the prototype is defined in libvol2bird.h, which is included in the header file of the C code where the exception occurs ( pyvol2bird.h).

Anders, why can't I call functions from the libvol2bird library, and what should be done to fix it?

there is no check whether both radial velocity and reflectivity are available

Implementing this will involve changing this code line:
https://github.com/adokter/baltrad/blob/1d55a12887e90ea008b9bd1e4fb90b02ae5767b1/libvol2bird.c#L834

algorithm doesn't take into account Nyquist interval

Because the algorithm doesn't take consider the Nyquist interval, radial velocity scans are included that are still folded. We want to exclude these scans

Proposed solution:

• use native rave function to extract the Nyquist velocity
• if no Nyquist velocity stored as an attribute, use the radial velocity offset instead
• add a parameter in options.conf that sets the minimum Nyquist velocity for including scans

odim h5 specification in wiki needs update

ubuntu install instructions fail

@jspaaks @LourensVeen: Maarten Reyniers volgde de INSTALL.md, maar liep vast:

email 1 jun 2016:
Hier dus een meer technische mail in verband met de installatie van vol2bird.
Ik ben vol2bird momenteel aan het installeren op een Ubuntu machine (64 bit) met Ubuntu 16.04.
De machine is een virtuele machine die ik volledig nieuw heb aangemaakt met VirtualBox (de host is een Mac).
Ik volg je installatie instructies van https://github.com/adokter/vol2bird/blob/master/INSTALL.md
Het loopt een eerste maal mis bij de installatie van Pillow (pip install Pillow).
De package dat eerst moet geïnstalleerd worden is python-dev (dit zit niet standaard in de distro).
Ergens zou je dus moeten toevoegen
sudo apt-get install python-dev
Dan gaat de installatie van Pillow wel door.

Daarna is er een probleem met de compilatie van rave.
Ik ben geen python specialist, maar blijkbaar is er een conflict met numpy.
Zie hieronder waar de make van rave fout loopt.
De fout heeft iets te maken met de virtual environment, want bij een system-wide installatie van de python packages lukt het me wel om rave te compileren (en aansluitend ook vol2bird). Maar ik zou graag jouw instructies volgen want met de virtual environment is het net iets eleganter.

Heb jij een idee wat ik moet doen om rave verder te kunnen compileren onder de virtual environment?

Vriendelijke groeten,
Maarten

...
make -C pyapi
make[2]: Entering directory '/home/maarten/birdsoft/src/rave/librave/pyapi'
In file included from /home/maarten/birdsoft/.venv/local/lib/python2.7/site-packages/numpy/core/include/numpy/ndarraytypes.h:4:0,
from /home/maarten/birdsoft/.venv/local/lib/python2.7/site-packages/numpy/core/include/numpy/ndarrayobject.h:18,
from /home/maarten/birdsoft/.venv/local/lib/python2.7/site-packages/numpy/core/include/numpy/arrayobject.h:4,
from getpy.h:11,
from getpy.c:7:
/home/maarten/birdsoft/.venv/local/lib/python2.7/site-packages/numpy/core/include/numpy/npy_common.h:319:2: error: #error Must use Python with unicode enabled.
#error Must use Python with unicode enabled.
^
cp: cannot stat '.dep/getpy.d': No such file or directory
/bin/sh: 3: cannot open .dep/getpy.d: No such file
x86_64-linux-gnu-gcc -c -DNDEBUG -g -fwrapv -O2 -Wall -Wstrict-prototypes -pthread -fPIC -DCREATE_ITRUNC -I. -I../toolbox -I/home/maarten/birdsoft/.venv/local/lib/python2.7/site-packages/numpy/core/include/numpy -I/home/maarten/birdsoft/.venv/include/python2.7 -I/home/maarten/birdsoft/opt/hlhdf/include -I/usr/include/hdf5/serial/ getpy.c
In file included from getpy.c:7:0:
getpy.h:10:20: fatal error: Python.h: No such file or directory
compilation terminated.
Makefile:27: recipe for target 'getpy.o' failed
make[2]: *** [getpy.o] Error 1
make[2]: Leaving directory '/home/maarten/birdsoft/src/rave/librave/pyapi'
Makefile:24: recipe for target 'libpyapi' failed
make[1]: *** [libpyapi] Error 2
make[1]: Leaving directory '/home/maarten/birdsoft/src/rave/librave'
Makefile:29: recipe for target 'librave' failed
make: *** [librave] Error 2

pyvol2bird.pth not found on vol2bird install

dbzThres threshold applied in a non-general way

see here and DBZMIN and dbzThresMin defined here

correct warning message wavelength

if wavelength stored at scan level, program gives message that default wavelength will be used, followed by a second message that wavelength of scan 1 will be used.

isnanf not available in Mac OSX

libvol2bird uses isnanf to check for NAN values on floats. I substituted this with the general isnan() function on OSX, but need to check whether this works also on Linux

mac has dyld_library_path instead of ld_library_path

change in install instructions

define and implement new standard profile output for stdout

add user option to select elevation range

Currently all elevations in the volume are included. Useful to have the possibility to exclude scans, especially low elevation scans with a lot of clutter

Quantity identifiers VRAD and DBZ should be made complient with dual-pol quantities

For reflectivity, the algorithm should select DBZ, DBZH, DBZV (first one available)
For radial velocity, the algorithm should select VRAD, VRADH, VRADV

include number of points used for reflectivity calculation in profile

hasGap has too narrow scope

If there is a gap, bird-related profile quantities should be set to NA

python _vol2bird.new method causes segfault

Python 2.7.10 (default, Aug 26 2015, 18:17:29)
[GCC 4.2.1 Compatible Apple LLVM 6.0 (clang-600.0.56)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> import _vol2bird, _raveio
>>> inobj = _raveio.open("test.h5").object
>>> inobj
>>> <PolarVolumeCore object at 0x10969b168>
>>> a=_vol2bird.new(inobj)
>>> Segmentation fault: 11

stdout output does not list radar / site information as a column

make code compatible with Windows

This is necessary if we want to distribute the C-code via R https://cran.r-project.org/

Missing /how elements

In https://github.com/adokter/vol2bird/wiki/ODIM-bird-profile-format-specification I noticed:

• /how/clutterMap is listed twice
• wavelength is missing (we find it in the example file)
• task_version is missing (we find it in the example file)

I have apparently the rights to edit the wiki right away, but I wanted to check.

outlier removal svdfit() is missing

read radar wavelength from ODIM_H5 input file

see rave manual http://git.baltrad.eu/manual/rave/radardefinition_8h.html

Radar wavelength is stored in a RAVE radar definition object, which is not contained in a RAVE polar volume object. Therefore also a radar object needs to be parsed if we want this information available automatically

add new user options to configuration print

bird density not set to zero when radial velocity standard deviation is above threshold

generate output to hdf5

use rave profile object to store the data, and rave file i/o to write the file

data assumed to be in unsigned character

check whether a RAVE field exists that gives the values

read in volume only up to ranges and altitudes needed

Only need to calculate it up to the maximum range considered in the profiles, truncating may speed up the code

release volume in vol2bird.c

The first thing I noticed is that there seems to be some memory leaks.
I could not verify my finding since there are other problems but if I
read the code correctly you for example seem to be missing a
RAVE_OBJECT_RELEASE of the object you assign at line 100 in
vol2bird.c. You can easilly active debug printouts by import _rave in
the script you are running:

missing data values are assigned as lowest possible value in Z and VRAD

pyvol2bird install error

On OSX we get

.dep/pyvol2bird.P:1: *** multiple target patterns. Stop

Is the problem that .dep folder is not cleaned? What is .dep folder anyhow?

move functionality from constants.h to options.conf

1. birdRadarCrossSection here / SIGMABIRD here
2. stdDevMinBird here / STDEVBIRD here
   To include S-band radars (not C-band as initially designed for), also these options should be user-adjustable:
3. cellDbzMin here / DBZCELL here
4. dbzmax here /DBZMAX here

separate specification of range and azimuth intervals for svdfit() and profile averaging

Take from earlier repository

It is desirable to have the flexibility to specify separate range intervals for the averaging of profile reflectivity values (set by rangeMin and rangeMax), and the range interval over which the svdfit() function fits a radial velocity model (set by RANGMINSTDEV and RANGMAXSTDEV, see this code for the range specification and this code for the azimuth specification). interval (RANGMINSTDEV, RANGMAXSTDEV) should be contained in the interval (rangeMin,RangeMax)

I would like to specify separately

1. the range interval used by svdfit()
2. the range interval and azimuth interval which is used to average values into profiles, i.e. to have the possibility that this is a subsector of the full surveyed area.

decide on encoding of nodetect and nodata

Nodetect and nodata are now both encoded by a NaN, but one of the two needs to get a different value.

pyvol2bird does not yet read config file