jgcri / moirai Goto Github PK

we currently have coarse resolution carbon data from literature for soil and vegetation that represents "potential" ecosystem carbon levels - not sure if all the source data really qualify here, but it is all we have. these data are output for each countryXgluXlandtype as a reference veg value

need to identify some spatial carbon data layers and read them into moirai, process them according to the land types, and determine an appropriate output format that is useful for gcam and its data system.

it would make sense to output these new data in the same format as the reference veg data, but as a new file that represents present-day values; have to denote which years this corresponds to

more to come...

The task is to replace the current protected vs non-protected land designations with a set of suitability classes. This requires:

Read in the epa data and get in onto the working grid - this will require addition of the gdal library and the creation of aggregated data arrays representing fractions of land area in each working grid cell assigned to each suitability class. These fractional data should be determined so that each class represents an exclusive fraction and that the fractions add up to 1 for land area. These fractional data can be stored in one grid array for each class, in place of the single protected area thematic array. The current read function to be replaced is read_protected.c

The epa data are organized differently than desired. Each file has binary values denoting whether the pixel meets the criteria or not. The base layer L1 simply denotes suitable or not-suitable, regardless of any other designation. The other layers subtract or add pixels from this layer, so to get to the desired format the epa layers will have to be differenced in appropriate ways. Note that the protected area data are not available for the not-suitable class; to get these data we would have to get the actual IUCN protected area data and preprocess it and read it in. I discussed this with Page and we decided to just go with the available epa data for now.

There will be five distinct classes, rather than the two current ones. The function proc_land_type_area.c is where these classes are assigned to areas that are aggregated to land types within countryXglu boundaries. There are 10 possible values for classes (0-9).Currently, non-protected (i.e., suitable) is assigned a value of 2, and protected is assigned a value of 1. The new classes constitute a mutually exclusive set and will be as follows:
1 - not suitable
2 - suitable and not protected
3 - suitable and iucn protected categories 1a, 1b, and 2 that have not been deforested
4 - suitable and iucn protected categories 1a, 1b, and 2 that have been deforested
5 - suitable and iucn protected categories 3, 4, and 5
We will recommend that the default suitability for GCAM is class 2, which is the highest level of protection (least amount of suitable land). The default EPA suitable layer is class 2 plus class 4 plus class 5 because they don't expect iucn categories 3-5 to really be protected and they add in the deforested protected area as suitable because it has been deforested.

The areas for output will have to be calculated slightly differently because now there will be fractional land area values for suitability categories instead of the current whole pixel assignment. In other words, rather than just adding whole pixel land area to one or the other for a given land type, each land type will be distributed to the 5 suitability classes based on the fractional values for each pixel. The output structure will stay the same, as these classes fit into the single digit code, and so the category mapping and number of output records will be expanded.

Note that proc_land_type_area.c takes at least half the run time of the total program, so this function will be the real efficiency test for porting to R.

this can then be used to create shapefiles of the desired boundaries
or maybe this is a few files, each for a specific boundary set
maybe this is just formalizing some of the diagnostic raster outputs

The code is designed to match the ISAM land cover in area. The ISAM data were derived from the same HYDE data used to do the ISAM downscaling, but apparently there are some inconsistencies still. In some cases the ISAM vegetation area is met, but there are still fine-res pixels that need assignment due to there being less crop/pasture/urban area than in ISAM. In these cases an "unknown" veg type is assigned. It is possible to reduce these unknowns by reverting back to the potential vegetation and deviating from the ISAM area restrictions.
This is another reason why we need our own historical reconstruction code.

the sage harvested area is applied to the hyde crop area, altering the multi-cropping ratios
first normalize the harvested area, then apply it to the hyde crop area to maintain the multi-cropping ratios in the sage data

We need carbon densities for Cropland, Pastures and Urban land. We have data for the same from SoilGrids and Spawn et al. along with land cover from ESA. Updating these would need two separate steps,

1. Updating the C code to get land cover for a given year.
2. Updating the preprocessing to make sure we get values for each of the same in the input data.

For the update version 3.1.0, I use the command line (git clone http://github.com/JGCRI/Moirai.git to clone. In the download file I can't find the moirai.xcodeproj file like pre-version, and just found the file named moirai.exe.stackdump. And another question is about the size of the total size which is 48GB, I'm not sure the size is correct or not. Because the instruction mentioned that 115 GB of free disk space is needed to obtain the software.

Recalibrating the livestock and forestry land rent should probably use the pasture and forest area in the recalibration year.
Check to see if this makes sense compared to using the year 2000 physical area.

I recall why 2000 was done separately. It is because there is a year 2000 physical cropland layer that goes along with the year 2000 crop harvested area and the mirca data as well. This was also used for the livestock and forestry land rent calculation.
But in general the land type area is determined by the hyde data first. So there may still need to be some special calculation for the year 2000 physical cropland and pasture area (for used by other functions), but the output land area is not based on these year 2000 layers.
In any case, store the needed years for any further calculations during this annual calculation. This includes the year 2000 special data, year 2010 for the carbon data, and the year designated for recalibration.

Set up an R package to begin development for version 4. Include initial build test and set up Travis-CI.

These can be added from latest SoilGrids and will only require R code changes. The 5 arcmin maps are required for demeter.

• Change "basin" to "glu" in all output file names, metadata
• Make sure last lines in all C code files end with the return line
• Change Mgc/ha to MgC/ha in carbon plot code
• Add documentation to diagnostics folder as .md file
• Add documentation to main md file with updates related to 3p1
• Add R code to get revised fao_ctry_rast.bil
• Change C makefile -std to 11
• Add bash script to calculate soil carbon from soil grids

are the fractions and yields the same?
can't check every file, but check a few

Links in README are broken. Paths need to be fixed.

it appears that there are only 384 valid land regionXglu land units, but moirai describes 397 (and in some cases GCAM may output 392?). The extra ones are caused by slight misalignment in the raster maps of basins and countries. Along these boundaries a few pixels are created that fall into two additional units (c1Xg2 and c2Xg1), where only the main two units should exist (c1Xg1 and c2Xg2).

Also change the batch of pixels in Namibia that is assigned to South Africa (Walvis Bay), to Namibia. Some older data must have slipped in here to cause this.

fix this for the managed area carbon release

In order to develop shape files for land/noland from moirai we will require raster boundary outputs for valid land cells and for valid country/region/basin cells without land from moirai. We currently get the former but not the latter. One challenge with respect to this is that our country boundaries inputs (fao_ctry_rast.bil) does not contain data for inland water bodies (i.e. the raster contains holes). So, to get the above mentioned output we will need to do 2 things,

1. Update a hole filled raster for fao_ctry_rast.bil
2. Update get_land_cells.c with code to get boundaries for valid country/region/glu without land.

Hello, I'm interested in calibrating the forest and arable land areas for the years 2000-2015 using more precise national statistics. Could you please provide any guidance or tips for Moirai starters?

Thank you in advance for suggestion!

The mirca data are harvested area and are based on the same circa 2000 crop data as used by moirai.
This potentially makes the mirca data eligible for recalibration to other years.

moirai outputs are unprojected and are at 5 arcmins. this comes to ~10 kms based on where a pixel is located relative to the equator. Need to describe this a bit more carefully in all input code files and output metadata.

Based on this- following details are probably required,

1. Look through all .c files and update definitions
2. Look through readme and edit the definitions
3. Make sure metadata is all consistent.

it needs to be:
UrbanLand with a capital L

this year is currently hardcoded in the header file
it should be added as an input argument in the input file.

Remove GitLFS due to large data required for this repository and extremely long clone times. Instead replace with functionality that permits installing from an archived, minted source like Zenodo (see a Python example for reference: https://github.com/JGCRI/xanthos/blob/remove-lfs/xanthos/install_supplement.py).

The contents of the zipped files that are extracted for each Harvested Area Yield 175 Crops are not cleaned up after the run and thus cause a failure on subsequent runs.

the zero values are aligned before rounding to the integer, but after rounding there are a few crops that have one zero and one non-zero values for these two outputs

There are a couple of typos in protected area names in the code file write_glu_mapping.c. These occur on line 117. The two typos are,

"SuitbaleHighProtectionDeforested" which should be "SuitableHighProtectionDeforested"

"SuitableLow Protection" which should be "SuitableLowProtection

We currently aggregate above and below ground biomass carbon when generating vegetation carbon numbers. This is mainly for speed. But given that the ratio of above to below ground biomass would stay the same across all states of carbon, it may be worth it to use a ratio to break this out.

Before we do this, we would need to check that the ratio is actually the same across all states (mean, median,q1,q3 etc). If that is the case, then we can calculate the ratio in read_veg_carbon.c and use this ratio in proc_ref_veg_carbon.c to do the breakout.

Hi,

I'm one of your JORS reviewers and I'm coming across issues with your makefile, these appear to be internal errors with the code and not system related. I apologize, I would debug further, but I'm caught up with a grant at the moment. I would like to be able to run your code before signing off on my review.

Here's the output of the make command:

$LDS_OBJS is [aggregate_crop2gcam.o aggregate_use2gcam.o calc_harvarea_prod_out_crop_aez.o calc_refveg_area.o calc_rent_ag_use_aez.o calc_rent_frs_use_aez.o copy_to_destpath.o get_aez_val.o get_cell_area.o get_in_args.o get_land_cells.o get_systime.o init_moirai.o moirai_main.o parse_utils.o proc_land_type_area.o proc_lulc_area.o proc_mirca.o proc_refveg_carbon.o proc_water_footprint.o read_aez_new.o read_aez_new_info.o read_aez_orig.o read_country87_info.o read_country_fao.o read_country_info_all.o read_crop_info.o read_cropland_sage.o read_harvestarea_fao.o read_hyde32.o read_land_area_hyde.o read_land_area_sage.o read_lu_hyde.o read_lulc_info.o read_lulc_isam.o read_lulc_land.o read_mirca.o read_potveg.o read_prodprice_fao.o read_production_fao.o read_protected.o read_region_info_gcam.o read_rent_orig.o read_sage_crop.o read_soil_carbon.o read_use_info_gtap.o read_veg_carbon.o read_water_footprint.o read_yield_fao.o write_csv_float2d.o write_csv_float3d.o write_glu_mapping.o write_harvestarea_crop_aez.o write_production_crop_aez.o write_raster_float.o write_raster_int.o write_raster_short.o write_rent_use_aez.o write_text_char.o write_text_int.o]

gcc -c /Users/jrca253/tmp/moirai/src/read_lu_hyde.c -o /Users/jrca253/tmp/moirai/obj/read_lu_hyde.o -O3 -std=c99 -I/Users/jrca253/tmp/moirai/include

/Users/jrca253/tmp/moirai/src/read_lu_hyde.c:107:27: error: no member named 'hist_crop_rast_name' in 'args_struct'
strcat(fname, in_args.hist_crop_rast_name);
~~~~~~~ ^

/Library/Developer/CommandLineTools/SDKs/MacOSX10.14.sdk/usr/include/secure/_string.h:131:33: note: expanded from macro 'strcat'
__builtin___strcat_chk (dest, __VA_ARGS__, __darwin_obsz (dest))
^~~~~~~~~~~

/Users/jrca253/tmp/moirai/src/read_lu_hyde.c:110:76: error: no member named 'hist_crop_rast_name' in 'args_struct'
fprintf(fplog,"Failed to open file %s: read_lu_hyde()\n", in_args.hist_crop_rast_name);
~~~~~~~ ^
/Users/jrca253/tmp/moirai/src/read_lu_hyde.c:122:25: error: no member named 'hist_crop_rast_name' in 'args_struct'
in_args.hist_crop_rast_name, num_read, NUM_CELLS);
~~~~~~~ ^

/Users/jrca253/tmp/moirai/src/read_lu_hyde.c:130:27: error: no member named 'hist_pasture_rast_name' in 'args_struct'
strcat(fname, in_args.hist_pasture_rast_name);
~~~~~~~ ^
/Library/Developer/CommandLineTools/SDKs/MacOSX10.14.sdk/usr/include/secure/_string.h:131:33: note: expanded from macro 'strcat'
__builtin___strcat_chk (dest, __VA_ARGS__, __darwin_obsz (dest))
^~~~~~~~~~~

/Users/jrca253/tmp/moirai/src/read_lu_hyde.c:133:76: error: no member named 'hist_pasture_rast_name' in 'args_struct'
fprintf(fplog,"Failed to open file %s: read_lu_hyde()\n", in_args.hist_pasture_rast_name);
~~~~~~~ ^

/Users/jrca253/tmp/moirai/src/read_lu_hyde.c:145:25: error: no member named 'hist_pasture_rast_name' in 'args_struct'
in_args.hist_pasture_rast_name, num_read, NUM_CELLS);
~~~~~~~ ^
/Users/jrca253/tmp/moirai/src/read_lu_hyde.c:153:27: error: no member named 'hist_urban_rast_name' in 'args_struct'
strcat(fname, in_args.hist_urban_rast_name);
~~~~~~~ ^

/Library/Developer/CommandLineTools/SDKs/MacOSX10.14.sdk/usr/include/secure/_string.h:131:33: note: expanded from macro 'strcat'
__builtin___strcat_chk (dest, __VA_ARGS__, __darwin_obsz (dest))
^~~~~~~~~~~

/Users/jrca253/tmp/moirai/src/read_lu_hyde.c:156:76: error: no member named 'hist_urban_rast_name' in 'args_struct'
fprintf(fplog,"Failed to open file %s: read_lu_hyde()\n", in_args.hist_urban_rast_name);
~~~~~~~ ^
/Users/jrca253/tmp/moirai/src/read_lu_hyde.c:168:25: error: no member named 'hist_urban_rast_name' in 'args_struct'
in_args.hist_urban_rast_name, num_read, NUM_CELLS);
~~~~~~~ ^

9 errors generated.
make: *** [/Users/jrca253/tmp/moirai/obj/read_lu_hyde.o] Error 1

Dear all, I have a question about adding new basins to GCAM. Sorry if this is not the right place, but I did not know where else to ask.

I want to create new basins because in some cases current GCAM basins include very differents regions. Example: Northern Chile, as in current GCAM basins, includes Atacama Desert and Mediterranean Central Chile, hence, water availability impacts could be under/overestimated as most of the agricultural activities is taken place in Central Chile.

1. Is it possible to add new GCAM basins using Moirai?
2. I know that rgcambreakout can create new economic regions, but I have not seen something like this for basins o gcam land regions. Does someone knows an article or documentation that can help to do something like this?

Thanks!

This is the main goal of the project: improve the land cover/use distribution.
Initially, these data will be aggregated to the current GCAM land types.
Together we will determine the new land type category set that GCAM will use and that LDS will provide.

Calculate these outputs based on REF_CARBON_YEAR (2010) instead of REF_YEAR (2000)