magpiemodel / magpie Goto Github PK

Dear all,
hope that the pandemics do not hurt you.
And that you have been able to progress on your researches.
In particular, on data, something new ?
Best

The Slurm submit scripts set --output (stdout) flag to full.log, however the submit.R script also sets the -lf parameter to full.log

e.g. here
https://github.com/magpiemodel/magpie/blob/master/scripts/run_submit/submit_standby.sh
and here
https://github.com/magpiemodel/magpie/blob/master/scripts/run_submit/submit.R#L25

It looks like this causes either a) complete job failure (two attempts to open the same file for writing) or b) mixed Slurm and GAMS output.

Suggest setting #SBATCH --output to a unique value for the run, e.g. full-%x-%j.log (include jobname and job ID in the filename)

-c

Demand for forestry products is calculated twice in the model; one is not used, and in postprocessing overwritten by the second version of the calculation. This should be cleaned up in gams, linking forestry demand in materail demand or bioenergy demand

warning("Double structure for forestry products. Estimates in material demand and supply chain losses are overwritten")
out[, , forestry_products] <- 0
out[, , forestry_products][,,"other_util"] <- forestry

We have included if conditions in the som module (presolve and postsolve) for the variable vm_land (> 10e-20) because otherwise some runs produced System Error (13 GAMS error) meassages. This has to be investigated further.

Under certain conditions vm_bioenergy_utility, which is subsidizing first generation bioenergy production, can lead to higher benefits than costs independent of the quantity of produced bioenergy, leading to a infinitely negative goal function.

Observed was this behavior when the model is run with trade realization "exo" and default data inputs.

The problematic lines of code are found here:

https://github.com/magpiemodel/magpie/blob/3803f1aea82efc215218404509d7150bed2ecf3c/modules/60_bioenergy/1stgen_priced_dec18/equations.gms#L74:L75

My understanding is that these subsidies were introduced to prevent overproduction of other good in order to use their secondary products for bioenergy production. My feeling is that we need to get rid of the subsidy and need to treat the potential overproduction of other goods differently.

Both yield and factor cost realization use the same data at the moment but channeled into magpie as two different input files.
task would be reading in this file for all yield realization and changing it to an interface to be used in the factor costs.

For example, I am playing a game and it is 60fps no problems. Suddenly game freezes for a second (the reason is not important) and magpie starts to drop utilizing GPU fully resulting in taking lower fps. Restarting magpie fixes it temporarily but it comes back again as soon as a freeze or stutter happens. Tried a few different games including an emulator but it happens in all of them.

Checked FPS counter display, only to find out the game running at 200+ fps was limited to 60fps. Is there a way to change this limit?

Dear colleagues, there is a bug in the REMIND-MAgPIE coupling, that lead to wrong ghg prices in MAgPIE. As I guess it must be fixed in this repo, I placed the issue here.

An example is /p/tmp/oliverr/debugging/magpie/output/C_h_cpol_bIT-mag-5, whose config.Rdata states cfg$mute_ghgprices_until = "y2150", but the report.mif lets you clearly seen huge Prices|GHG|… before.

My suspicion how this can happen (edit: probably wrong, see below):

The MAgPIE scripts

1. locks the model
2. calculates the ghgprices using the functions getReportData and .emission_prices and saves them into ./modules/56_ghg_policy/input/f56_pollutant_prices_coupling.cs3.
3. unlocks the model
4. starts the run that at some point loads the …coupling.cs3

I guess that once the model is unlocked, the next scenario with different setting overwrites f56_pollutant_prices_coupling.cs3 before it is loaded by the first scenario.

The reason this didn't show up in the past is probably that it is quite rare for REMIND-MAgPIE coupled runs to queue for MAgPIE, as the REMIND run take so long. It happened twice in my case, but only when I restarted the coupling to get more coupling iterations, and then all runs more or less simultaneously wanted to run mag-5 and had to queue.

Ideas I came up with to avoid that:

• very ugly: add Sys.sleep(300) to emission_prices ;)
• save f56 in the output folder? Or with a scenario stamp in the filename?
• …

For some applications it would be useful if MAgPIE could be fixed to the values of existing GDX files until a certain year.
Let's assume you have a run that is infeasible in 2050. Now you want to restart the run but keep the results until 2020 from the first run that was infeasible only in 2050 (based on magpie_y1995.gdx ... magpie_y2020.gdx). This makes sure that the results for 1995-2020 are identical to other model runs from the same scenario set. From 2025 onwards the model should use the gdx files from the first run for all years until 2045 as starting point (cfg$gms$s_use_gdx <- 2). This will push the solver to a slightly different trajectory from 2025 onwards and might lead to a feasible solution for 2050 and beyond.

Note 1: Re-running the model for the period 1995-2020 without gdx files (or cfg$gms$s_use_gdx <- 0) should yield the same result but of course needs much more time than just fixing all variables based on existing gdx files.
Note 2: Re-running the model for the period 1995-2020 with gdx files and cfg$gms$s_use_gdx <- 2 produces different results.

#154 (comment)

The merge #614 deleted/ renamed the switch
cfg$gms$c31_past_suit_scen from default.cfg.
However, in the scenario_config.cfg this switch still exists.
This leads to an error when using gms::setScenario.

• missing general comment in realization.gms (A bit more detail on the purpose of this realization would be helpful. Maybe add Limitations also. I think it's better to have the realization descriptions independent of other realizations, so that they are still clear if the other (referred to) realization is deleted one day.)

• equation comments:

  • L9: is "aggregation" the right word here? I don't understand the sentence entirely...
  • L12: Why do we want to replicate the structure? The end result is what matters, no?
  • utility?

• presolve

  • I always thought that when we set .fx we don't (shouldn't) set .up and .lo
  • using i or i2 in presolve? (it is mixed here)

I noticed some of the output scripts load libraries that are not required and actually failed to load causing an error. For example, the LandusePlots.R script throws error when it attempts to load the libraries moinput and mrvalidation. By commenting them out, I was able to make the script run successfully and produce output. It seems these library calls were added during development and were not deleted (moinput is even called twice). Since these are not available to non-PIK users, they should be removed if not required.

It would be easy to go through all the output scripts and simply check if the libraries called upon are all required or are just left over from development.

Great that PIK open sourced MAgPIE! But you unfortunately chose an license incompatible with GAMS.

Version 3 of the GPL family of licenses is incompatible with GAMS code because Richard Stallman introduced a clause aimed at preventing its application to Java. Back when version 3 was being drafted, Java was still proprietary and the new version of the GPL requires that the implementing language has either a recognized international standard (like C++) or an open source implementation. As I understand it, GAMS currently fulfills neither of those requirements.

Quoting from the SPDX site in relation to GPL‑3.0:

A "Standard Interface" means an interface that either is an official standard defined by a recognized standards body, or, in the case of interfaces specified for a particular programming language, one that is widely used among developers working in that language.

Note that there is no AGPL‑2.0 to apply instead. I suggest you drop the "Affero" variant and use a GPL‑2.0‑only license. Or simply drop back to a permissive license like MIT or ISC.

I undertook some analysis on selecting licenses for GAMS and I am of the general view that a copyleft license is not necessary, let alone the (incompatible) network variant:

• Morrison, Robbie (April 2018). "Energy system modeling: public transparency, scientific reproducibility, and open development". Energy Strategy Reviews. 20: 49–63. ISSN 2211-467X. doi:10.1016/j.esr.2017.12.010. Open access.

I also contacted the Free Software Foundation on this matter and suggested they add this issue to their FAQ. I had some traffic with their staff lawyer but FSF never added the question.

Happy to provide more details or put you in touch with people who served on the GPL version 3 drafting committee. And, as I said, fantastic that PIK is heading open source.

Dear colleagues,

while running NGFS scenarios, I noticed the following:

Eight Emissions|CO2|* variables differ between scenarios already in 2005, 2010, 2015 and 2020, and I guess they shouldn't.

I also found that pattern in the MAgPIE automated tests:

library(quitte); library(tidyverse); options(width = 160);
files <- file.path("/p/projects/landuse/tests/magpie/output", c("weeklyTests_SSP2EU-NDC", "weeklyTests_SSP2EU-PkBudg650"), "report.mif")
d <- quitte::as.quitte(files)
fails <- piamInterfaces::fixOnRef(files[[2]], files[[1]], startyear = 2025, ret = "fails")
for (f in levels(fails$variable)) {
  message("\n", f)
  d %>%
    filter(.data$region == "World", .data$variable == f, .data$period < 2022, .data$period > 2000) %>%
    df_variation() %>%
    droplevels() %>%
    pivot_wider(names_from = period) %>%
    print()
}

### Incorrect fixing for 8 variables (grouped below) for model=MAgPIE and scenario=weeklyTests_SSP2EU-PkBudg650
  group                          variables period              reldiff
1 Emissions|CO2|Land                     4 2005,2010,2015,2020 17.3 %
2 Emissions|CO2|Land Carbon Sink         2 2005,2010,2015,2020 9.09 %
3 Emissions|CO2|Land RAW                 2 2005,2010,2015,2020 36.1 %

Emissions|CO2|Land
  scenario                     `2005` `2010` `2015` `2020`
1 weeklyTests_SSP2EU-NDC        2025.  1644.  1993.  1622.
2 weeklyTests_SSP2EU-PkBudg650  2015.  1654.  1993.  1672.

Emissions|CO2|Land Carbon Sink|Cumulative|Grassi
  scenario                     `2005` `2010` `2015` `2020`
1 weeklyTests_SSP2EU-NDC        -18.6  -43.1  -68.4  -96.2
2 weeklyTests_SSP2EU-PkBudg650  -18.7  -43.1  -68.4  -96

Emissions|CO2|Land Carbon Sink|Grassi|Managed Land|Managed Forest
  scenario                     `2005` `2010` `2015` `2020`
1 weeklyTests_SSP2EU-NDC        -3720  -4900  -5060  -5570
2 weeklyTests_SSP2EU-PkBudg650  -3730  -4890  -5060  -5520

Emissions|CO2|Land RAW
  scenario                     `2005` `2010` `2015` `2020`
1 weeklyTests_SSP2EU-NDC        1596.  2135.  2797.  1313.
2 weeklyTests_SSP2EU-PkBudg650  1586.  2145.  2797.  1363.

Emissions|CO2|Land RAW|+|Indirect RAW
  scenario                     `2005` `2010` `2015` `2020`
1 weeklyTests_SSP2EU-NDC        -3720  -4900  -5060  -5570
2 weeklyTests_SSP2EU-PkBudg650  -3730  -4890  -5060  -5520

Emissions|CO2|Land|+|Indirect
  scenario                     `2005` `2010` `2015` `2020`
1 weeklyTests_SSP2EU-NDC        -3720  -4900  -5060  -5570
2 weeklyTests_SSP2EU-PkBudg650  -3730  -4890  -5060  -5520

Emissions|CO2|Land|Cumulative
  scenario                     `2005` `2010` `2015` `2020`
1 weeklyTests_SSP2EU-NDC         35.9   44.1   54.1   62.2
2 weeklyTests_SSP2EU-PkBudg650   35.8   44.1   54.1   62.4

Emissions|CO2|Land|Cumulative|+|Indirect
  scenario                     `2005` `2010` `2015` `2020`
1 weeklyTests_SSP2EU-NDC        -18.6  -43.1  -68.4  -96.2
2 weeklyTests_SSP2EU-PkBudg650  -18.7  -43.1  -68.4  -96

A similar result can be obtained for our recent NGFS runs using the magpie release 4.7.0. They can be found in /p/tmp/oliverr/NGFS_v5/remind-2024-02-27/magpie. (Please don't run any experiment etc. in this folder.)

library(quitte); library(tidyverse); options(width = 160);
d <- as.quitte(Sys.glob("/p/tmp/oliverr/NGFS_v5/remind-2024-02-27/magpie/output/C_*-mag-5/report.mif"))
fails <- piamInterfaces::fixOnRef(d, "C_h_cpol-mag-5", startyear = 2025, ret = "fails")

for (f in levels(fails$variable)) {
  message("\n", f)
  d %>%
    filter(.data$region == "World", .data$variable == f, .data$period < 2022) %>%
    df_variation() %>%
    droplevels() %>%
    pivot_wider(names_from = period) %>%
    print()
}

Emissions|CO2|Land
  scenario          `2005` `2010` `2015` `2020`
1 C_d_delfrag-mag-5   738.  -52.4  -91.7  -605.
2 C_d_strain-mag-5    858.   47.6   48.3  -475.
3 C_h_cpol-mag-5      858.   47.6   48.3  -475.
4 C_h_ndc-mag-5       858.   47.6   48.3  -475.
5 C_o_1p5c-mag-5      848.   57.6   48.3  -425.
6 C_o_2c-mag-5        738.  -52.4  -91.7  -605.
7 C_o_lowdem-mag-5    848.   57.6   48.3  -425.
8 C_SSP2-Base-mag-5   858.   47.6   48.3  -475.

Emissions|CO2|Land Carbon Sink|Cumulative|Grassi
  scenario          `2005` `2010` `2015` `2020`
1 C_d_delfrag-mag-5  -19.2  -44.2  -70.2  -98.7
2 C_d_strain-mag-5   -18.6  -43.1  -68.4  -96.2
3 C_h_cpol-mag-5     -18.6  -43.1  -68.4  -96.2
4 C_h_ndc-mag-5      -18.6  -43.1  -68.4  -96.2
5 C_o_1p5c-mag-5     -18.7  -43.1  -68.4  -96
6 C_o_2c-mag-5       -19.2  -44.2  -70.2  -98.7
7 C_o_lowdem-mag-5   -18.7  -43.1  -68.4  -96
8 C_SSP2-Base-mag-5  -18.6  -43.1  -68.4  -96.2

Emissions|CO2|Land Carbon Sink|Grassi|Managed Land|Managed Forest
  scenario          `2005` `2010` `2015` `2020`
1 C_d_delfrag-mag-5  -3840  -5000  -5200  -5700
2 C_d_strain-mag-5   -3720  -4900  -5060  -5570
3 C_h_cpol-mag-5     -3720  -4900  -5060  -5570
4 C_h_ndc-mag-5      -3720  -4900  -5060  -5570
5 C_o_1p5c-mag-5     -3730  -4890  -5060  -5520
6 C_o_2c-mag-5       -3840  -5000  -5200  -5700
7 C_o_lowdem-mag-5   -3730  -4890  -5060  -5520
8 C_SSP2-Base-mag-5  -3720  -4900  -5060  -5570

Emissions|CO2|Land RAW
  scenario          `2005` `2010` `2015` `2020`
1 C_d_delfrag-mag-5   394.   766. -210.   -645.
2 C_d_strain-mag-5    514.   866.  -69.7  -515.
3 C_h_cpol-mag-5      514.   866.  -69.7  -515.
4 C_h_ndc-mag-5       514.   866.  -69.7  -515.
5 C_o_1p5c-mag-5      504.   876.  -69.7  -465.
6 C_o_2c-mag-5        394.   766. -210.   -645.
7 C_o_lowdem-mag-5    504.   876.  -69.7  -465.
8 C_SSP2-Base-mag-5   514.   866.  -69.7  -515.

Emissions|CO2|Land RAW|+|Indirect RAW
  scenario          `2005` `2010` `2015` `2020`
1 C_d_delfrag-mag-5  -3840  -5000  -5200  -5700
2 C_d_strain-mag-5   -3720  -4900  -5060  -5570
3 C_h_cpol-mag-5     -3720  -4900  -5060  -5570
4 C_h_ndc-mag-5      -3720  -4900  -5060  -5570
5 C_o_1p5c-mag-5     -3730  -4890  -5060  -5520
6 C_o_2c-mag-5       -3840  -5000  -5200  -5700
7 C_o_lowdem-mag-5   -3730  -4890  -5060  -5520
8 C_SSP2-Base-mag-5  -3720  -4900  -5060  -5570

Emissions|CO2|Land|+|Indirect
  scenario          `2005` `2010` `2015` `2020`
1 C_d_delfrag-mag-5  -3840  -5000  -5200  -5700
2 C_d_strain-mag-5   -3720  -4900  -5060  -5570
3 C_h_cpol-mag-5     -3720  -4900  -5060  -5570
4 C_h_ndc-mag-5      -3720  -4900  -5060  -5570
5 C_o_1p5c-mag-5     -3730  -4890  -5060  -5520
6 C_o_2c-mag-5       -3840  -5000  -5200  -5700
7 C_o_lowdem-mag-5   -3730  -4890  -5060  -5520
8 C_SSP2-Base-mag-5  -3720  -4900  -5060  -5570

Emissions|CO2|Land|Cumulative
  scenario          `2005` `2010` `2015` `2020`
1 C_d_delfrag-mag-5   24.9   24.7   24.2   21.2
2 C_d_strain-mag-5    25.5   25.8   26.0   23.6
3 C_h_cpol-mag-5      25.5   25.8   26.0   23.6
4 C_h_ndc-mag-5       25.5   25.8   26.0   23.6
5 C_o_1p5c-mag-5      25.5   25.8   26.0   23.9
6 C_o_2c-mag-5        24.9   24.7   24.2   21.2
7 C_o_lowdem-mag-5    25.5   25.8   26.0   23.9
8 C_SSP2-Base-mag-5   25.5   25.8   26.0   23.6

Emissions|CO2|Land|Cumulative|+|Indirect
  scenario          `2005` `2010` `2015` `2020`
1 C_d_delfrag-mag-5  -19.2  -44.2  -70.2  -98.7
2 C_d_strain-mag-5   -18.6  -43.1  -68.4  -96.2
3 C_h_cpol-mag-5     -18.6  -43.1  -68.4  -96.2
4 C_h_ndc-mag-5      -18.6  -43.1  -68.4  -96.2
5 C_o_1p5c-mag-5     -18.7  -43.1  -68.4  -96
6 C_o_2c-mag-5       -19.2  -44.2  -70.2  -98.7
7 C_o_lowdem-mag-5   -18.7  -43.1  -68.4  -96
8 C_SSP2-Base-mag-5  -18.6  -43.1  -68.4  -96.2

The following settings are used:

d_delfrag: SSP2EU|NDC|nocc_hist|rcp2p6
d_strain : SSP2EU|NDC|nocc_hist|rcp4p5
h_cpol   : SSP2EU|NPI|nocc_hist|rcp4p5
h_ndc    : SSP2EU|NDC|nocc_hist|rcp4p5
o_1p5c   : SSP2EU|NDC|nocc_hist|rcp1p9
o_2c     : SSP2EU|NDC|nocc_hist|rcp2p6
o_lowdem : SSP2EU|NDC|nocc_hist|rcp1p9|NGFS_o_lowdem
SSP2-Base: SSP2EU|NPI|nocc_hist|rcp4p5

• As all runs using the same rcp setting are identical, I suspect that this setting is causing the problem.

Otherwise the budget calculations in the food demand model lead to an increased food demand in countries with fish consumption.
By default, a global cost term similar to the price would make sense

Which module: Livestock? New module fish? @isabelle?

@flohump @tscheypidi is it important to have no time stamp in the folder name of the weekly test runs? The current implementation of the testing suite will not be able to distinguish them from each other.

• correct in-code documentation in equations.gms w.r.t. v42_irrig_eff
• go through realization description, declarations etc. to make sure variables are described correctly