irena-flextool / flextool

IRENA FlexTool is an energy and power systems model for understanding the role of variable power generation in future energy systems.

IRENA FlexTool is an energy and power systems model for understanding the role of variable power generation in future energy systems. It performs capacity expansion planning as well as operational planning.

This is IRENA FlexTool v3.x.x (see current version from RELEASE.md) in beta testing. Report any bugs or difficulties in the issue tracker. The previous version of IRENA FlexTool can be found in https://www.irena.org/energytransition/Energy-System-Models-and-Data/IRENA-FlexTool.

Introduction and installation

Note

Spine Toolbox has received a major upgrade 29th of April 2024. Next time you update FlexTool, update Spine Toolbox first. Follow the upgrade instructions of your Toolbox installation method.

Installation, user guide and documentation can be found at: https://irena-flextool.github.io/flextool/. User guide and documentation are under development.

flextool's Issues

Energy not served: flextool creates energy "from nothing"

Hello,
To keep track of what we discussed yesterday, here is a summary.
Looking forward for your feedback!

Assumptions about existing units

For pathway capacity expansion problems, FlexTool is currently assuming that all existing units will stay in the system forever unless 'retire_forced' parameter is used to remove them in a future period.

Better assumption would be that if the unit has a 'lifetime', then it would be automatically retired after lifetime. There should possibly be a method to choose the behavior (instead of relying on parameter value given or not given).

Implement minimum up and down times

Frequency plots for ramps

It would be nice to have frequency plots for ramping. Best to do it for groups.

E.g.

FlexTool 2 to FlexTool 3 converter

Hi team,
hope to find you well. Just out of curiosity, is there any open tool for the conversion of FlexTool 2 input files to FlexTool 3 type?
Thanks,
Ioannis

Marginal cost of additional investment

Marginal cost of additional investment should be an output

Error importing scenario coal_wind_ev : UNIQUE constraint failed: alternative.name

When I:

• purge totally Input_data (Select all)
• run Initialize to import Init to Input_data

I obtain an error for coal_wind_ev scenario:

FlexTool3, coal_wind_ev - Init
[07-12-2022 12:26:50] ***Executing Merger Initialize***
[07-12-2022 12:27:21] Import errors. Logfile
[07-12-2022 12:27:47] Executing Merger Initialize finished

In logfile:
DBAPIError while inserting alternative items: ('UNIQUE constraint failed: alternative.name',)

Fail nicely if no scenario chosen for FlexTool

If the user has not chosen FlexTool tool or no scenarios in the filter after init.sqlite, then FlexTool fails in ways that are not obvious to the user. We should find ways to make sure that cannot happen. (#36)

Capacity margin equation needs more thinking when used with storages

Flows between a node within a capacity constraint group and a storage (presented by another node) are not currently restricted based on the storage contents.
Also, at present, nodes with storage that are part of the capacity constraint group include the changes in v_state. Not sure if this ok in all circumstances.
Maybe there needs to be an explicit choice what flows to include in any given capacity constraint group (and whether they are based on available capacity or in actual flow).

Ramps should not be constrained across time jumps

At the moment equations ramp_up_variable, ramp_up_constraint and ramp_down_constraint jump over time holes. Better not to force those constraints there.

Documentation: reserves and storage

add documentation

• how reserve groups work
• discount and end effects

Print out reserve price results

Warn if discount_years is not given

In situations where discount_years should be given (multi-year solve), then a warning should be given when this has not been set. At present, it results in division by zero.

Allocating CO2 to conversion process

At the moment CO2 emissions are output for the commodity node that has the CO2_content parameter. However, it would make more sense to have CO2 coming out of the unit that will process the commodity. Even more fancy would be to account for the possibility of transporting carbon (connections would not release CO2, only units).
Furthermore, this is probably slow:

• sum {(g, c, n, d) in group_commodity_node_period_co2, t in time : (d, t) in dt}
  and should be reformulated.

Improve plots

[ ] Add header information to plot title row
[ ] Remove unnecessary items from the plot title row
[ ] Shorten scenario name (remove __import_results)
[ ] Separate timestamp from scenario name
[ ] Ensure legible Tick lables (issue at least on categorical x-axises)

More control over outputs

A large model can produce a quite a lot of outputs and it would be nice to able to limit outputs. At the moment there is some control through group entities that can use output_results parameter. There is also results parameter for the model object, but it is not actually doing anything. It should be implemented and possibly modified (e.g. to consider omit outputting unit, connection and node level results in favor of group results).

Inform user about updates

It would be very handy, if FlexTool in Toolbox would inform the user (e.g. a pop-up) a) when a FlexTool update is available, b) when a FlexTool update requires a Spine Toolbox update, and c) when an update requires changes to the input data. Also maybe allow user to check if their FlexTool is up to date via Toolbox.

Capture solver infeasibility correctly

For example the init db scenario wind_battery.
Has something to do with the infeasibility of the storage constraints.
In the example can be produced by having a storage_start_end_method fix_start and value 0.1 and horizon reverence_value 0.4
or
if horizon method removed, by storage_start_end_method fix_start_end and the start value as 0.1 and end value as 0.4.
In both of the possibilities, reducing the end or the reference value to 0.3 fixes the problem.

The console says that it generates the problem, but does not start to process it. So it fails on the solve row. Still claims "INFO: Success! "

Limiting resource use on sub-period basis

Consider ways to limit resource use on sub-period level (e.g. hydro)

Objects does not appear in the input_data after commit

I had this issue in some occasions.
I create an object (or relationship), I commit, and I close the editor. When I reopen the editor, the object has "disappeared". However, it seems it still exists, because if I try to recreate it, I get an error message.
See attached screenshot for reserve_upDown_unit_node.

If I exit SpineToolbox and reopen it, my object reappears.

N-1 unit loss

Add scaling for profiles

Right now inflow can be scaled using annual_flow and a inflow_method. Scaling should also be enabled for profiles - e.g. VRE capacity factor time series should be possible to scale to a target CF (which could be the one in the total time series or then exogenous number).

Enable "fusion power plants"

It's currently not possible to have units that have only variable cost and that therefore create energy out of thin air. It's not a big difference from a unit that has a commodity node as the energy source, but still nice to have.

Map vs array

It's not clear whether to enter a map or an array for some of the parameters. While waiting for Toolbox implementation of restricted data types, FlexToolRunner.py could implement it's own checks.

Typo in Results

In the Results database, model -> cost_dt -> cost_type says downward reserves slack penalty" - I don't think the quotation mark " should be there.

change variable cost name to other O&M

Documentation: binding of storages

Write documentation on how to use binding methods

Improving the documentation

Observations based on a test user

• explain the point of init.sql. What 'init' means. The different ways how to go through the tutorial
• explain the basic functioning of Toolbox (choosing alternatives, how to edit data, how to filter data)
• say where data is for each thing visible in figure and text
• confusing instructions -- too much assumed to be known. Is the solution to make a lot longer tutorial or a lot simpler tutorial?
• explain where the results are
• also what results you will get
• To database editor in brief: what classes, objects, relationships and paragraphs. scenarios and alternatives.
• Explain curtailment penalty (how it should be calculated after the model run)
• Prioritization of what you need when you first start working with Spine Toolbox / FlexTool project
• Move db editor, browser editor etc. to own pages (browser editor to web-interface docs)
• Explain how to make model faster and how integers affect model (startups and ramps too)

Fixed cost reporting

Fixed costs are not reported from operational runs. Should be added, but with the caveat that the objective value from the solver will not include it, so should not be part of the cost comparison at that stage.

Add ramp cost functionality

There is ramp cost method, but it does not cause ramp cost at the moment. Should be added.

Clean up root folder

There are too many files in the root folder. It should have only major files like flexModel.mod, flextoolRunner.py, readme.md, license.md, release.md.

• DLL and EXEs should be in folder lib
• Templates should be in folder Templates
• Data should be in folder Data or Input

Commercial solvers

Add possibility to run with specific commercial solvers like CPLEX and Gurobi

Model leakage

Report model leakage in the results
Update documentation (https://github.com/irena-flextool/flextool/blob/test_system_docs/docs/index.md#adding-a-battery)

Cost of deviating from a profile

This would allow e.g. curtailment cost (but should be more generic)

Exit empty model cleanly

If the model is empty (as seen by the solver) for some reason, solution file will not be proper and there is no nice error message.

Add storage start and end values

At the moment, it's not possible to define start and/or end values for storages (the start will equal the end value plus changes at the last time step, but the model is free to choose those values).

Separate period_realized

Period_realized needs to distinguish between investment solves and dispatch solves. Investment decisions are to be realized from investment solves and other decisions from the dispatch solves (when dispatch solve is performed). There needs to be consideration of other 'realizations' too, like using storage value from higher level solve in a lower level solve.

Nice to have: Exporter from Input_data to Excel input template

When the model is created directly in FlexTool, it would still be nice to be able to export it in the input template (that is used to create a model from Excel).
It would allow:

• easier review of the model
• sharing the model with non-users of FlexTool
• go-and-forth between Excel and FlexTool.

Two possible data sources in the workflow is confusing

New user can take data from the template Excel or from the init.sqlite. However, if the user runs the whole workflow, the input_data.sqlite will get data from both. It gets very confusing in the input_data after that.

Harmonize Init.sqlite and template excel

At present init.sqlite has more data than the template excel - would be better if they were identical. Some automation needed to make it reasonable to update whenever there are changes.

Storage duration

Can we add (or is already there?) the possibility to have fixed MW/MWh ratio in storage units?

Multiple timelines

FlexTool will probably fail miserably if trying to have multiple timelines in the DB. Needs to be fixed (does not need to support multiple timelines in single model - just to handle if it's in data).

Improve the investment / retirement formulation

At present FlexTool reinvests automatically after lifetime of an asset ends. This makes retirements awkward (salvage value needs to include de-investment value, i.e. the investment cost). It is also not very nice for vintage based investments.

Documentation: describe discounting and end-effects

Output inertia with largest online failure mode

as per title

Need to add results.sql in home folder to avoid error

Hello,

When going through the Tutorial, I found that, to complete the SpineToolbox workflow from initialising to producing Excel tables with results, I had to manually add a file "results.sql" in the home folder before running (I used a copy of results_template.sql, which was then overwritten while running the workflow).

Without doing this, I obtained an error "couldn't connect to the Database".

It should maybe be clarified in the manual that either this file needs to be created, or alternatively the file name in Spine workflow element no. 7 needs to be changed? Unless I am doing something wrong.

Best,
Sebastian

Rolling window optimization

Rolling window optimization can currently be performed by setting the solve sequence manually, but this would be very impractical for hundreds of roll forwards.

As we want to enable nested rolling windows (e.g. investment at the highest level, water value calculation in the middle and then dispatch at the bottom), solve objects should have an optional parameter include_solve. When set, it would call the named lower level solve, which can then be a set of rolling solves when using rolling_window value for the solve_mode in the lower level solve (as defined below). If the calling (higher level) solve is itself a rolling solve, it will then include the lower level (rolling) solve between every roll forward it takes. solves parameter of model class will need to hold only the solves for the highest level.

For rolling solves, we should add parameters that allow to define the roll forward properties for a 'solve' object:

• rolling_solve_horizon (initially hours, but could also be a map that defines the time resolution for different timeblocks in the solve, i.e. decreasing resolution with increasing horizon)
• rolling_solve_jump (hours)
• optional rolling_start_time (timestamp index that needs to be found from the block_duration parameter of timeBlockSet). This would not be needed when there is a higher level solve that sets the start time.
• rolling_duration (hours). This also potentially optional since it could be deduced from the upper level solve when that's available.

There needs to be an update of parameters that choose what values to fix/lock/keep form specific solves. There is another issue for that: #58

FlexToolRunner.py will then generate the solve sequence the model will need.

upDown objects deleted when purging the DB

upDown objects from the upDown class (two objects: up and down) are deleted when purging the DB from its items.
It would be great to find a way to "protect them" so they are never deleted.

Variable costs for one variable connection

If a connection is presented with one variable (lossless), then it should not have variable costs either, since they will be negative in the opposite direction. The connection must use two variables if variable costs are included. Probably not covered at the moment.

Failure when running example

Hi, I downloaded the last versions of Flextool and SpineToolBox and I have the following errors when running the default example.

When looking at the console, I have this

>>> # Running python flextoolrunner.py
solve_data folder existed
Traceback (most recent call last):
  File "<stdin>", line 3, in <module>
  File "flextoolrunner.py", line 583, in <module>
    main()
  File "flextoolrunner.py", line 551, in main
    jumps = runner.make_step_jump(active_time_list)
  File "flextoolrunner.py", line 390, in make_step_jump
    first_period_name = list(active_time_list)[0]
IndexError: list index out of range
>>> # Running python flextoolrunner.py
>>> # Running python flextoolrunner.py
>>> # Running python flextoolrunner.py
>>> # Running python flextoolrunner.py
Traceback (most recent call last):
  File "<stdin>", line 3, in <module>
  File "flextoolrunner.py", line 583, in <module>
    main()
  File "flextoolrunner.py", line 532, in main
    runner = FlexToolRunner()
  File "flextoolrunner.py", line 29, in __init__
    self.model_solve = self.get_solves()
  File "flextoolrunner.py", line 56, in get_solves
    model__solve[datain[0]].append((datain[1]))
IndexError: list index out of range
Traceback (most recent call last):
  File "<stdin>", line 3, in <module>
  File "flextoolrunner.py", line 583, in <module>
    main()
  File "flextoolrunner.py", line 532, in main
    runner = FlexToolRunner()
  File "flextoolrunner.py", line 29, in __init__
    self.model_solve = self.get_solves()
  File "flextoolrunner.py", line 56, in get_solves
    model__solve[datain[0]].append((datain[1]))
IndexError: list index out of range
Traceback (most recent call last):
  File "<stdin>", line 3, in <module>
  File "flextoolrunner.py", line 583, in <module>
    main()
  File "flextoolrunner.py", line 532, in main
    runner = FlexToolRunner()
  File "flextoolrunner.py", line 29, in __init__
    self.model_solve = self.get_solves()
  File "flextoolrunner.py", line 56, in get_solves
    model__solve[datain[0]].append((datain[1]))
IndexError: list index out of range
>>> 

What should I do?
Thanks a lot,