blue-marble / gridpath

A versatile simulation and optimization platform for power-system planning and operations.

Welcome to GridPath

GridPath is a versatile power-system planning platform capable of a range of planning approaches including production-cost, capacity-expansion, asset-valuation, and reliability modeling.

Documentation

GridPath's documentation is hosted on Read the Docs.

Installation

Python

GridPath is tested on Python 3.9, 3.10, and 3.11. Get one of those Python versions here.

GridPath Python environment

You should create a Python environment for your GridPath installation, e.g. via venv, a lightweight environment manager that is part of the standard Python distribution. Make sure to create activate the environment before installing GridPath.

Install GridPath from PyPi

Once you have created and activated the GridPath Python environment, you can install the latest version of GridPath from PyPi with:

pip install GridPath

Install GridPath from source

You can alternatively download the GridPath source code and install from source.

pip install .[all]

NOTE: If you plan to edit the GridPath code, you should install with the -e flag.

Solver

You will need a solver to use this platform. GridPath assumes you will be using Cbc (Coin-or branch and cut) by default, but you can specify a different solver.

Usage

The gridpath_run and gridpath_run_e2e commands

If you install GridPath via the setup script following the instructions above, you can use the command gridpath_run to run a scenario from any directory -- as long as your GridPath Python environment is enabled -- as follows:

gridpath_run --scenario SCENARIO_NAME --scenario_location 
/PATH/TO/SCENARIO 

If you are using the database, you can use the command gridpath_run_e2e to run GridPath end-to-end, i.e. get inputs for the scenario from the database, solve the scenario problem, import the results into the database, and process them. Refer to the documentation for how to build the database.

gridpath_run_e2e --scenario SCENARIO_NAME --scenario_location 
/PATH/TO/SCENARIO 

To see usage and other optional arguments, e.g. how to specify a solver, check the help menu, e.g.:

gridpath_run --help

gridpath's Issues

Add ramp rate limits to no_commit generators

For consistency, it would be good to include optional ramp-rate limits to these type of generators.

Make sure these updates address #30 and #34

Add a process for specifying solver options

Update Installation Instructions, Environment and Reqs

• add setup.py and environment/requirement files
• terminal
• managing environments
• info on where to get solver
• how to deal with path, etc.

Wheeling rates

Put example problems in a database

Change units of hydro_conventional_horizon_params

Currently the units are MW, which makes it the hydro operational type incompatible with new build resources. It also doesn't allow for a proper evaluation of when to skip ramp rates, since the pmin as fraction of capacity is not determined until model the model is solved (capacity is an expression).

These issues are easily solved by changing all hydro input units to "fraction of operational capacity".

Add binary capacity types for new build

• Binary Generator Build
• Binary Storage Build

Required new build

Best way to prevent storage from acting as load

Storage will sometimes charge and discharge at the same time to appear as load, so that, for example, it appears that RPS energy is being delivered to load (in the simple case where a MWh RPS target is specified), even though it's really just losses.

Heat rate curves

Heat rate curves with arbitrary number of piecewise linear (?) segments.

Write setup script

DC power flow

Drop support for Python 2 and clean up compatibility code

Ramp rate constraints

Standardize naming conventions, code style, and function descriptions

• model vs. mod vs. m usage
• project vs. generator vs. resource (I prefer resource)
• p vs. g vs. r usage for projects/gen/resource
• replace dynamic_inputs with dynamic_components throughout code base
• return None vs. pass
• standardize line breaks with long formulas (I suggest to use brackets to avoid "\" and to put operators at the start of the line. Always separate line for the (in)equality sign?
• standardize function descriptions. Often the same general functions have slightly different descriptions depending on where they appear, even though they do the same thing (e.g. test_data_loaded_correctly()).
• standardize input/output type description in function docs
• rename prm_group_costs (?) - based on #todo in code

Additional production-cost features

Medium Priority

• fuel budgets (think about whether we can generalize budgets). In first stage can do simple average allocation, but could think about more sophisticated ways to allocate budgets.
• startup/shutdown budgets (think about whether we can generalize budgets). In first stage can do simple average allocation, but could think about more sophisticated ways to allocate budgets.

Low Priority

• schedule imports/exports with certain shape (can always adjust input load for easy hack; but might have to adjust transmission limits too)
• ensure availability derate is available and applied for all generators and transmission. Think about how to allow for variability by timepoint without creating tons of excess inputs

Allow option to make soft constraints hard

• unserved energy
• overgen
• rps policy
• carbon policy
• reserves
• prm
• local capacity

Ramp rate, minup/mindown, and startup constraints for all dispatch types

Ramp Rates:

• dispatchable continuous commit
• dispatchable binary commit
• dispatchable no commit
• always on

Minup / Mindown:

• dispatchable continuous commit
• dispatchable binary commit

Startup / Shutdown:

• dispatchable continuous commit
• dispatchable binary commit

Database export functions

• dispatchable continuous commit
• dispatchable binary commit

Fix foreign keys in temporal tables in database

Also check other tables for incorrect directionality of foreign keys.

Change units of ramp rate limits

Current ramp rate limits are in units of "fraction of capacity per hour", with a minimum of zero and a maximum of 1 (Pyomo PercentFraction). There are 2 problems with this approach:

1. Even when the ramp rate is 1, i.e. the unit can ramp its full capacity in an hour, the ramp rate could still be limiting when you are looking at timesteps of less than an hour (e.g. 5 minutes). However, most ramp rate constraints are skipped when the ramp rate limit is 1, which could lead to infeasible ramps when modeling subhourly timesteps.
2. When users convert their ramp rates from MW/min to fraction/hour, they will likely input "1" if the ramp rate allows for full ramping within the hour, even if it could do much more than that (e.g. a 100 MW unit which can ramp 10 MW/min, would be able to ramp 6 times its capacity in an hour, but users will likely just input 1). Also, because of the current PercentFraction constraint, users wouldn't even be able to input a number larger than 1.

If we change ramp rate limit units to "fraction of capacity per minute", users won't under-input the ramp rates since most units can't ramp their full capacity within a minute. Assuming that units who can ramp their full capacity within a minute won't ever be ramp constrained, we could keep the PercentFraction constraint for the inputs. If we don't want that, we should not limit the inputs to 1 and allow inputs larger than 1 (at which point we might as well keep the units in fraction per hour).
Whatever ramp rate unit we choose, the checks that happen when skipping ramp constraints should not assume hourly timesteps and properly address the ramp rate units.

Create unit tests for database queries

Skip min up/down constraint for certain timepoints

According to Morales-Espana et al. (2013), the min up/down constraint for unit commitment is only applied to [UT, T], where UT is the min up/down time, and T is the horizon length (e.g. 24 hours). This means you can skip the constraint for the first UT-1 hours.

We should investigate whether this also applies to the capacity commit module, where we have linear capacity commitment rather than binary commitment, and adjust the constraint
if needed. See PR #42 for discussion.

Carbon cap

Save and load solutions

We should implement functionality to save and load solutions (e.g. in order to access solution elements that were not exported without having to re-run the scenario).

Fix gridpath module location error

PR #78 seems to have caused issues with importing modules now that the scripts that do the imports are within the gridpath module. I get the following error:

Traceback (most recent call last):
  File "run_scenario.py", line 22, in <module>
    from gridpath.auxiliary.auxiliary import Logging
ModuleNotFoundError: No module named 'gridpath'

If I remove gridpath from the import statements in run_scenario.py I get the following error:

ERROR! Module temporal.operations.timepoints not found..

So it seems little more complicated than simply removing gridpath from all the import statements.

Create script to run tests

It would be good to have the option to skip the non-linear example problems and maybe all example problems.

Run-up rates (0 to minimum loading) for starts and shutdowns

Incorporate reserve-provision into operational type's ramp constraints

Fix ramp constraints of operational module to take reserve-provision into account.

When you are providing downward reserves in the previous timepoint, and/or upward reserves in the current timepoint should reduce the amount of upward ramping you can do between timepoint setpoints. E.g. if you're going from 50 MW in t-1 to 100 MW at t, but you were also provding 10 MW reg down in t-1 and 10 MW reg up in t, your unit must at least be able to ramp up 70 MW between timepoints.
Likewise, if you are providing upward reserves in the previous timepoint, and/or downward reserves in the current timepoint sould reduce the amount of downward ramping you can do between timepoint setpoints. E.g. if you're going from 100 MW in t-1 to 50 MW at t, but you were also provding 10 MW reg up in t-1 and 10 MW reg down in t, your unit must at least be able to ramp down 70 MW between timepoints.

This is an issue for the following operational types:

• capacity_commit
• always_on
• hydro_curtailable
• hydro_non_curtailable

The following operational types will also have this issue, but will be rewritten anyway:

• binary_commit
• continuous_commit

The following operational type will have this issue once we add ramp rate constraints to it:

• no_commit

Make database name and location a user input

Currently, we're assuming the database is in the 'db' folder and called 'io.db' -- we need to allow the user to specify a different location and database name.

Add minimum duration constraint for storage

Adjust capacity_commit min up/down time constraint for different timepoint durations

The current minimum up time and minimum down constraints in the capacity_commit operational module do not properly take into account how varying timepoint durations will change what timepoints should be included in the relevant_tmps when creating the constraint.

Rather than a for-loop based on the current timepoint's duration, the constraint should evaluate the elapsed time as you go back to previous timepoints, until you hit the minimum uip or down time.

This will also be an issue for the min up and down time constraints for the binary_commit and continuous commit generators.

Add reserve-provision functionality to always-on operational type

In theory these type of generators should be able to provide reserves. If a generator is not able to do this, you can simply not provide a reserve zone.

Make sure to include reserve provision into ramp rates (see #30).

Check experimental ramp rate limits capacity commit

The capacity commit module has some non-conventional ways to break out the startup ramps and the operational ramps. Those are experimental and haven't been tested as thoroughly as they should.

Move various 'run gridpath' scripts to gridpath directory

That makes more sense as an 'entry point' and the scripts can then be added to setup.py.

Link production-cost functionality with database

We need to figure out how to write the appropriate directory and file structure for horizon and stage subproblems from the database.

Various Cleanup

Standardize naming conventions for variables

• standardize and shorten capacity and operational type names
• standardize and shorten variable names, constraints, and functions (see #278). Edit: Addressed for most of the main modules but not all.
• align database columns with tab file columns. They are often still different, e.g. new_binary_build_storage_vintage_costs and new_build_storage_vintage_costs (project vs. vintage). (vintage is perhaps a more general thing that needs to be cleared up since it is not used consistently everywhere).
• stop vs. shut down vs. turn off ; start vs. start-up vs. turn on
• model vs. mod vs. m usage
• project vs. generator vs. resource (I prefer resource)
• p vs. g vs. r usage for projects/gen/resource
• replace dynamic_inputs with dynamic_components throughout code base
• use tonnes to describe carbon emissions (which are in metric tonnes) and avoid confusion with short tons.
• rename database tables, input tables and results tables to be in line with the new names of the cap-types and op-types (gen_new_bin_xxxx instead of new_binary_generator_xxx.tab).

Code Style

• standardize use of string quotes, see #151
• return None vs. pass
• standardize line breaks with long formulas (I suggest to use brackets to avoid "\" and to put operators at the start of the line. Always separate line for the (in)equality sign?

Doc Strings

• Clarify hydro whether hydro operating characteristics (budget, pmax, pmin) are before or after curtailment and make sure parameter names are consistent (see ##341).
• standardize function descriptions. Often the same general functions have slightly different descriptions depending on where they appear, even though they do the same thing (e.g. test_data_loaded_correctly()).
• standardize input/output type description in function docs
• rename prm_group_costs (?) - based on #todo in code
• clarify that fuel_burn, variable_om, carbon_emissions, and recs are all "per timepoint" and not per hour and should be multiplied by the number of hours in the timepoint to get the actual amount ("power" vs. "energy" issue). This could also be clarified by changing the variable name. See #461.

Results Formatting

• Sort tables before outputting to csv / database (makes comparison a lot easier)
• Round all results to a default precision (e.g. .2f); right now we don't so I get lots of "git modified" hits because the objective function is off by 0.00001.
• Standardize use of value()/float()/... when writing out results (seems inconsistent right now)
• The summary results (summary_results.txt) for binary and linear retirements currently use the same headers, which could be confusing if you are modeling both linear and binary retirements (you would get 2 'Retired Capacity' sections, one with the linear retirements and one with the binary retirements).

Re-factoring repetitive code

• The gen_commit_lin and gen_commit_bin have some code blocks that are used at least twice, such as summing the startup power across all startup types, that could be re-factored into a Pyomo Expression for easier readabilty. Other modules might have similar issues, so this could be a broader look into re-factoring repetitive code into expressions across all subtype modules.
• The reserve rules are the same in all modules, so could be consolidated, see #329.
• The validation of operational characteristics by operational type has lots of repetitive code and hard-coding that makes it cumbersome to add new characteristics (e.g. expected_na_columns had to be updated everywhere in #526). It should be refactored.
• operational_periods_by_new_build_transmission_vintage could re-use the operational_periods_by_project_vintage functions in capacity_types.common_methods

Other

• Standardize database naming. We're using both 'conn' and 'db' for the same type of sqlite database connection objects (conn seems most clear to me; db could be the name of the db or the connection, or the path).
• Remove redundant passing of both cursor and connection in functions. Since creating cursor is very low overhead, simply pass connection object and create cursor where needed. Examples: imports_results_into_database in import_scenario_results.py, process_results.py, spin_on_database_lock.
• In some of the modules we use a local determine_xxx function to read in inputs, but we often call that function many times (instead of calling it once and assigning it to a variable).
• Find a way to avoid having to type stdout_original = sys.stdout all the time in run_scenario.py
• Standardize return behavior of run_scenario() to avoid all the if statements
• The order order of 'temporal' IDs in the table definitions and SQL statements is inconsistent and needs to be standardized.

Modify inputs for example test cases

The current test cases in the 'gridpath/examples' folder contain combinations of continuous commit and new build generators, which shouldn't be combined because it creates non-linear problems and messes up the fuel burn calculations.
We need to modify the examples such that new build resources are not of the continuous/binary commit type, while making sure that there are still other non-new build resources of the continuous/binary commit type (to make sure these are still covered in the test examples).

Input data checks and validation

• Mismatching operational types (e.g. binary commit and new build)
• New build projects have to be self consistent with new build vintage costs (gen / storage / tx)
• Validate that at least one project is contributing to each reserve/policy/reliability zone. Currently, we can have reserve/policy/reliability zone with no projects that can contribute toward meeting their target, see #347.
• Disallow combination of certain availability types and capacity types. Since capacity is multiplied by derate, we can't allow combinations of availability types and capacity types where both are decision variables. That means we have to disallow endogenous availability derates and new build or retirement capacity types. See #249. Edit: closing since non-linear problems are actually okay, just harder to solve.
• Provide useful error messages when the wrong key words are given to e.g run_scenario.py; right now the original wrong keyword error is obscured and the model will fail at a later stage; see #111.
• Validate missing var profiles (right now just a warning, see discussion in #543).
• Validate that hydro avg is between min and max.
• Validate that the horizons specified in structure.csv matched those in horizon_params.csv (or the equivalent database tables).
• add basic general validation to capacity types similar to operational types
• Validate that fully deliverable energy iimited projects are assigned a min duration for full cap credit (removed in #605)
• Validate that energy only allowed projects are assigned a deliverability group and that other types don’t have one (removed in #605)
• Check that all 'fully_deliverable_energy_limited' projects are assigned a min_duration_for_full_capacity_credit_hours
• Check if all ''energy_only_allowed' projects are assigned a 'deliverability group' and that other types are not assigned one
• Mismatched temporal setup and project balancing types (e.g. #819)

Clean up tuning costs module

Current implementation is fairly ad-hoc and could use clean up.

Ramp rate limit default values could bind at very short timepoint durations

Ramp rate limit units are in 'fraction per minute' and default to 1, i.e. the unit can ramp its full capacity in 1 minute. If you were to do sub 1-minute modeling (e.g. 1-second timesteps), this default ramp rate limit could become binding, which is not the expected behavior (limit shouldn't bind if not specified).

We need to adjust the constraints so that they are skipped at the default input value, regardless of the timepoint duration.

Maximum build constraints

Flesh out binary and continuous commit modules

Add the following functionality:

• min up time

• min down time

• ramp rate limits

• database import/export functions

Note that continuous commit will simply be the linear relaxation of binary commit (change binary variables to continuous variables).

Make sure that these updates address #30, #34, and #36

Fix Logging - Stdout Interaction

find a way to avoid having to type stdout_original = sys.stdout all the time in run_scenario.py

Binary retirements

Energy effect of reserves

E.g. storage reserves effect on state of charge, thermal reserves effect on emissions (and costs?), hydro reserves effect on budget

Create unittests for input validation

See also #6

Link production-cost functionality to database and improve multi-stage functionality

We need to figure out how to write the appropriate directory and file structure for horizon and stage subproblems from the database. We also need to think about additional functionality on linking stages and horizons (e.g. look-ahead days, spin-up days. etc.). Also think about how to allocate hydro/fuel budgets to horizons (can be simple average or something more sophisticated).

• Restructure ScenarioStructure class and actual file organization to deal with subproblems more easily
• Create database schemas/tables to deal with multistage production cost inputs
• Create functions to read/write from database
• Implement some sort of look-ahead and/or spin-up day logic with throwaway days
• Ensure that model supports different time resolutions in different stages (careful with inputs that assume constant timepoints resolution; e.g. ramp rates)

Set up coverage.py

Standardize SQL table definitions and statements

The order order of 'temporal' IDs in the table definitions and SQL statements is inconsistent and needs to be standardized.

Fix unconstrained reserve-provision for generators without reserve constraints

Currently, operational types that are assumed not to provide reserves have no constraints on their reserve provision in the operational_type's module. However, if a user provides a reserve sharing zone for any of these type of resources, the model will create a decision variable for that type of reserve, and the variable will be totally unconstrained.

This currently applies to the following operational types:

• must_run
• always_on (note: we should probably just allow this type of generator to provide reserves and add the proper constraints; create new issue for this?)
• variable_no_curtailment

There are a few options to solve this:

• disallow inputting any reserve sharing zone for certain type of generators
• add a constraint for these operational types that sets the sum of all headroom variables
  and the sum of all footroom variables to zero.