grid-parity-exchange / egret

Tools for building power systems optimization problems

EGRET Overview

EGRET is a Python-based package for electrical grid optimization based on the Pyomo optimization modeling language. EGRET is designed to be friendly for performing high-level analysis (e.g., as an engine for solving different optimization formulations), while also providing flexibility for researchers to rapidly explore new optimization formulations.

Major features:

• Solution of Unit-Commitment problems
• Solution of Economic Dispatch (optimal power flow) problems (e.g., DCOPF, ACOPF)
• Library of different problem formulations and approximations
• Generic handling of data across model formulations
• Declarative model representation to support formulation development

EGRET is available under the BSD License (see LICENSE.txt)

Installation

• EGRET is a Python package and therefore requires a Python installation. We recommend using Anaconda with the latest Python (https://www.anaconda.com/distribution/).

• These installation instructions assume that you have a recent version of Pyomo installed, in addition to a suite of relevant solvers (see www.pyomo.org for additional details).

• Download (or clone) EGRET from this GitHub site.

• From the main EGRET folder (i.e., the folder containing setup.py), use a terminal (or the Anaconda prompt for Windows users) to run setup.py to install EGRET into your Python installation - as follows:

  pip install -e .
  

Requirements

• Python 3.7 or later
• Pyomo version 6.4.0 or later
• pytest
• Optimization solvers for Pyomo - specific requirements depends on the models being solved. EGRET is tested with Gurobi or CPLEX for MIP-based problems (e.g., unit commitment) and Ipopt (with HSL linear solvers) for NLP problems.

We additionally recommend that EGRET users install the open source CBC MIP solver. The specific mechanics of installing CBC are platform-specific. When using Anaconda on Linux and Mac platforms, this can be accomplished simply by:

conda install -c conda-forge coincbc

The COIN-OR organization - who developers CBC - also provides pre-built binaries for a full range of platforms on https://bintray.com/coin-or/download.

Testing the Installation

To test the functionality of the unit commitment aspects of EGRET, execute the following command from the EGRET models/tests sub-directory:

pytest test_unit_commitment.py

If EGRET can find a commerical MIP solver on your system via Pyomo, EGRET will execute a large test suite including solving several MIPs to optimality. If EGRET can only find an open-source solver, it will execute a more limited test suite which mostly relies on solving LP relaxations. Example output is below.

=================================== test session starts ==================================
platform darwin -- Python 3.7.7, pytest-5.4.2, py-1.8.1, pluggy-0.13.0
rootdir: /home/some-user/egret
collected 21 items

test_unit_commitment.py s....................                                       [100%]

========================= 20 passed, 1 skipped in 641.80 seconds =========================

How to Cite EGRET in Your Research

If you are using the unit commitment functionality of EGRET, please cite the following paper:

On Mixed-Integer Programming Formulations for the Unit Commitment Problem Bernard Knueven, James Ostrowski, and Jean-Paul Watson. INFORMS Journal on Computing (Ahead of Print) https://pubsonline.informs.org/doi/10.1287/ijoc.2019.0944

egret's Issues

Tests fail when cbc is not installed.

Some of the tests need an appropriate skip when cbc is not involved.

Pull in Unit Commitment baselines from Prescient

Support for Python 2.7?

I see imports for six. Do we want to support Python 2.7?

LHS v. RHS orientation of equality constraints

For example the power balances need to be flipped so that the duals have a sign that corresponds to a standard formulation where the RHS is the constant.

Reorganization

What does everyone think about a slight reorganization of Egret? I am thinking of the directory structure below. I have not fully flushed it out, but I think you will get the idea. I don't think the current structure is intuitive, but the structure I outlined below might be missing some complexities.

• Egret
  • data
  • solvers
    • lazy ptdf
    • acopf
    • ...
  • models
    • acopf
      • declarations
      • models
      • useful acopf things
    • commitment
    • dcopf
    • bilevel
    • ...

Small differences in objective between b-theta DCOPF and PTDF DCOPF

Specifically, on pglib_opf_case_300_ieee, but is noticeable on other larger cases. Could just be a numerical issue with the PTDF calculation itself.

Expose slack option

AC relaxations should expose the slack option for all relaxations. Here's one example:

Present:
def create_soc_relaxation(model_data, use_linear_relaxation=True):
model, md = _create_base_ac_model(model_data, include_feasibility_slack=False)

Proposed:
def create_soc_relaxation(model_data, use_linear_relaxation=True, include_feasibility_slack=False):
model, md = _create_base_ac_model(model_data, include_feasibility_slack=include_feasibility_slack)

We need a PowerWorld parser front-end for Egret

_include_feasibility_slack improvments

_include_feasibility_slack currently has a named argument called "penalty" that defaults to 1000. I suggest the following improvements:

1. replace this argument with two named arguments with defaults shown :
   per_unit_slack_penalty=None and slack_factor=10
   The code should assert that one is None and the other is not None.
   The argument slack_factor would work exactly like the current argument called penalty.
   The argument per_unit_slack_penalty would be assigned directly to p_penalty and q_penalty

2. These arguments should be exposed all the way up to the create_* function (e.g.,
   all the way up to create_riv_acopf_model, etc.)

Aside: It might be good to have a standardized way to pass arguments from the highest levels of Egret to the lowest, but that is a different issue.

Resolve inconsistencies in how Egret handles infeasiblity or other solver issues

egret.common.solver_interface._solve_model raises an exception if a solver terminates with a 'not good' termination condition. Conversely, the lazy PTDF solve loops do not raise exceptions (though likely cause them) if a solver terminates with an unsafe condition.

What should the default behavior of Egret be? Should we throw exceptions if a model is infeasible, or attempt to do something more graceful?

Need to protect stack graph generation viz tests, in case matplotlib and seaborn are not installed.

For @rconcep : Can you add skip-if logic for the stackgraph generation tests, in case matplotlib or seaborn are not installed? I am going to add both of those packages as a dependency for auto install in setup.py, but I think it still would be a good idea in case viz packages are not properly installed / functioning on some platforms.

Revisit why logging tests were failing

We temporarily disabled the test_logging() test function (by re-naming it to _test_logging()), as it was failing for reasons we did not understand - and the test failures were blocking merging of multiple PRs. However, the issue needs to be revisited and the test eventually restored.

The matpower parser should use None for -inf/inf generator bounds instead of 9999.99

The second test fails: --runmip error

I have installed it successfully: Windows 10, newest anaconda, ILOG CPLEX 12.10, etc.
1-This is the output of pytest test_unit_commitment.py

================================================= test session starts ====================
platform win32 -- Python 3.7.6, pytest-5.3.5, py-1.8.1, pluggy-0.13.1
rootdir: C:\Users\some-user\Egret-master
plugins: hypothesis-5.5.4, arraydiff-0.3, astropy-header-0.1.2, doctestplus-0.5.0, openfiles-0.4.0, remotedata-0.3.2
collected 21 items

test_unit_commitment.py ..................... [100%]

=========================================== 21 passed in 2804.02s (0:46:44) ==================
it took more than 46 min, isn't it a bit slow on my system?

2-This is the output of pytest --runmip test_unit_commitment.py

ERROR: usage: pytest [options] [file_or_dir] [file_or_dir] [...]
pytest: error: unrecognized arguments: --runmip
inifile: None
Any idea about fixing the error?

Correct the citation

You should ask users to cite the IJOC paper

Tests fail for Gurobi solver

I have run the tests following the installation instructions (execute pytest test_unit_commitment.py from models/tests sub-directory). I have the latest Gurobi (AMPL-Gurobi 9.0.1) installed at macOS.

The result of the test is following: 21 failed, 1 warning in 814.40s (0:13:34)
The full log is here Gurobi log

I've tried to run the tests with different solvers, e.g.
SCIP: 18 failed, 3 passed, 1 warning in 241.02s (0:04:01)
log: SCIP log

CBC: 1 failed, 20 passed, 1 warning in 2765.15s (0:46:05)
log: CBC log

Can anybody give me some pointers how to fix the tests? Thanks!

Broaden the definition of "load"

It is common for data sources to have load across an area, and then "load participation factors" for each bus. We should consider making this part of the Egret ModelData specification to keep the user from having to re-implement boilerplate code to distribute the loads to the buses.

This wouldn't necessitate changing our models, just adding another "transformation" to the ModelData before handing it to a model. (The only transformation we have now does p.u. scaling.) We could also consider building-in other common transformations, such as reducing a network to zonal regions.

Missing else statement

In egret.model_library.transmission.branch.declare_ineq_p_branch_thermal_lbub, there is an if statement that checks if the approximation type is is BTHETA or PTDF. If so, constraints are added to limit the power flow. However, there is no else statement. We should at least raise an exception or warning rather than silently doing nothing.

Running unit_commitment.py standalone

Hi,
I tried to run unit_commitment.py in a standalone way by un-commenting the main at the end.
The first error is that there is no read_from_json(file) in ModelData, only read(file) method which is a typo.
The second error after fixing the above:
_File "c:\users\me\desktop\python-test\egret-master\egret\model_library\unit_commitment\uc_model_generator.py", line 100, in generate_model
baseMVA = model_data.data['system']['baseMVA']
KeyError: 'baseMVA'

I fixed this by adding an instance (self) version of the class method read(file) to ModelData. Any idea?

Thanks in advance!

Need new test case for relaxations

Per #140, one of the relaxation test instances needed to be removed. Should a new one be added to take its place?

Installation Unsuccessful

I am trying to follow through the instructions and install the package but I have been unsuccessful. I would appreciate some guidance and help, please point out if I am doing something obviously wrong. I am using Anaconda Prompt all through to attempt the package installation. I am using Windows 10 and Conda version 4.8.3.

1. Git Clone the repository - successful, no problem
2. Open Anaconda Prompt and create an environment

      conda create --name erget_test

3. Install Pyomo

      conda install -c conda-forge pyomo

4. Install open-source solvers

      conda install -c conda-forge ipopt glpk

5. Install pytest

      pip install pytest

6. Install coincbc

      conda install -c conda-forge coincbc

   This fails throwing a PackagesNotFoundError

Additionally, the last instruction around running setup.py is not really clear, I have tried several different ways to invoke and run setup.py, but I am not sure I understand what works:

Add fuel cost to RTS-GMLC parser

Tests needed for bi-level capabilities.

The title says it all!

Implement start-up and shut-down ramping trajectories

...for the unit commitment / dispatch model. This is to enable more reasonable results at shorter time-intervals (e.g., 5-minutes).

Stand up readthedocs for the EGRET package

Add the readthedocs harness for EGRET, including introductory documentation.

• Introduction
• Using EGRET
• ModelData
• ACOPF
• DCOPF
• Unit Commitment

Consistent and correct handling of None values needed for line thermal limits in AC and UC model constructors

Lines with thermal limits equal to None are not handled consistently in AC model constructors (acopf.py) and are not presently handled at all in UC model constructors (e.g., see ptdf_utils.py).

Headroom may be incorrectly reported when zero-cost reserves are used

Headroom is measured as the least slack on all ramping-up constraints. However, when zero-cost reserves are used, the solver is indifferent to assigning that extra slack to the headroom. Two potential solutions:

1. Adjust optimization formulations such that reserves are not over-procured. This has the disadvantage that reserves will be slightly less flexible (changing an inequality constraint to equality), and reserve shadow prices may be negative if there is an over-production of energy.
2. Adjust the solution. When reserves are over-procured, adjust every generator's procurement down by the ratio the reserves are over, and re-assign it to the headroom. This may be somewhat tricky to implement with zonal/area reserves procured from the same pool as system reserves.

Calculate generator power available for stack graph generation

SOC relaxation directly

It would be good to have a convex relaxation of AC as a model itself.

Add support for N-1 transmission constraints

1. Compute LODF matrix
2. Add N-1 security constraints with PTDF lazy logic for unit commitment and economic dispatch

Duplicated power flow limit constraint declarations

We should decide if both of these functions are needed

egret.model_library.transmission.branch.declare_ineq_p_branch_thermal_bounds

and

egret.model_library.transmission.branch.declare_ineq_p_branch_thermal_lbub

Inconsistent handling of loads in power balance

See lines 275-276 in egret/model_library/transmission/bus.py. We check to see if the load is zero. If it is not, then we subtract m.pl[bus] from the power balance. However, there is no indication that m.pl[bus] should be a constant (immutable param or fixed var).

path to data in acopf.py

The path in the file name construction line in acopy.py main should be:

matpower_file = os.path.join(path, 'tests/transmission_test_instances/pglib-o

pf-master/', filename)

Failure when running pytest on models/tests/test_unit_commitment.py

With pyomo master (as of mid last week) and egret master (as of today), I am seeing the following when running "pytest test_unit_commitment" from the "models/tests" sub-directory. I am using gurobi 9.0.2. But this also happens with CBC 2.10.5.

Long version of output is further below - but short version is that the objective function for the model resulting from create_compact_unit_commitment is well off tolerance (by 7-ish asolute cost units).

Could someone - probably @bknueven - attempt to replicate? Thanks!

Long version of output is as follows:

test_unit_commitment.py F.................... [100%]

================================================== FAILURES ==================================================
___________________________________________ test_int_all_uc_models ___________________________________________

@unittest.skipUnless(comm_mip_avail, "Neither Gurobi or CPLEX solver is available")
def test_int_all_uc_models():
    _test_uc_model(create_tight_unit_commitment_model)

  _test_uc_model(create_compact_unit_commitment_model)

test_unit_commitment.py:86:

---

uc_model = <function create_compact_unit_commitment_model at 0x10fcba710>, relax = False
test_objvals = [4201915.017320504, 5454367.7670904165, 5999272.361123627, 5461120.3231092375, 6062406.32677043]

def _test_uc_model(uc_model, relax=False, test_objvals=test_int_objvals):

    for test_case, ref_objval in zip(test_cases, test_objvals):

        md_dict = json.load(open(test_case,'r'))
        md = ModelData(md_dict)

        model = uc_model(md, relaxed=relax)
        opt = SolverFactory(test_solver)
        _set_options(opt, mipgap=0.0)

        if isinstance(opt, PersistentSolver):
            opt.set_instance(model)

        result = opt.solve(model, tee=False)

        assert result.solver.termination_condition == TerminationCondition.optimal

      assert math.isclose(ref_objval, result.problem.upper_bound, rel_tol=rel_tol)

E AssertionError: assert False
E + where False = (5454367.7670904165, 5454374.111340323, rel_tol=1e-08)
E + where = math.isclose
E + and 5454374.111340323 = <pyomo.opt.results.container.ListContainer object at 0x1137203d0>.upper_bound
E + where <pyomo.opt.results.container.ListContainer object at 0x1137203d0> = {'Problem': [{'Name': 'x8497', 'Lower bound': 5454374.111340323, 'Upper bound': 5454374.111340323, 'Number of objectiv...': 1.5290980339050293}], 'Solution': [OrderedDict([('number of solutions', 0), ('number of solutions displayed', 0)])]}.problem

test_unit_commitment.py:72: AssertionError

Testing ToDo's

1. Switch to GitHub actions
2. Make sure we test against oldest supported Pyomo release and Pyomo master
3. Make sure we run tests nightly
4. Add tests for python 3.7 and 3.8

Separate and categorize tests

• We need to split up the tests so that different test cases are recognized as different tests in pytest. One potential package is the parameterized package.
• We can also categorize tests to avoid long running tests on travis (but still make it easy to run them locally).

change path in acopf.py __main__

Re-work unit commitment tests

Most unit commitment tests are regression-style tests and are generally too large, and hence take a bit of time to run. The unit commitment tests should be re-worked to have the following features:

1. Testing one constraint or component (e.g., minimum up time constraints, production cost, initial condition constraints)
2. Small; 1-3 generators at most, 1-3 buses at most, no more than 12 time periods.
3. Have an obvious solution which can be worked out easily by hand.
4. New/existing test cases should be auto-discovered by the test harness and not hard-coded.

We could keep some flagged regression tests (probably based of the current suite, or pglib-uc) as part of the re-work.

Path issue when running generate_graphs.py for RTS-GMLC test case

For @rconcep - I suspect you want to restore the code in the main() in generate_graphs.py that is presently commented out - which would use the built-in test data in egret/models/tests/uc_test_instances. I would like to retain the general test functionality in this module, as it's a nice quick+dirty way to generate a graph - and provide an example of how to interact with the library.

namedtuple() has unexpected kwarg for Python 3.7

I pointed this out before but for the calls to collections.namedtuple() in egret/parsers/rts_gmlc_parser.py, the kwarg for "verbose" causes a TypeError for unexpected kwarg in Python 3.7. This was addressed in the prescient repo.

Add monitored flag for transmission lines

so the user can hand-pick lines to monitor or not, instead of/in addition to using the kV filter.

Some features not precisely preserved after dumping/loading from json

The most vexing of which is the description of the polynomial cost curve, for which we assume the 'values' dictionary has integer keys in several places. Here is the original.

{'data_type': 'cost_curve',
 'cost_curve_type': 'polynomial',
 'values': {0: 37, 1: 25.26, 2: 0.00712}}

Same after dumping/loading from json.

{'data_type': 'cost_curve',
 'cost_curve_type': 'polynomial',
 'values': {'0': 37, '1': 25.26, '2': 0.00712}}

Obviously this isn't a big deal technically, but how should we address it? Two obvious things are to make the model builders aware some things may be strings (and convert appropriately) or write a json parser that makes these substitutions where appropriate.

We should be careful modifying the `pdtf_options` dictionary passed in by users

We should be careful modifying the pdtf_options dictionary passed in by users

Mipgap to None when invoking _solve_model from solve_dcopf

This is minor, but also easy to fix if agreed upon.

The function solve_dcopf in models/dcopf.py calls _solve_model (in common/solver_interface.py) to solve the DCOPF model once constructed. The default mipgap in _solve_model is 0.001. Further, solve_dcopf does not over-ride, even though it's continuous problem. I am proposing to set the mipgap keyword argument in the call to _solve_model from solve_dcopf to None, mainly to stop people from wondering why a mipgap is set for a continuous problem, when staring at the traces.

More generally, why is the mipgap default to something other than None in _solve_model? Especially given that it's distinct (I think) from at least the Gurobi default.

Clean CDM Parser

Improve the parser for RTS-GMLC data, stored in the "Common Data Model" (CDM) format.

Main tasks:

• Get rid of magic constants
• Clean and test code

Pytest should not be a requirement in setup.py

Unable to run egret/models/tests/test_copperplate_dispatch.py

The .m files that represent the various cases to be solved (e.g. pglib_opf_case3_lmbd.m) cannot be located. In fact, the folder where these files are supposed to be located, 'pglib-opf-master', is also not located in the designated path i.e. egret\models\tests\transmission_test_instances. This causes the test to fail. Can someone please fix this issue?

Tighten bounds on slack variables in DC and AC model libraries.

To make life easier on interior point solvers, at a minimum.

Prescient dat parser should be moved to a test directory

Complete the automatic API documentation harness for readthedocs

Add the automatic API documentation harness for readthedocs.