grid-parity-exchange / egret Goto Github PK
View Code? Open in Web Editor NEWTools for building power systems optimization problems
License: Other
Tools for building power systems optimization problems
License: Other
Improve the parser for RTS-GMLC data, stored in the "Common Data Model" (CDM) format.
Main tasks:
Lines with thermal limits equal to None are not handled consistently in AC model constructors (acopf.py) and are not presently handled at all in UC model constructors (e.g., see ptdf_utils.py).
For example the power balances need to be flipped so that the duals have a sign that corresponds to a standard formulation where the RHS is the constant.
We should decide if both of these functions are needed
egret.model_library.transmission.branch.declare_ineq_p_branch_thermal_bounds
and
egret.model_library.transmission.branch.declare_ineq_p_branch_thermal_lbub
Add the automatic API documentation harness for readthedocs.
I have installed it successfully: Windows 10, newest anaconda, ILOG CPLEX 12.10, etc.
1-This is the output of pytest test_unit_commitment.py
================================================= test session starts ====================
platform win32 -- Python 3.7.6, pytest-5.3.5, py-1.8.1, pluggy-0.13.1
rootdir: C:\Users\some-user\Egret-master
plugins: hypothesis-5.5.4, arraydiff-0.3, astropy-header-0.1.2, doctestplus-0.5.0, openfiles-0.4.0, remotedata-0.3.2
collected 21 items
test_unit_commitment.py ..................... [100%]
=========================================== 21 passed in 2804.02s (0:46:44) ==================
it took more than 46 min, isn't it a bit slow on my system?
2-This is the output of pytest --runmip test_unit_commitment.py
ERROR: usage: pytest [options] [file_or_dir] [file_or_dir] [...]
pytest: error: unrecognized arguments: --runmip
inifile: None
Any idea about fixing the error?
The most vexing of which is the description of the polynomial cost curve, for which we assume the 'values' dictionary has integer keys in several places. Here is the original.
{'data_type': 'cost_curve',
'cost_curve_type': 'polynomial',
'values': {0: 37, 1: 25.26, 2: 0.00712}}
Same after dumping/loading from json.
{'data_type': 'cost_curve',
'cost_curve_type': 'polynomial',
'values': {'0': 37, '1': 25.26, '2': 0.00712}}
Obviously this isn't a big deal technically, but how should we address it? Two obvious things are to make the model builders aware some things may be strings (and convert appropriately) or write a json parser that makes these substitutions where appropriate.
Headroom is measured as the least slack on all ramping-up constraints. However, when zero-cost reserves are used, the solver is indifferent to assigning that extra slack to the headroom. Two potential solutions:
The path in the file name construction line in acopy.py main should be:
matpower_file = os.path.join(path, 'tests/transmission_test_instances/pglib-o
pf-master/', filename)
It is common for data sources to have load across an area, and then "load participation factors" for each bus. We should consider making this part of the Egret ModelData specification to keep the user from having to re-implement boilerplate code to distribute the loads to the buses.
This wouldn't necessitate changing our models, just adding another "transformation" to the ModelData before handing it to a model. (The only transformation we have now does p.u. scaling.) We could also consider building-in other common transformations, such as reducing a network to zonal regions.
so the user can hand-pick lines to monitor or not, instead of/in addition to using the kV filter.
In egret.model_library.transmission.branch.declare_ineq_p_branch_thermal_lbub
, there is an if statement that checks if the approximation type is is BTHETA
or PTDF
. If so, constraints are added to limit the power flow. However, there is no else statement. We should at least raise an exception or warning rather than silently doing nothing.
Most unit commitment tests are regression-style tests and are generally too large, and hence take a bit of time to run. The unit commitment tests should be re-worked to have the following features:
We could keep some flagged regression tests (probably based of the current suite, or pglib-uc) as part of the re-work.
What does everyone think about a slight reorganization of Egret? I am thinking of the directory structure below. I have not fully flushed it out, but I think you will get the idea. I don't think the current structure is intuitive, but the structure I outlined below might be missing some complexities.
We should be careful modifying the pdtf_options
dictionary passed in by users
I pointed this out before but for the calls to collections.namedtuple()
in egret/parsers/rts_gmlc_parser.py
, the kwarg for "verbose" causes a TypeError for unexpected kwarg in Python 3.7. This was addressed in the prescient repo.
Some of the tests need an appropriate skip when cbc is not involved.
To make life easier on interior point solvers, at a minimum.
I see imports for six. Do we want to support Python 2.7?
Per #140, one of the relaxation test instances needed to be removed. Should a new one be added to take its place?
AC relaxations should expose the slack option for all relaxations. Here's one example:
Present:
def create_soc_relaxation(model_data, use_linear_relaxation=True):
model, md = _create_base_ac_model(model_data, include_feasibility_slack=False)
Proposed:
def create_soc_relaxation(model_data, use_linear_relaxation=True, include_feasibility_slack=False):
model, md = _create_base_ac_model(model_data, include_feasibility_slack=include_feasibility_slack)
With pyomo master (as of mid last week) and egret master (as of today), I am seeing the following when running "pytest test_unit_commitment" from the "models/tests" sub-directory. I am using gurobi 9.0.2. But this also happens with CBC 2.10.5.
Long version of output is further below - but short version is that the objective function for the model resulting from create_compact_unit_commitment is well off tolerance (by 7-ish asolute cost units).
Could someone - probably @bknueven - attempt to replicate? Thanks!
Long version of output is as follows:
test_unit_commitment.py F.................... [100%]
================================================== FAILURES ==================================================
___________________________________________ test_int_all_uc_models ___________________________________________
@unittest.skipUnless(comm_mip_avail, "Neither Gurobi or CPLEX solver is available")
def test_int_all_uc_models():
_test_uc_model(create_tight_unit_commitment_model)
_test_uc_model(create_compact_unit_commitment_model)
test_unit_commitment.py:86:
uc_model = <function create_compact_unit_commitment_model at 0x10fcba710>, relax = False
test_objvals = [4201915.017320504, 5454367.7670904165, 5999272.361123627, 5461120.3231092375, 6062406.32677043]
def _test_uc_model(uc_model, relax=False, test_objvals=test_int_objvals):
for test_case, ref_objval in zip(test_cases, test_objvals):
md_dict = json.load(open(test_case,'r'))
md = ModelData(md_dict)
model = uc_model(md, relaxed=relax)
opt = SolverFactory(test_solver)
_set_options(opt, mipgap=0.0)
if isinstance(opt, PersistentSolver):
opt.set_instance(model)
result = opt.solve(model, tee=False)
assert result.solver.termination_condition == TerminationCondition.optimal
assert math.isclose(ref_objval, result.problem.upper_bound, rel_tol=rel_tol)
E AssertionError: assert False
E + where False = (5454367.7670904165, 5454374.111340323, rel_tol=1e-08)
E + where = math.isclose
E + and 5454374.111340323 = <pyomo.opt.results.container.ListContainer object at 0x1137203d0>.upper_bound
E + where <pyomo.opt.results.container.ListContainer object at 0x1137203d0> = {'Problem': [{'Name': 'x8497', 'Lower bound': 5454374.111340323, 'Upper bound': 5454374.111340323, 'Number of objectiv...': 1.5290980339050293}], 'Solution': [OrderedDict([('number of solutions', 0), ('number of solutions displayed', 0)])]}.problem
test_unit_commitment.py:72: AssertionError
I am trying to follow through the instructions and install the package but I have been unsuccessful. I would appreciate some guidance and help, please point out if I am doing something obviously wrong. I am using Anaconda Prompt all through to attempt the package installation. I am using Windows 10 and Conda version 4.8.3.
conda create --name erget_test
conda install -c conda-forge pyomo
conda install -c conda-forge ipopt glpk
pip install pytest
conda install -c conda-forge coincbc
Additionally, the last instruction around running setup.py is not really clear, I have tried several different ways to invoke and run setup.py, but I am not sure I understand what works:
We temporarily disabled the test_logging() test function (by re-naming it to _test_logging()), as it was failing for reasons we did not understand - and the test failures were blocking merging of multiple PRs. However, the issue needs to be revisited and the test eventually restored.
...for the unit commitment / dispatch model. This is to enable more reasonable results at shorter time-intervals (e.g., 5-minutes).
The title says it all!
egret.common.solver_interface._solve_model
raises an exception if a solver terminates with a 'not good' termination condition. Conversely, the lazy PTDF solve loops do not raise exceptions (though likely cause them) if a solver terminates with an unsafe condition.
What should the default behavior of Egret be? Should we throw exceptions if a model is infeasible, or attempt to do something more graceful?
The .m files that represent the various cases to be solved (e.g. pglib_opf_case3_lmbd.m) cannot be located. In fact, the folder where these files are supposed to be located, 'pglib-opf-master', is also not located in the designated path i.e. egret\models\tests\transmission_test_instances. This causes the test to fail. Can someone please fix this issue?
For @rconcep - I suspect you want to restore the code in the main() in generate_graphs.py that is presently commented out - which would use the built-in test data in egret/models/tests/uc_test_instances. I would like to retain the general test functionality in this module, as it's a nice quick+dirty way to generate a graph - and provide an example of how to interact with the library.
See lines 275-276 in egret/model_library/transmission/bus.py. We check to see if the load is zero. If it is not, then we subtract m.pl[bus]
from the power balance. However, there is no indication that m.pl[bus]
should be a constant (immutable param or fixed var).
This is minor, but also easy to fix if agreed upon.
The function solve_dcopf in models/dcopf.py calls _solve_model (in common/solver_interface.py) to solve the DCOPF model once constructed. The default mipgap in _solve_model is 0.001. Further, solve_dcopf does not over-ride, even though it's continuous problem. I am proposing to set the mipgap keyword argument in the call to _solve_model from solve_dcopf to None, mainly to stop people from wondering why a mipgap is set for a continuous problem, when staring at the traces.
More generally, why is the mipgap default to something other than None in _solve_model? Especially given that it's distinct (I think) from at least the Gurobi default.
I have run the tests following the installation instructions (execute pytest test_unit_commitment.py
from models/tests sub-directory
). I have the latest Gurobi (AMPL-Gurobi 9.0.1) installed at macOS.
The result of the test is following: 21 failed, 1 warning in 814.40s (0:13:34)
The full log is here Gurobi log
I've tried to run the tests with different solvers, e.g.
SCIP: 18 failed, 3 passed, 1 warning in 241.02s (0:04:01)
log: SCIP log
CBC: 1 failed, 20 passed, 1 warning in 2765.15s (0:46:05)
log: CBC log
Can anybody give me some pointers how to fix the tests? Thanks!
Hi,
I tried to run unit_commitment.py in a standalone way by un-commenting the main at the end.
The first error is that there is no read_from_json(file) in ModelData, only read(file) method which is a typo.
The second error after fixing the above:
_File "c:\users\me\desktop\python-test\egret-master\egret\model_library\unit_commitment\uc_model_generator.py", line 100, in generate_model
baseMVA = model_data.data['system']['baseMVA']
KeyError: 'baseMVA'
I fixed this by adding an instance (self) version of the class method read(file) to ModelData. Any idea?
Thanks in advance!
Add the readthedocs harness for EGRET, including introductory documentation.
_include_feasibility_slack currently has a named argument called "penalty" that defaults to 1000. I suggest the following improvements:
replace this argument with two named arguments with defaults shown :
per_unit_slack_penalty=None and slack_factor=10
The code should assert that one is None and the other is not None.
The argument slack_factor would work exactly like the current argument called penalty.
The argument per_unit_slack_penalty would be assigned directly to p_penalty and q_penalty
These arguments should be exposed all the way up to the create_* function (e.g.,
all the way up to create_riv_acopf_model, etc.)
Aside: It might be good to have a standardized way to pass arguments from the highest levels of Egret to the lowest, but that is a different issue.
It would be good to have a convex relaxation of AC as a model itself.
For @rconcep : Can you add skip-if logic for the stackgraph generation tests, in case matplotlib or seaborn are not installed? I am going to add both of those packages as a dependency for auto install in setup.py, but I think it still would be a good idea in case viz packages are not properly installed / functioning on some platforms.
Specifically, on pglib_opf_case_300_ieee, but is noticeable on other larger cases. Could just be a numerical issue with the PTDF calculation itself.
You should ask users to cite the IJOC paper
A declarative, efficient, and flexible JavaScript library for building user interfaces.
๐ Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. ๐๐๐
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google โค๏ธ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.