ucd-cws / calvin Goto Github PK
View Code? Open in Web Editor NEWreservoir network optimization
License: MIT License
calvin's People
Contributors
Stargazers
Watchers
Forkers
hieulel muguangyuze zs00802 tanxuezhi nsosorio guohan0421 msdogan mlmaskey water-systems-management-ucm wyattarnold chenchonghust ruihui0821 maxwellmkondiwacalvin's Issues
blank lines printed by postprocess.py
Piecewise link upper bound
We currently do not explicitly impose a physical upper bound on the piecewise (economically-represented) links.
I do not think this is a problem, because the physical upper bound is used by Justin's matrix export code to divide up the pieces. In other words, as long as the sum of the upper bounds of the piecewise pieces is equal to the physical upper bound, we should be ok.
Say we have one link with bounds [0,100]. If we divide this into 4 pieces, they all have bounds [0,25], there is no danger of exceeding the physical upper bound of 100.
BUT, we may be picking the wrong dual values for these links. We currently choose the maximum one among the pieces. But should it be the last one instead?
Reason: we only want the dual on the PHYSICAL upper bound. The other ones are meaningless.
I'm putting this here just so we have a record of the conversation. But if we agree that this should be changed, I will assign myself to fix postprocess.py.
postprocessor translation
Based on meeting with @mfefer let's try to keep track of postprocessor features that we need. Here is a list of the current Excel postprocessors, with my understanding of what they do, how important they are, and how close we are to having those features written in python between postprocess.py and aggregate_regions.py. Please edit or correct anything that seems off.
-
Ag Post Processor: deliveries, shortage volume & cost, and marginal cost at each demand node. This is all done in
postprocess.pyexcept the marginal cost, which will be an important feature to add. Possibly by separating the economic stuff into a separate script as suggested by @msdogan. -
Ag Supply Portfolio: breaks down the % of supply type for each region. @mfefer is working on this now. Will be important to compare to previous work.
-
Basin Outlet Flow Comparison: This looks like it compares flows in CALVIN with CALSIM. I hadn't seen this before. Was this just for comparison purposes?
-
Debug Flow Finder: As the name suggests. We already have this written.
-
Desal Plants: Probably don't need this right now for CEC. Especially if desal is already included in the urban/ag portfolio calculations.
-
Flow Finder: Extract flow time series for particular link(s). We already have this ability by running
postprocess.pyand then filtering the resultingflow.csvfile with pandas. -
GW Recharge: Not sure what this does. Is this based on amplitudes?
-
GW Storage: We already get the GW storage time series from
postprocess.py. -
Hydropower: Will be interesting to do this later, maybe not an immediate goal. Needs to somehow take monthly flow values out of reservoirs and estimate the amount of energy generated in the process.
-
MIF_Marginals: Specific to environmental flow constraints. We already have these dual values, and is probably not a near-term focus.
-
Operating Costs: don't worry about for now
-
Pumping: not now
-
Refuge marginals: not now
-
SW_Evap: Max already did this in
postprocess.py -
SW_Pumping: not now ?
-
SW Storage Marginals: we already have this data
-
SW Storage Postprocessor: this just extracts the time series, we have this
-
Urban postprocessor: deliveries, shortage volume & cost, and marginal cost for urban delivery links (same as ag). We already have this.
-
Urban supply portfolio: Max is working on this now.
Reading through the Excel postprocessors, it seems like the most valuable information is the pathnames. (Which links are SW vs. GW vs. environmental ... etc.) We should think about how to preserve this somehow.
Right now I think the priorities are the portfolios. Then we can compare data with past CALVIN runs as soon as we get it working without debug mode. Thanks!
Solution gets infeasible if time period is greater than 4 years
@jdherman @whiteellie in debug mode, where we are two links: source and sink, to each node with a very high penalty to guarantee optimal solution, the GLPK solver can find a solution for a statewide run if the time-period is 4 years or less. For a 4-year run, the number of decision variables is 255,884, including debug links. However, for bigger time periods (5 or 10 years), pyomo creates the model, but the solver cannot find a solution.
This may not be a problem for limited foresight runs because we will run for one-year period. It might become an issue for someone wants to make a bigger time period run, though.
AD3+msdogan@CWS-Amargosa /z/pyvin_test_07242016
$ cnf matrix --format csv --start 1997-10 --stop 2002-10 --ts . --fs :tab: --to networklinks --max-ub 10000000 --outnodes networknodes --debug All
AD3+msdogan@CWS-Amargosa /z/pyvin_test_07242016
$ pyomo solve --solver=glpk --solver-suffix=dual /z/pyvin/pyvin.py data.dat --report-timing --json
[ 0.00] Setting up Pyomo environment
[ 0.00] Applying Pyomo preprocessing actions
[ 0.02] Creating model
0.26 seconds required to construct component=N; 64744 indicies total
0 seconds required to construct component=k; 16 indicies total
0 seconds required to construct component=A_domain_index_0; 67068568576 indicies total
0 seconds required to construct component=A_domain; 67068568576 indicies total
2.73 seconds required to construct component=A; 319858 indicies total
0 seconds required to construct component=source; 1 indicies total
0 seconds required to construct component=sink; 1 indicies total
0.98 seconds required to construct component=u; 319858 indicies total
0.99 seconds required to construct component=l; 319858 indicies total
0.97 seconds required to construct component=a; 319858 indicies total
0.99 seconds required to construct component=c; 319858 indicies total
0.44 seconds required to construct component=X; 319858 indicies total
3.06 seconds required to construct component=total; 1 indicies total
6.81 seconds required to construct component=limit_upper; 319858 indicies total
6.96 seconds required to construct component=limit_lower; 319858 indicies total
2.36 seconds required to construct component=_; 319858 indicies total
9.54 seconds required to construct component=flow; 64742 indicies total
Cloning detected! (clone counters: 29194, 128882, 0)
36.10 seconds required to construct instance=unknown
0.00 seconds required for problem transformations
[ 132.27] Applying solver
[ 199.33] Processing results
Number of solutions: 0
Solver results file: results.json
[ 199.35] Applying Pyomo postprocessing actions
[ 199.35] Pyomo Finished
results.json file:
{
"Problem": [
{
"Lower bound": -Infinity,
"Name": "unknown",
"Number of constraints": 704459,
"Number of nonzeros": 1279251,
"Number of objectives": 1,
"Number of variables": 319859,
"Sense": "minimize",
"Upper bound": Infinity
}
],
"Solver": [
{
"Error rc": 0,
"Statistics": {
"Branch and bound": {
"Number of bounded subproblems": 0,
"Number of created subproblems": 0
}
},
"Status": "ok",
"Termination condition": "infeasible",
"Time": 7.408999919891357
}
]
}
P.S. there is still some issues with matrix export, such as piecewise cost representation (ucd-cws/calvin-network-tools#36), but it should not affect the run and solution feasibility.
new debug mode: equal cost network
Following up March 2, 2017 meeting, we were having some problem with assigning debug flow amounts to constraints because LP model with perfect foresight puts debug flows in time and place where it is most beneficial (least-cost). For this reason, we were unable to pin point exact locations where mass balance problems occur. However, we can trick the system by equalizing the cost throughout the network. We can set all network cost to $1, except for SOURCE and SINK, which have a unit cost of $2000000, and then run the model. That way, we can make sure that PyVIN (in debug mode) will put debug flows in exact time and location where mass balance problems occur. After calibrating the model, we can run full-cost network and analyze results.
I have created a new branch: https://github.com/msdogan/pyvin/tree/debug_feasibility. There are two examples in this new branch. https://github.com/msdogan/pyvin/blob/debug_feasibility/examples/3-year_reduced_inflow/readme.md also explains those example runs.
Here are debug flows results:
full cost network 3-year run with 50% reduced inflow
Link: DBUGSRC-C37, Time: 2000-10-31, Flow: 1.6
Link: DBUGSRC-C37, Time: 2000-11-30, Flow: 1.75
Link: DBUGSRC-C37, Time: 2000-12-31, Flow: 2.25
Link: DBUGSRC-C37, Time: 2001-01-31, Flow: 4.2
Link: DBUGSRC-C37, Time: 2001-02-28, Flow: 9.5
Link: DBUGSRC-C37, Time: 2001-03-31, Flow: 13.75
Link: DBUGSRC-C37, Time: 2001-04-30, Flow: 13.5
Link: DBUGSRC-C37, Time: 2001-05-31, Flow: 8.05
Link: DBUGSRC-C37, Time: 2001-06-30, Flow: 1.3
Link: DBUGSRC-C37, Time: 2001-07-31, Flow: 0.45
Link: DBUGSRC-C37, Time: 2001-08-31, Flow: 0.2
Link: DBUGSRC-C37, Time: 2001-09-30, Flow: 0.2
Link: DBUGSRC-C37, Time: 2001-10-31, Flow: 0.3
Link: DBUGSRC-C37, Time: 2001-11-30, Flow: 1.7
Link: DBUGSRC-C37, Time: 2001-12-31, Flow: 9.25
Link: DBUGSRC-C37, Time: 2002-01-31, Flow: 16.5
Link: DBUGSRC-C37, Time: 2002-02-28, Flow: 15.85
Link: DBUGSRC-C37, Time: 2002-03-31, Flow: 26.45
Link: DBUGSRC-C37, Time: 2002-04-30, Flow: 16.9
Link: DBUGSRC-C37, Time: 2002-05-31, Flow: 9.85
Link: DBUGSRC-C37, Time: 2002-06-30, Flow: 3.05
Link: DBUGSRC-C37, Time: 2002-07-31, Flow: 0.7
Link: DBUGSRC-C37, Time: 2002-08-31, Flow: 0.25
Link: DBUGSRC-C37, Time: 2002-09-30, Flow: 0.15
Link: DBUGSRC-C37, Time: 2002-10-31, Flow: 0.25
Link: DBUGSRC-C37, Time: 2002-11-30, Flow: 1.75
Link: DBUGSRC-C37, Time: 2002-12-31, Flow: 7.6
Link: DBUGSRC-C37, Time: 2003-01-31, Flow: 8.5
Link: DBUGSRC-C37, Time: 2003-02-28, Flow: 7.15
Link: DBUGSRC-C37, Time: 2003-03-31, Flow: 10.55
Link: DBUGSRC-C37, Time: 2003-04-30, Flow: 30.7
Link: DBUGSRC-C37, Time: 2003-05-31, Flow: 33.6
Link: DBUGSRC-C37, Time: 2003-06-30, Flow: 6.25
Link: DBUGSRC-C37, Time: 2003-07-31, Flow: 1.15
Link: DBUGSRC-C37, Time: 2003-08-31, Flow: 0.65
Link: DBUGSRC-C37, Time: 2003-09-30, Flow: 0.25
Link: DBUGSRC-C41, Time: 2001-02-28, Flow: 22.5371260741254
Link: DBUGSRC-C41, Time: 2001-11-30, Flow: 10.6827673719919
Link: DBUGSRC-C41, Time: 2001-12-31, Flow: 1.89721894770561
Link: DBUGSRC-C56, Time: 2000-11-30, Flow: 2.12491921442664
Link: DBUGSRC-C56, Time: 2000-12-31, Flow: 6.72742782086565
Link: DBUGSRC-C58, Time: 2001-09-30, Flow: 0.801027323180296
Link: DBUGSRC-C9, Time: 2003-09-30, Flow: 0.595
Link: DBUGSRC-D17, Time: 2002-09-30, Flow: 49.9323045974117
Link: DBUGSRC-D17, Time: 2002-10-31, Flow: 52.0634529261703
Link: DBUGSRC-D17, Time: 2002-11-30, Flow: 57.1846660387567
Link: DBUGSRC-D17, Time: 2002-12-31, Flow: 25.8315687199047
Link: DBUGSRC-D17, Time: 2003-02-28, Flow: 7.31078327054582
Link: DBUGSRC-D17, Time: 2003-08-31, Flow: 63.5127363507922
Link: DBUGSRC-GW_IM, Time: 2003-09-30, Flow: 37.4999999999995
Link: DBUGSRC-HSR202, Time: 2001-10-31, Flow: 0.59
Link: DBUGSRC-HSR202, Time: 2002-09-30, Flow: 0.85
Link: DBUGSRC-HSUR101, Time: 2000-10-31, Flow: 18.065
Link: DBUGSRC-HSUR101, Time: 2000-11-30, Flow: 15.575
Link: DBUGSRC-HSUR101, Time: 2000-12-31, Flow: 10.275
Link: DBUGSRC-HSUR101, Time: 2001-05-31, Flow: 4.315
Link: DBUGSRC-HSUR101, Time: 2001-06-30, Flow: 6.5
Link: DBUGSRC-HSUR101, Time: 2001-07-31, Flow: 9.33
Link: DBUGSRC-HSUR101, Time: 2001-08-31, Flow: 12.705
Link: DBUGSRC-HSUR101, Time: 2001-10-31, Flow: 18.065
Link: DBUGSRC-HSUR101, Time: 2002-05-31, Flow: 4.315
Link: DBUGSRC-HSUR101, Time: 2002-06-30, Flow: 6.5
Link: DBUGSRC-HSUR101, Time: 2002-07-31, Flow: 9.33
Link: DBUGSRC-HSUR101, Time: 2002-08-31, Flow: 12.705
Link: DBUGSRC-HSUR101, Time: 2002-09-30, Flow: 16.065
Link: DBUGSRC-HSUR101, Time: 2002-10-31, Flow: 18.065
Link: DBUGSRC-HSUR101, Time: 2002-11-30, Flow: 15.575
Link: DBUGSRC-HSUR101, Time: 2003-01-31, Flow: 4.825
Link: DBUGSRC-HSUR101, Time: 2003-06-30, Flow: 4.46722500000001
Link: DBUGSRC-HSUR101, Time: 2003-07-31, Flow: 9.33
Link: DBUGSRC-HSUR101, Time: 2003-08-31, Flow: 12.705
Link: DBUGSRC-HSUR101, Time: 2003-09-30, Flow: 16.065
Link: DBUGSRC-HSUR201, Time: 2001-05-31, Flow: 2.06
Link: DBUGSRC-HSUR201, Time: 2001-06-30, Flow: 2.42
Link: DBUGSRC-HSUR201, Time: 2001-07-31, Flow: 2.43
Link: DBUGSRC-HSUR201, Time: 2001-08-31, Flow: 3.11
Link: DBUGSRC-HSUR201, Time: 2001-10-31, Flow: 12.62
Link: DBUGSRC-HSUR201, Time: 2002-06-30, Flow: 2.42
Link: DBUGSRC-HSUR201, Time: 2002-07-31, Flow: 2.43
Link: DBUGSRC-HSUR201, Time: 2002-08-31, Flow: 3.11
Link: DBUGSRC-HSUR201, Time: 2002-09-30, Flow: 8.65
Link: DBUGSRC-HSUR201, Time: 2002-10-31, Flow: 12.62
Link: DBUGSRC-HSUR201, Time: 2002-11-30, Flow: 3.33
Link: DBUGSRC-HSUR201, Time: 2003-07-31, Flow: 0.48
Link: DBUGSRC-HSUR201, Time: 2003-08-31, Flow: 3.11
Link: DBUGSRC-HSUR201, Time: 2003-09-30, Flow: 1.95623999999999
Link: DBUGSRC-HSUR202, Time: 2001-09-30, Flow: 0.1
Link: DBUGSRC-HSUR202, Time: 2001-10-31, Flow: 1.31
Link: DBUGSRC-HSUR202, Time: 2002-09-30, Flow: 0.15
Link: DBUGSRC-HSUR202, Time: 2002-10-31, Flow: 2.25
Link: DBUGSRC-N201, Time: 2000-11-30, Flow: 0.990315022641533
Link: DBUGSRC-N201, Time: 2000-12-31, Flow: 19.1303157006838
Link: DBUGSRC-N201, Time: 2001-01-31, Flow: 13.858223946341
Link: DBUGSRC-N201, Time: 2001-02-28, Flow: 8.98405217983515
Link: DBUGSRC-N201, Time: 2001-03-31, Flow: 5.70702971066056
Link: DBUGSRC-N201, Time: 2001-04-30, Flow: 8.82066328270771
Link: DBUGSRC-N201, Time: 2001-05-31, Flow: 10.5912792542244
Link: DBUGSRC-N201, Time: 2001-06-30, Flow: 10.8447967745505
Link: DBUGSRC-N201, Time: 2001-07-31, Flow: 9.33430672213884
Link: DBUGSRC-N201, Time: 2001-08-31, Flow: 10.0904134111961
Link: DBUGSRC-N201, Time: 2001-09-30, Flow: 15.2710010300783
Link: DBUGSRC-N201, Time: 2001-10-31, Flow: 19.6338744477468
Link: DBUGSRC-N201, Time: 2001-11-30, Flow: 5.72479018264154
Link: DBUGSRC-N201, Time: 2002-09-30, Flow: 8.0762006373674
Link: DBUGSRC-N201, Time: 2002-10-31, Flow: 24.4093744477468
Link: DBUGSRC-N201, Time: 2002-11-30, Flow: 16.9597901826415
Link: DBUGSRC-N201, Time: 2003-02-28, Flow: 3.17256062411766
Link: DBUGSRC-N201, Time: 2003-05-31, Flow: 1.03324978088385
Link: DBUGSRC-N201, Time: 2003-06-30, Flow: 8.78479677455053
Link: DBUGSRC-N201, Time: 2003-07-31, Flow: 8.49830672213884
Link: DBUGSRC-N201, Time: 2003-08-31, Flow: 18.1674134111961
Link: DBUGSRC-N201, Time: 2003-09-30, Flow: 36.32
Link: DBUGSRC-N202, Time: 2001-05-31, Flow: 2.323
Link: DBUGSRC-N202, Time: 2001-06-30, Flow: 3.067
Link: DBUGSRC-N202, Time: 2001-07-31, Flow: 4.174
Link: DBUGSRC-N202, Time: 2001-08-31, Flow: 3.128
Link: DBUGSRC-N202, Time: 2001-09-30, Flow: 0.409999999999999
Link: DBUGSRC-N202, Time: 2002-09-30, Flow: 1.618
Link: DBUGSRC-N202, Time: 2003-06-30, Flow: 3.469
Link: DBUGSRC-N202, Time: 2003-07-31, Flow: 4.174
Link: DBUGSRC-N202, Time: 2003-08-31, Flow: 0.793
Link: DBUGSRC-N202, Time: 2003-09-30, Flow: 1.376
Link: DBUGSRC-SR_BLB, Time: 2000-10-31, Flow: 0.00010195873386389
Link: DBUGSRC-SR_BLB, Time: 2000-11-30, Flow: 0.749372017005574
Link: DBUGSRC-SR_BLB, Time: 2000-12-31, Flow: 0.0647471914053523
Link: DBUGSRC-SR_BLB, Time: 2001-10-31, Flow: 0.131754679723546
Link: DBUGSRC-SR_BLB, Time: 2002-07-31, Flow: 0.140856496739714
Link: DBUGSRC-SR_BLB, Time: 2002-08-31, Flow: 0.130788479267339
Link: DBUGSRC-SR_BLB, Time: 2002-09-30, Flow: 1.88794905082794
Link: DBUGSRC-SR_BLB, Time: 2002-10-31, Flow: 0.0758713109716156
Link: DBUGSRC-SR_BLB, Time: 2003-08-31, Flow: 1.95366818296689
Link: DBUGSRC-SR_BLB, Time: 2003-09-30, Flow: 71.985
Link: DBUGSRC-SR_BUL, Time: 2003-09-30, Flow: 5.51096536715919
Link: DBUGSRC-SR_CLE, Time: 2002-10-31, Flow: 110.078076271044
Link: DBUGSRC-SR_CLE, Time: 2002-11-30, Flow: 208.800675087544
Link: DBUGSRC-SR_CLE, Time: 2003-01-31, Flow: 170.506787279709
Link: DBUGSRC-SR_CLE, Time: 2003-08-31, Flow: 76.8003623582125
Link: DBUGSRC-SR_CLE, Time: 2003-09-30, Flow: 890.154
Link: DBUGSRC-SR_FOL, Time: 2003-09-30, Flow: 616.3085
Link: DBUGSRC-SR_GNT, Time: 2001-01-31, Flow: 2.48489524382484
Link: DBUGSRC-SR_GNT, Time: 2001-02-28, Flow: 3.77295064830636
Link: DBUGSRC-SR_GNT, Time: 2001-03-31, Flow: 3.05050324898091
Link: DBUGSRC-SR_GNT, Time: 2001-04-30, Flow: 0.671484158676749
Link: DBUGSRC-SR_GNT, Time: 2001-09-30, Flow: 4.25921509268358
Link: DBUGSRC-SR_GNT, Time: 2001-10-31, Flow: 4.6026138465586
Link: DBUGSRC-SR_GNT, Time: 2001-11-30, Flow: 4.00187519876014
Link: DBUGSRC-SR_GNT, Time: 2001-12-31, Flow: 3.9312210553793
Link: DBUGSRC-SR_GNT, Time: 2002-01-31, Flow: 3.78730066566933
Link: DBUGSRC-SR_GNT, Time: 2002-02-28, Flow: 3.38145064830636
Link: DBUGSRC-SR_GNT, Time: 2002-03-31, Flow: 3.36450324898091
Link: DBUGSRC-SR_GNT, Time: 2002-09-30, Flow: 0.854476197877762
Link: DBUGSRC-SR_GNT, Time: 2002-10-31, Flow: 4.55911384655859
Link: DBUGSRC-SR_GNT, Time: 2002-11-30, Flow: 3.93937519876014
Link: DBUGSRC-SR_GNT, Time: 2002-12-31, Flow: 4.2247210553793
Link: DBUGSRC-SR_GNT, Time: 2003-01-31, Flow: 3.91780066566933
Link: DBUGSRC-SR_GNT, Time: 2003-02-28, Flow: 3.19845064830636
Link: DBUGSRC-SR_GNT, Time: 2003-03-31, Flow: 3.19400324898091
Link: DBUGSRC-SR_GNT, Time: 2003-04-30, Flow: 3.13398415867675
Link: DBUGSRC-SR_GNT, Time: 2003-09-30, Flow: 2.35417898173205
Link: DBUGSRC-SR_HID, Time: 2003-09-30, Flow: 4.07676144275653
Link: DBUGSRC-SR_NHG, Time: 2003-09-30, Flow: 17.3676775600299
Link: DBUGSRC-SR_RLL_CMB, Time: 2003-09-30, Flow: 31.914
Link: DBUGSRC-SR_SCC, Time: 2002-07-31, Flow: 4.79721228594413
Link: DBUGSRC-SR_SCC, Time: 2002-08-31, Flow: 2.09279684659845
Link: DBUGSRC-SR_SCC, Time: 2002-09-30, Flow: 3.00468323769867
Link: DBUGSRC-SR_SCC, Time: 2003-09-30, Flow: 10.0064066780757
Link: DBUGSRC-SR_SHA, Time: 2002-10-31, Flow: 4.62726971446928
Link: DBUGSRC-SR_SHA, Time: 2002-11-30, Flow: 1.84803598880489
Link: DBUGSRC-SR_SHA, Time: 2003-09-30, Flow: 1627.7472
Link: DBUGSRC-SR_TRM, Time: 2003-09-30, Flow: 10.4805637012471
Link: DBUGSRC-SR_WHI, Time: 2002-10-31, Flow: 41.5171300682457
Link: DBUGSRC-SR_WHI, Time: 2002-11-30, Flow: 40.2775196274784
Link: DBUGSRC-SR_WHI, Time: 2003-09-30, Flow: 262.6185
Link: DBUGSRC-U102, Time: 2000-10-31, Flow: 11.18
Link: DBUGSRC-U102, Time: 2000-11-30, Flow: 10.09
Link: DBUGSRC-U102, Time: 2000-12-31, Flow: 8.74
Link: DBUGSRC-U102, Time: 2001-01-31, Flow: 9.7
Link: DBUGSRC-U102, Time: 2001-02-28, Flow: 10.47
Link: DBUGSRC-U102, Time: 2001-03-31, Flow: 14.06
Link: DBUGSRC-U102, Time: 2001-04-30, Flow: 19.64
Link: DBUGSRC-U102, Time: 2001-05-31, Flow: 25.96
Link: DBUGSRC-U102, Time: 2001-06-30, Flow: 30.88
Link: DBUGSRC-U102, Time: 2001-07-31, Flow: 29.69
Link: DBUGSRC-U102, Time: 2001-08-31, Flow: 15.784709999977
Link: DBUGSRC-U102, Time: 2001-11-30, Flow: 10.09
Link: DBUGSRC-U102, Time: 2001-12-31, Flow: 8.74
Link: DBUGSRC-U102, Time: 2002-01-31, Flow: 9.7
Link: DBUGSRC-U102, Time: 2002-10-31, Flow: 11.18
Link: DBUGSRC-U102, Time: 2002-12-31, Flow: 8.74
Link: DBUGSRC-U102, Time: 2003-01-31, Flow: 9.7
Link: DBUGSRC-U102, Time: 2003-02-28, Flow: 10.47
Link: DBUGSRC-U102, Time: 2003-08-31, Flow: 22.9
Link: DBUGSRC-U102, Time: 2003-09-30, Flow: 15.87
Link: DBUGSRC-U103, Time: 2000-10-31, Flow: 2.36
Link: DBUGSRC-U103, Time: 2000-11-30, Flow: 2.15
Link: DBUGSRC-U103, Time: 2000-12-31, Flow: 2.15
Link: DBUGSRC-U103, Time: 2001-01-31, Flow: 2.15
Link: DBUGSRC-U103, Time: 2001-02-28, Flow: 2.2
Link: DBUGSRC-U103, Time: 2001-03-31, Flow: 3.37
Link: DBUGSRC-U103, Time: 2001-04-30, Flow: 5.41
Link: DBUGSRC-U103, Time: 2001-05-31, Flow: 7.2
Link: DBUGSRC-U103, Time: 2001-06-30, Flow: 7.99
Link: DBUGSRC-U103, Time: 2001-07-31, Flow: 7.8
Link: DBUGSRC-U103, Time: 2001-08-31, Flow: 5.81
Link: DBUGSRC-U103, Time: 2001-09-30, Flow: 3.71
Link: DBUGSRC-U103, Time: 2001-10-31, Flow: 2.36
Link: DBUGSRC-U103, Time: 2001-11-30, Flow: 2.15
Link: DBUGSRC-U103, Time: 2001-12-31, Flow: 2.15
Link: DBUGSRC-U103, Time: 2002-01-31, Flow: 2.15
Link: DBUGSRC-U103, Time: 2002-02-28, Flow: 2.2
Link: DBUGSRC-U103, Time: 2002-03-31, Flow: 3.37
Link: DBUGSRC-U103, Time: 2002-04-30, Flow: 5.41
Link: DBUGSRC-U103, Time: 2002-05-31, Flow: 7.2
Link: DBUGSRC-U103, Time: 2002-06-30, Flow: 7.09386625466021
Link: DBUGSRC-U103, Time: 2002-07-31, Flow: 7.8
Link: DBUGSRC-U103, Time: 2002-10-31, Flow: 2.36
Link: DBUGSRC-U103, Time: 2002-11-30, Flow: 2.15
Link: DBUGSRC-U103, Time: 2002-12-31, Flow: 2.15
Link: DBUGSRC-U103, Time: 2003-01-31, Flow: 2.15
Link: DBUGSRC-U103, Time: 2003-02-28, Flow: 2.2
Link: DBUGSRC-U103, Time: 2003-03-31, Flow: 3.37
Link: DBUGSRC-U103, Time: 2003-04-30, Flow: 5.41
Link: DBUGSRC-U103, Time: 2003-06-30, Flow: 7.99
Link: DBUGSRC-U103, Time: 2003-08-31, Flow: 5.81
Link: DBUGSRC-U103, Time: 2003-09-30, Flow: 3.71
Link: DBUGSRC-U104, Time: 2000-10-31, Flow: 1.64634342312337
Link: DBUGSRC-U104, Time: 2003-09-30, Flow: 3.17
Link: DBUGSRC-U202, Time: 2000-10-31, Flow: 5.79
Link: DBUGSRC-U202, Time: 2000-11-30, Flow: 5.34
Link: DBUGSRC-U202, Time: 2000-12-31, Flow: 3.96
Link: DBUGSRC-U202, Time: 2001-01-31, Flow: 0.085444860016877
Link: DBUGSRC-U202, Time: 2003-08-31, Flow: 9.58
Link: DBUGSRC-U202, Time: 2003-09-30, Flow: 7.27
equal cost network 3-year run with 50% reduced inflow
Link: DBUGSRC-C37, Time: 2000-10-31, Flow: 1.6
Link: DBUGSRC-C37, Time: 2000-11-30, Flow: 1.75
Link: DBUGSRC-C37, Time: 2000-12-31, Flow: 2.25
Link: DBUGSRC-C37, Time: 2001-01-31, Flow: 4.2
Link: DBUGSRC-C37, Time: 2001-02-28, Flow: 9.5
Link: DBUGSRC-C37, Time: 2001-03-31, Flow: 13.75
Link: DBUGSRC-C37, Time: 2001-04-30, Flow: 13.5
Link: DBUGSRC-C37, Time: 2001-05-31, Flow: 8.05
Link: DBUGSRC-C37, Time: 2001-06-30, Flow: 1.3
Link: DBUGSRC-C37, Time: 2001-07-31, Flow: 0.45
Link: DBUGSRC-C37, Time: 2001-08-31, Flow: 0.2
Link: DBUGSRC-C37, Time: 2001-09-30, Flow: 0.2
Link: DBUGSRC-C37, Time: 2001-10-31, Flow: 0.3
Link: DBUGSRC-C37, Time: 2001-11-30, Flow: 1.7
Link: DBUGSRC-C37, Time: 2001-12-31, Flow: 9.25
Link: DBUGSRC-C37, Time: 2002-01-31, Flow: 16.5
Link: DBUGSRC-C37, Time: 2002-02-28, Flow: 15.85
Link: DBUGSRC-C37, Time: 2002-03-31, Flow: 26.45
Link: DBUGSRC-C37, Time: 2002-04-30, Flow: 16.9
Link: DBUGSRC-C37, Time: 2002-05-31, Flow: 9.85
Link: DBUGSRC-C37, Time: 2002-06-30, Flow: 3.05
Link: DBUGSRC-C37, Time: 2002-07-31, Flow: 0.7
Link: DBUGSRC-C37, Time: 2002-08-31, Flow: 0.25
Link: DBUGSRC-C37, Time: 2002-09-30, Flow: 0.15
Link: DBUGSRC-C37, Time: 2002-10-31, Flow: 0.25
Link: DBUGSRC-C37, Time: 2002-11-30, Flow: 1.75
Link: DBUGSRC-C37, Time: 2002-12-31, Flow: 7.6
Link: DBUGSRC-C37, Time: 2003-01-31, Flow: 8.5
Link: DBUGSRC-C37, Time: 2003-02-28, Flow: 7.15
Link: DBUGSRC-C37, Time: 2003-03-31, Flow: 10.55
Link: DBUGSRC-C37, Time: 2003-04-30, Flow: 30.7
Link: DBUGSRC-C37, Time: 2003-05-31, Flow: 33.6
Link: DBUGSRC-C37, Time: 2003-06-30, Flow: 6.25
Link: DBUGSRC-C37, Time: 2003-07-31, Flow: 1.15
Link: DBUGSRC-C37, Time: 2003-08-31, Flow: 0.65
Link: DBUGSRC-C37, Time: 2003-09-30, Flow: 0.25
Link: DBUGSRC-C41, Time: 2001-02-28, Flow: 22.5371260741254
Link: DBUGSRC-C41, Time: 2001-11-30, Flow: 10.6827673719919
Link: DBUGSRC-C41, Time: 2001-12-31, Flow: 1.89721894770561
Link: DBUGSRC-C56, Time: 2000-11-30, Flow: 2.12491921442664
Link: DBUGSRC-C56, Time: 2000-12-31, Flow: 6.72742782086565
Link: DBUGSRC-C56, Time: 2003-09-30, Flow: 0.79
Link: DBUGSRC-C58, Time: 2001-09-30, Flow: 0.801027323180296
Link: DBUGSRC-C58, Time: 2002-07-31, Flow: 4.79721228594413
Link: DBUGSRC-C58, Time: 2002-08-31, Flow: 2.09279684659845
Link: DBUGSRC-C58, Time: 2002-09-30, Flow: 2.996
Link: DBUGSRC-C89, Time: 2003-09-30, Flow: 0.001
Link: DBUGSRC-C9, Time: 2003-09-30, Flow: 0.595
Link: DBUGSRC-D5, Time: 2002-10-31, Flow: 28.2201891360836
Link: DBUGSRC-D5, Time: 2002-11-30, Flow: 81.2713845422047
Link: DBUGSRC-D5, Time: 2003-09-30, Flow: 193.387
Link: DBUGSRC-D61, Time: 2002-10-31, Flow: 127.9856
Link: DBUGSRC-D61, Time: 2002-11-30, Flow: 146.7566514466
Link: DBUGSRC-D61, Time: 2003-08-31, Flow: 57.8358060054333
Link: DBUGSRC-D61, Time: 2003-09-30, Flow: 66.1797
Link: DBUGSRC-D9, Time: 2002-09-30, Flow: 49.9323045974117
Link: DBUGSRC-D9, Time: 2002-10-31, Flow: 52.0634529261703
Link: DBUGSRC-D9, Time: 2002-11-30, Flow: 57.1846660387567
Link: DBUGSRC-D9, Time: 2002-12-31, Flow: 25.8315687199047
Link: DBUGSRC-D9, Time: 2003-02-28, Flow: 7.31078327054582
Link: DBUGSRC-D9, Time: 2003-08-31, Flow: 63.5127363507922
Link: DBUGSRC-D9, Time: 2003-09-30, Flow: 91.239
Link: DBUGSRC-D94, Time: 2002-11-30, Flow: 12.8006750875437
Link: DBUGSRC-D94, Time: 2003-01-31, Flow: 17.573
Link: DBUGSRC-D94, Time: 2003-08-31, Flow: 15.1759709712083
Link: DBUGSRC-D94, Time: 2003-09-30, Flow: 26.7415
Link: DBUGSRC-GW_02, Time: 2003-09-30, Flow: 254.814709999935
Link: DBUGSRC-GW_03, Time: 2003-09-30, Flow: 121.963866254605
Link: DBUGSRC-GW_05, Time: 2003-09-30, Flow: 37.975444860035
Link: DBUGSRC-GW_IM, Time: 2003-09-30, Flow: 37.4999999999995
Link: DBUGSRC-HSUR101, Time: 2000-10-31, Flow: 18.065
Link: DBUGSRC-HSUR101, Time: 2000-11-30, Flow: 15.575
Link: DBUGSRC-HSUR101, Time: 2000-12-31, Flow: 10.275
Link: DBUGSRC-HSUR101, Time: 2001-05-31, Flow: 4.315
Link: DBUGSRC-HSUR101, Time: 2001-06-30, Flow: 6.5
Link: DBUGSRC-HSUR101, Time: 2001-07-31, Flow: 9.33
Link: DBUGSRC-HSUR101, Time: 2001-08-31, Flow: 12.705
Link: DBUGSRC-HSUR101, Time: 2001-10-31, Flow: 18.065
Link: DBUGSRC-HSUR101, Time: 2002-05-31, Flow: 4.315
Link: DBUGSRC-HSUR101, Time: 2002-06-30, Flow: 6.5
Link: DBUGSRC-HSUR101, Time: 2002-07-31, Flow: 9.33
Link: DBUGSRC-HSUR101, Time: 2002-08-31, Flow: 12.705
Link: DBUGSRC-HSUR101, Time: 2002-09-30, Flow: 16.065
Link: DBUGSRC-HSUR101, Time: 2002-10-31, Flow: 18.065
Link: DBUGSRC-HSUR101, Time: 2002-11-30, Flow: 15.575
Link: DBUGSRC-HSUR101, Time: 2003-01-31, Flow: 4.825
Link: DBUGSRC-HSUR101, Time: 2003-06-30, Flow: 4.46722500000001
Link: DBUGSRC-HSUR101, Time: 2003-07-31, Flow: 9.33
Link: DBUGSRC-HSUR101, Time: 2003-08-31, Flow: 12.705
Link: DBUGSRC-HSUR101, Time: 2003-09-30, Flow: 16.065
Link: DBUGSRC-HSUR201, Time: 2001-05-31, Flow: 0.11
Link: DBUGSRC-HSUR201, Time: 2001-06-30, Flow: 0.47
Link: DBUGSRC-HSUR201, Time: 2001-07-31, Flow: 0.48
Link: DBUGSRC-HSUR201, Time: 2001-08-31, Flow: 1.16
Link: DBUGSRC-HSUR201, Time: 2001-10-31, Flow: 10.67
Link: DBUGSRC-HSUR201, Time: 2002-06-30, Flow: 0.47
Link: DBUGSRC-HSUR201, Time: 2002-07-31, Flow: 0.48
Link: DBUGSRC-HSUR201, Time: 2002-08-31, Flow: 1.16
Link: DBUGSRC-HSUR201, Time: 2002-09-30, Flow: 6.7
Link: DBUGSRC-HSUR201, Time: 2002-10-31, Flow: 10.67
Link: DBUGSRC-HSUR201, Time: 2002-11-30, Flow: 3.33
Link: DBUGSRC-HSUR201, Time: 2003-07-31, Flow: 2.43
Link: DBUGSRC-HSUR201, Time: 2003-08-31, Flow: 3.11
Link: DBUGSRC-HSUR201, Time: 2003-09-30, Flow: 1.95623999999999
Link: DBUGSRC-N201, Time: 2000-11-30, Flow: 0.990315022641525
Link: DBUGSRC-N201, Time: 2000-12-31, Flow: 19.1303157006838
Link: DBUGSRC-N201, Time: 2001-01-31, Flow: 13.858223946341
Link: DBUGSRC-N201, Time: 2001-02-28, Flow: 8.98405217983515
Link: DBUGSRC-N201, Time: 2001-03-31, Flow: 5.70702971066056
Link: DBUGSRC-N201, Time: 2001-04-30, Flow: 8.82066328270771
Link: DBUGSRC-N201, Time: 2001-05-31, Flow: 10.5912792542244
Link: DBUGSRC-N201, Time: 2001-06-30, Flow: 10.8447967745505
Link: DBUGSRC-N201, Time: 2001-07-31, Flow: 9.33430672213884
Link: DBUGSRC-N201, Time: 2001-08-31, Flow: 10.0904134111961
Link: DBUGSRC-N201, Time: 2001-09-30, Flow: 15.2710010300783
Link: DBUGSRC-N201, Time: 2001-10-31, Flow: 19.6338744477468
Link: DBUGSRC-N201, Time: 2001-11-30, Flow: 5.72479018264154
Link: DBUGSRC-N201, Time: 2002-09-30, Flow: 8.0762006373674
Link: DBUGSRC-N201, Time: 2002-10-31, Flow: 24.4093744477468
Link: DBUGSRC-N201, Time: 2002-11-30, Flow: 16.9597901826415
Link: DBUGSRC-N201, Time: 2003-02-28, Flow: 3.17256062411766
Link: DBUGSRC-N201, Time: 2003-05-31, Flow: 1.03324978088384
Link: DBUGSRC-N201, Time: 2003-06-30, Flow: 8.78479677455054
Link: DBUGSRC-N201, Time: 2003-07-31, Flow: 8.49830672213884
Link: DBUGSRC-N201, Time: 2003-08-31, Flow: 18.1674134111961
Link: DBUGSRC-N201, Time: 2003-09-30, Flow: 36.32
Link: DBUGSRC-N202, Time: 2001-05-31, Flow: 2.323
Link: DBUGSRC-N202, Time: 2001-06-30, Flow: 3.067
Link: DBUGSRC-N202, Time: 2001-07-31, Flow: 4.174
Link: DBUGSRC-N202, Time: 2001-08-31, Flow: 3.128
Link: DBUGSRC-N202, Time: 2001-09-30, Flow: 0.409999999999999
Link: DBUGSRC-N202, Time: 2002-09-30, Flow: 1.618
Link: DBUGSRC-N202, Time: 2003-06-30, Flow: 3.469
Link: DBUGSRC-N202, Time: 2003-07-31, Flow: 4.174
Link: DBUGSRC-N202, Time: 2003-08-31, Flow: 0.793
Link: DBUGSRC-N202, Time: 2003-09-30, Flow: 1.376
Link: DBUGSRC-SR_BLB, Time: 2000-10-31, Flow: 0.00010195873386389
Link: DBUGSRC-SR_BLB, Time: 2000-11-30, Flow: 0.749372017005574
Link: DBUGSRC-SR_BLB, Time: 2000-12-31, Flow: 0.064747191405353
Link: DBUGSRC-SR_BLB, Time: 2001-10-31, Flow: 0.131754679723546
Link: DBUGSRC-SR_BLB, Time: 2002-07-31, Flow: 0.140856496739714
Link: DBUGSRC-SR_BLB, Time: 2002-08-31, Flow: 0.130788479267339
Link: DBUGSRC-SR_BLB, Time: 2002-09-30, Flow: 1.88794905082794
Link: DBUGSRC-SR_BLB, Time: 2002-10-31, Flow: 0.0758713109716156
Link: DBUGSRC-SR_BLB, Time: 2003-08-31, Flow: 1.95366818296689
Link: DBUGSRC-SR_BLB, Time: 2003-09-30, Flow: 71.985
Link: DBUGSRC-SR_CLE, Time: 2003-01-31, Flow: 152.933787279709
Link: DBUGSRC-SR_CLE, Time: 2003-09-30, Flow: 667.4125
Link: DBUGSRC-SR_ENG, Time: 2003-09-30, Flow: 5.51096536715984
Link: DBUGSRC-SR_FOL, Time: 2003-09-30, Flow: 525.0695
Link: DBUGSRC-SR_GNT, Time: 2001-01-31, Flow: 2.48489524382484
Link: DBUGSRC-SR_GNT, Time: 2001-02-28, Flow: 3.77295064830636
Link: DBUGSRC-SR_GNT, Time: 2001-03-31, Flow: 3.05050324898091
Link: DBUGSRC-SR_GNT, Time: 2001-04-30, Flow: 0.671484158676749
Link: DBUGSRC-SR_GNT, Time: 2001-09-30, Flow: 4.25921509268358
Link: DBUGSRC-SR_GNT, Time: 2001-10-31, Flow: 4.6026138465586
Link: DBUGSRC-SR_GNT, Time: 2001-11-30, Flow: 4.00187519876014
Link: DBUGSRC-SR_GNT, Time: 2001-12-31, Flow: 3.9312210553793
Link: DBUGSRC-SR_GNT, Time: 2002-01-31, Flow: 3.78730066566933
Link: DBUGSRC-SR_GNT, Time: 2002-02-28, Flow: 3.38145064830636
Link: DBUGSRC-SR_GNT, Time: 2002-03-31, Flow: 3.36450324898091
Link: DBUGSRC-SR_GNT, Time: 2002-09-30, Flow: 0.854476197877762
Link: DBUGSRC-SR_GNT, Time: 2002-10-31, Flow: 4.55911384655859
Link: DBUGSRC-SR_GNT, Time: 2002-11-30, Flow: 3.93937519876014
Link: DBUGSRC-SR_GNT, Time: 2002-12-31, Flow: 4.2247210553793
Link: DBUGSRC-SR_GNT, Time: 2003-01-31, Flow: 3.91780066566933
Link: DBUGSRC-SR_GNT, Time: 2003-02-28, Flow: 3.19845064830636
Link: DBUGSRC-SR_GNT, Time: 2003-03-31, Flow: 3.19400324898091
Link: DBUGSRC-SR_GNT, Time: 2003-04-30, Flow: 3.13398415867675
Link: DBUGSRC-SR_GNT, Time: 2003-09-30, Flow: 2.35417898173205
Link: DBUGSRC-SR_HID, Time: 2003-09-30, Flow: 4.07676144275653
Link: DBUGSRC-SR_NHG, Time: 2003-09-30, Flow: 17.3676775600299
Link: DBUGSRC-SR_RLL_CMB, Time: 2003-09-30, Flow: 31.914
Link: DBUGSRC-SR_SCC, Time: 2002-09-30, Flow: 0.00868323769867457
Link: DBUGSRC-SR_SCC, Time: 2003-09-30, Flow: 10.0064066780757
Link: DBUGSRC-SR_SHA, Time: 2003-09-30, Flow: 1594.3605
Link: DBUGSRC-SR_TRM, Time: 2003-09-30, Flow: 9.68956370124711
Link: DBUGSRC-SR_WHI, Time: 2002-11-30, Flow: 10.0975196274785
Link: DBUGSRC-SR_WHI, Time: 2003-08-31, Flow: 3.78756196477622
Link: DBUGSRC-SR_WHI, Time: 2003-09-30, Flow: 232.4385
Link: DBUGSRC-U102, Time: 2003-08-31, Flow: 22.9
Link: DBUGSRC-U102, Time: 2003-09-30, Flow: 15.87
Link: DBUGSRC-U103, Time: 2003-08-31, Flow: 5.81
Link: DBUGSRC-U103, Time: 2003-09-30, Flow: 3.71
Link: DBUGSRC-U104, Time: 2003-08-31, Flow: 1.64634342313956
Link: DBUGSRC-U104, Time: 2003-09-30, Flow: 3.17
Link: DBUGSRC-U202, Time: 2003-08-31, Flow: 9.58
Link: DBUGSRC-U202, Time: 2003-09-30, Flow: 7.27
Finding CPLEX Solution File
@jdherman
Our super fast CPLEX solver solved 82-year problem in 33 minutes but the job has not ended yet because the last task in job.slurm.sh is to move the solution results.json file to a defined directory. However, CPLEX solver did not write results into results.json file. The log file says the solution's stored in this directory /tmp/tmpjv33ppbx.cplex.sol but I couldn't find it there. Any suggestion?
...
Iteration: 1112824 Dual objective = -20793742582.627415
Elapsed time = 1942.64 sec. (412086.80 ticks, 1113177 iterations)
Iteration: 1113177 Dual objective = -20793734881.635632
Iteration: 1113521 Dual objective = -20793726495.888832
Removing perturbation.
Barrier solved model.
Primal simplex - Optimal: Objective = -1.9923665139e+10
Solution time = 2032.28 sec. Iterations = 8 (0)
Deterministic time = 177381.39 ticks (87.28 ticks/sec)
CPLEX> Solution written to file '/tmp/tmpjv33ppbx.cplex.sol'.
CPLEX>
1-year run is not running
@jdherman @mfefer @josue-medellin
Previously, 82-year run was fine but small subset runs (1-year or small network) were not running. I was thinking that problem would be boundary condition difference between S09I05 and PyVIN. Now Justin updated data in HOBBES database with data from PyVIN (ucd-cws/calvin-network-tools@198f581). But, small subsets and 1-year model are still not running. We need to investigate more. Maybe run in debug mode and look at where we are getting debug flows. (82-year run is fine and running) Error code is below.
LP HAS NO PRIMAL FEASIBLE SOLUTION
glp_simplex: unable to recover undefined or non-optimal solution
If you need actual output for non-optimal solution, use --nopresol
Time used: 2.7 secs
Memory used: 59.9 Mb (62827240 bytes)
Writing basic solution to 'C:\Users\msdogan\AppData\Local\Temp\tmp4gzolcir.glpk.raw'...
120468 lines were written
[ 25.53] Processing results
Number of solutions: 0
Solver results file: results.json
[ 25.53] Applying Pyomo postprocessing actions
[ 25.53] Pyomo Finished
Warning in the log file- suspicious....
@nickrsan and @msdogan ,
I noticed a warning message the log file of CALVIN, which @nickrsan recently implemented in the calvin.py. This message (below) tells you that the model model did not reach an optimal solution but founds the solution without getting the required iteration. It created me some doubt about the result, and I did not see any significant difference while you added the facilities in the system as well.
What do you think? Is there a way to optimize iterations inside the program? Or shall we just ignore it.
2020-08-25 20:50:09,749 - calvin - INFO - HSD203.1997-12-31_C18.1997-12-31 UB raised by 0.00 (0.00%)
**2020-08-25 20:50:09,767 - calvin - INFO - Warning: Debug mode maximum iterations reached. Will still try to solve without debug mode.**
2020-08-25 20:50:09,990 - calvin - INFO - Creating Pyomo Model (debug=False)
If you think it can be solved, let's have a meeting and have discussed it.
Thank you...
calvin.log
HostID mismatch for GUROBI license
@jdherman gurobi does not let me run the model because of license issues, I think.
Here what *.err file says:
Here what *.out file says:
Copying files/folders to cluster and running job scripts
@jdherman you showed us last time we met but can you show one more time how to copy files/folders between our computer and the cluster? I want to copy pyvin folder under z:/pyvin to cluster, but the command I am using does not seem to work scp z/pyvin [email protected]:/home/msdogan
Also, could you please show us how to run job scripts? Thanks.
I am sure @whiteellie and @mfefer would benefit from this :)
Solution is infeasible
After solving recent issues (#36,#37) in calvin-network-tools, I run pyvin but the solution is infeasible.
Here is the solver message:
AD3+msdogan@CWS-Amargosa /z/pyvin_working
$ pyomo solve --solver=glpk --solver-suffix=dual /z/pyvin/pyvin.py data.dat --stream-solver --json --report-timing
[ 0.00] Setting up Pyomo environment
[ 0.00] Applying Pyomo preprocessing actions
[ 0.07] Creating model
0.01 seconds required to construct component=N; 2158 indicies total
0 seconds required to construct component=k; 16 indicies total
0 seconds required to construct component=A_domain_index_0; 74511424 indicies total
0 seconds required to construct component=A_domain; 74511424 indicies total
0.05 seconds required to construct component=A; 5828 indicies total
0 seconds required to construct component=source; 1 indicies total
0 seconds required to construct component=sink; 1 indicies total
0.02 seconds required to construct component=u; 5828 indicies total
0.02 seconds required to construct component=l; 5828 indicies total
0.02 seconds required to construct component=a; 5828 indicies total
0.02 seconds required to construct component=c; 5828 indicies total
0.01 seconds required to construct component=X; 5828 indicies total
0.06 seconds required to construct component=total; 1 indicies total
0.14 seconds required to construct component=limit_upper; 5828 indicies total
0.14 seconds required to construct component=limit_lower; 5828 indicies total
0.04 seconds required to construct component=_; 5828 indicies total
0.23 seconds required to construct component=flow; 2156 indicies total
Cloning detected! (clone counters: 334, 2536, 0)
0.75 seconds required to construct instance=unknown
0.00 seconds required for problem transformations
[ 2.96] Applying solver
GLPSOL: GLPK LP/MIP Solver, v4.57
Parameter(s) specified in the command line:
--write C:\Users\msdogan\AppData\Local\Temp\tmpq370bd4_.glpk.raw --wglp C:\Users\msdogan\AppData\Local\Temp\tmpgvys96h3.glpk.glp
--cpxlp C:\Users\msdogan\AppData\Local\Temp\tmp67dxa2zi.pyomo.lp
Reading problem data from 'C:\Users\msdogan\AppData\Local\Temp\tmp67dxa2zi.pyomo.lp'...
13813 rows, 5829 columns, 23307 non-zeros
73667 lines were read
Writing problem data to 'C:\Users\msdogan\AppData\Local\Temp\tmpgvys96h3.glpk.glp'...
63522 lines were written
GLPK Simplex Optimizer, v4.57
13813 rows, 5829 columns, 23307 non-zeros
Preprocessing...
PROBLEM HAS NO PRIMAL FEASIBLE SOLUTION
If you need actual output for non-optimal solution, use --nopresol
Time used: 0.0 secs
Memory used: 8.4 Mb (8810012 bytes)
Writing basic solution to 'C:\Users\msdogan\AppData\Local\Temp\tmpq370bd4_.glpk.raw'...
19644 lines were written
[ 4.37] Processing results
Number of solutions: 0
Solver results file: results.json
[ 4.39] Applying Pyomo postprocessing actions
[ 4.39] Pyomo Finished
Last time when we got PROBLEM HAS NO PRIMAL FEASIBLE SOLUTION message, for some links, lb was greater than ub. I checked that and no link seems to have a greater lb than ub. data.dat is attached.
data.zip
Updates to Postprocessor to Include Dual Values for Nodes
@jdherman I have updated post-processor in example SR_WHI (https://github.com/msdogan/pyvin/tree/master/examples/SR_WHI) to include dual values for nodes. Could you please take a look at when you have a chance to verify? I added a new csv file for nodes, which are from *.dat file.Similar to network links, network nodes are read from this nodes.csv file. If this works well, only thing missing for postprocessor is to include end-of-period storage values for reservoirs.
Update: I have also updated the postprocessor to include end-of-period storage values for reservoirs.
CBC says infeasible
(not urgent, we can work on this after you get back to Davis!)
For the full 82-year model, the CBC solver says that the problem is infeasible. I may have missed a few fixes to the *dat file ... the only things I changed were adding in the lower bounds where they were missing for the final storages.
Here is some of the output:
[ 0.00] Setting up Pyomo environment
[ 0.00] Applying Pyomo preprocessing actions
[ 0.04] Creating model
2.97 seconds required to construct component=N; 1048792 indicies total
0 seconds required to construct component=k; 16 indicies total
0 seconds required to construct component=A_domain_index_0; 17599434548224 indicies total
0 seconds required to construct component=A_domain; 17599434548224 indicies total
30.18 seconds required to construct component=A; 5178353 indicies total
0 seconds required to construct component=source; 1 indicies total
0 seconds required to construct component=sink; 1 indicies total
13.13 seconds required to construct component=u; 5178353 indicies total
12.94 seconds required to construct component=l; 5178353 indicies total
12.06 seconds required to construct component=a; 5178353 indicies total
12.61 seconds required to construct component=c; 5178353 indicies total
6.03 seconds required to construct component=X; 5178353 indicies total
31.46 seconds required to construct component=total; 1 indicies total
68.89 seconds required to construct component=limit_upper; 5178353 indicies total
68.14 seconds required to construct component=limit_lower; 5178353 indicies total
26.00 seconds required to construct component=_; 5178353 indicies total
105.33 seconds required to construct component=flow; 1048790 indicies total
Cloning detected! (clone counters: 472830, 2087759, 0)
389.75 seconds required to construct instance=unknown
0.00 seconds required for problem transformations
[ 1323.98] Applying solver
Welcome to the CBC MILP Solver
Version: 2.9
Build Date: Aug 19 2016
command line - /group/hermangrp/cbc/bin/cbc -threads 32 -printingOptions all -import /tmp/tmp30qbh3jp.pyomo.lp -import -stat=1 -solve -solu /tmp/tmp30qbh3jp.pyomo.soln (default strategy 1)
threads was changed from 0 to 32
Option for printingOptions changed from normal to all
Current default (if $ as parameter) for import is /tmp/tmp30qbh3jp.pyomo.lp
Presolve determined that the problem was infeasible with tolerance of 1e-08
Presolved model looks infeasible - will use unpresolved
Problem has 11405497 rows, 5178354 columns (1566046 with objective) and 20710495 elements
There are 1 singletons with no objective
Column breakdown:
1 of type 0.0->inf, 0 of type 0.0->up, 0 of type lo->inf,
0 of type lo->up, 5178353 of type free, 0 of type fixed,
0 of type -inf->0.0, 0 of type -inf->up, 0 of type 0.0->1.0
Row breakdown:
1048790 of type E 0.0, 1 of type E 1.0, 0 of type E -1.0,
0 of type E other, 4853486 of type G 0.0, 2828 of type G 1.0,
322039 of type G other, 207347 of type L 0.0, 2019 of type L 1.0,
4968987 of type L other, 0 of type Range 0.0->1.0, 0 of type Range other,
0 of type Free
Presolve determined that the problem was infeasible with tolerance of 1e-08
Analysis indicates model infeasible or unbounded
1 infeasibilities
Analysis indicates model infeasible or unbounded
Perturbing problem by 0.001 %% of 5461.6424 - largest nonzero change 0 (%% 0) - largest zero change 3.7819156e-05
0 Obj 0 Primal inf 25811481 (324868) Dual inf 5.0111883e+08 (1566046) w.o. free dual inf (0)
1000 Obj -1.5564458e+14 Primal inf 4.2166107e+12 (324933) Dual inf 5.0110932e+08 (1560163) w.o. free dual inf (0)
1944 Obj -5.0740185e+15 Primal inf 1.3746081e+14 (326987) Dual inf 5.010936e+08 (1558432) w.o. free dual inf (1)
1949 Obj -5.0800263e+15 Primal inf 1.3811876e+14 (327000) Dual inf 5.0109357e+08 (1558420) w.o. free dual inf (1)
1950 Obj -5.0860341e+15 Primal inf 1.3877667e+14 (327013) Dual inf 5.0109357e+08 (1558416) w.o. free dual inf (1)
[......................]
9874428 Obj 3.6553868e+12 Primal inf 4034454.1 (32) Dual inf 0.031600209 (2817) w.o. free dual inf (2433)
9875428 Obj 3.6553868e+12 Primal inf 4034454.1 (32) Dual inf 0.015992448 (2216) w.o. free dual inf (2199)
9876428 Obj 3.6553868e+12 Primal inf 4034454.1 (32) Dual inf 0.0039303817 (1596)
9876846 Obj 3.6553868e+12 Primal inf 4034454.1 (32) Dual inf 0.0010344485 (1177)
Primal infeasible - objective value 3.6553868e+12
PrimalInfeasible objective 3.655386811e+12 - 9876846 iterations time 301988.262
Result - Linear relaxation infeasible
Enumerated nodes: 0
Total iterations: 0
Time (CPU seconds): 302031.26
Time (Wallclock Seconds): 314617.10
Total time (CPU seconds): 302048.81 (Wallclock seconds): 314638.00
[316470.43] Processing results
Number of solutions: 0
Solver results file: results.json
[316470.43] Applying Pyomo postprocessing actions
[316470.43] Pyomo Finished
The results.json file doesn't have anything useful in it.
I think I might be missing something in the *dat file. Maybe some of those negative upper bounds or other weird stuff. Anyway we can come back to it, it's all set up to run.
Deciding precision of parameters in matrix export
This is a minor enhancement but @jrmerz asked in our last chat about what precision we want for each parameter (cost, amplitude, lower bound, upper bound) in matrix export. Some numbers are ridiculously long.
SR_SHA.2002-10-31 SR_SHA.2002-11-30 1 -2.973775927491251 0.9974443500372581 737.3889770000001 737.3889770000001
Checking bounds and costs with random samples
I randomly picked links for all possible bound and cost cases I can think of from the database. I am going to check those cases only to see if there is any problem since it is hard to check all of them one-by-one. If everything looks good but we still get some considerable flow differences, then we can focus on that problematic links.
Here are random links for each bound and cost cases:
Bound Examples
Lower bound only
Constant: D511-D55
Monthly Varying: C120-SR_OL
Time-series: D507-D509, D675-D676
Upper Bound only
Constant: C49-C51, C62-N10
Monthly Varying: C75-D845
Time-series: HU206-CVPM09S, HXI210-Ext_Stockton
Lower Bound & Upper Bound (not fixed)
Constant: SR_GNT-SR_GNT, SR_BER-SR_BER
Monthly Varying: SR_BUL-SR_BUL
Time-series: SR_SHA-SR_SHA, SR_CLE-SR_CLE, SR_FOL-SR_FOL
Constrained (fixed)
Constant: Do not exist
Monthly Varying: Do not exist
Time-series: D541-Required_Delta_Outflow, C20-D55
Cost Examples
Constant
Unit Cost: HGP21-A409A
Piecewise Curve
Negative Slope: C23-C25, HU203_CVPM06G
Positive Slope: PMP_OSO-D884
Flow differences between S09I05 (CALVIN) and pyvin
Now that pyvin is running without debug mode, we can compare flow results. Tables below show links with monthly average (TAF/m) flow differences (S09I05-pyvin) greater than 1 TAF/m and less than -1 TAF/m.
Here is the cumulative distribution of errors:
Link_Flow_Comparison.pdf
Avg. flow difference > 1 TAF/m (Sorted large to small)
Link S09I05_avg pyvin_avg
D523-D525 781.6787633 591.67432
D522-D523 819.9958928 630.3933275
D515-D522 528.008249 393.2333143
D513-D515 486.5349756 352.6453817
D616-C42 256.0514138 145.6792348
C42-D521 268.3969079 159.2392462
C309-D59 569.3837406 465.9362483
D525-D550 584.5392239 481.0940955
D550-C309 579.0554182 475.6215628
D676-D616 258.6287234 163.474426
D612-D676 193.596771 99.623167
D689-D612 147.0590394 53.08543553
D525-D528 197.1395393 110.5802245
D528-D509 196.8778724 110.3185578
D712-D722 211.4920777 127.3056605
D699-D683 113.8574162 31.53270243
D683-D689 165.615928 85.08676601
D511-D513 519.2325421 440.7415609
D695-D697 113.2036319 36.68345202
D697-D699 121.8469573 45.32677738
D710-D712 196.2670988 126.6142803
D694-D695 87.69708414 22.0825765
D619-D691 86.11558013 20.65716946
D691-D694 88.05623051 22.59781987
D731-D732 64.42653107 1.936525627
D732-D619 65.02854844 2.538543284
D608-D731 67.77790935 8.511235921
D722-D723 108.0343814 48.84486741
D723-D608 108.0343814 48.84486741
D521-D522 291.9876479 237.1600132
D801-D803 327.6546412 274.0828638
D800-D801 327.7888817 274.2201671
D803-D804 327.5845773 274.0779858
D804-D814 326.9431606 273.4365692
D59-PMP_BANKS 325.5374547 273.5503749
PMP_BANKS-D800 325.5374547 273.5503749
D59-PMP_TRACY 243.8462845 192.3858734
PMP_TRACY-D701 243.8462845 192.3858734
D703-D710 240.0539089 190.1751316
D700-D703 241.5948572 191.7160812
D701-D700 241.5948572 191.7160812
C30-HSU303C30 103.4576956 78.46079304
D722-C30 103.4576956 78.46079304
D37-C32 507.4699125 484.4740644
C31-D37 482.0339231 460.053713
C83-C31 151.1321929 129.2539676
C80-HSU202C80 66.60247714 44.86999146
C32-C80 66.57156612 44.83920776
HSU303C30-A303 74.37338561 52.98824876
HSU202C80-A202 58.13530012 39.01028786
D814-PMP_GIANEL 67.82482563 48.75842331
PMP_GIANEL-SR_SNL 67.82482563 48.75842331
D816-D742 326.4629903 308.0852808
PWP_GIANEL-D816 38.78282508 20.53766385
SR_SNL-PWP_GIANEL 38.78282508 20.53766385
D814-D816 302.9051435 288.2389971
D816-PWP_ONEILL 15.22497819 0.691380196
PWP_ONEILL-D712 15.22497819 0.691380196
D664-HSU302D664 19.52586112 5.451949758
C79-WTP301 13.64860477 0
D647-D649 35.80701506 23.21745529
D646-D647 38.30255871 25.71299902
D645-D646 27.54941947 14.9598598
D743-D743A 318.846535 306.4885273
D742-D743 319.9544192 307.596413
D745-D747 314.6986092 302.5986296
D747-D749 314.6986092 302.5986296
D743A-PMP_DAMIGO 318.5885043 306.4885273
PMP_DAMIGO-D744A 318.5885043 306.4885273
C92-D745 317.5306449 305.4306699
D744A-D744 317.5306449 305.4306699
D744-C92 317.5306449 305.4306699
D662-D663 56.23863223 44.22299142
D663-D664 73.11645779 61.10081729
D749-D750 257.1865286 245.4923298
D751-C75 253.3531936 241.6589964
D750-D751 254.1948587 242.5006631
HSU302D664-A302 16.01120609 4.470598762
D649-D695 24.15888663 13.25320583
SR_WHI-D73 11.21628471 1.136087984
D606-HSU306D606 15.26307935 6.415667275
D16-HSU302D16 22.74824393 14.12181611
HSU306D606-A306 14.34729473 6.030727223
HSU302D16-A302 18.65355975 11.57988911
C75-D845 247.4650748 240.6704342
D845-D850 229.7598306 222.9651903
D605-D606 28.23995003 21.53709347
SR_MIL-D605 15.33395013 8.631093809
C85-HSU303C85 6.766601838 0.48886783
SR_GNT-SR_ML 6.275451584 0.214308148
D742-C85 6.508571298 0.48886783
HSU303C85-A303 6.292939901 0.454647085
D850-D851 149.3076568 143.9506185
D851-D853 198.9501456 193.5931083
C95-D752 197.9584786 192.6014416
D853-C95 197.9584786 192.6014416
C25-WTP201 5.939398709 0.759735772
C75-N9 5.888119312 0.988562258
N9-HSU404AN9 4.81924606 0.008439114
D731-HSU306D731 7.07485776 2.373963042
SR_TUL-D16 80.6831387 76.06183871
D670-SR_TUL 78.16223905 73.61167338
HSU404AN9-A404A 4.481898826 0.007848376
HSU306D731-A306 6.225874813 2.089087466
D541-SURP_DELTA 766.33473 762.6269795
SURP_DELTA-SINK 766.33473 762.6269795
D509-D541 1182.482774 1178.775026
D752-C74 42.7752344 39.09655086
WTP204-U204 17.11132138 13.57653363
C53-HSU401C53 32.58497045 29.1254492
D860-D861 195.7541622 192.3439125
D861-PMP_WHEELR 195.4541609 192.0439123
D862A-D863 195.4541609 192.0439123
D863-PMPCHRIS 195.4541609 192.0439123
PMP_WHEELR-D862A 195.4541609 192.0439123
PMPCHRIS-D862B 195.4541609 192.0439123
D855-D859 173.4938085 170.0835602
D859-PMP_BUENA 190.0334425 186.6231944
PMP_BUENA-D860 190.0334425 186.6231944
D888-C161 55.3209544 52.02005447
SR_CAS-D888 63.29152746 60.00899147
D865-PMP_OSO 66.87248151 63.64238363
D884-D885 66.87248151 63.64238363
PMP_OSO-D884 66.87248151 63.64238363
D885-PWP_WARNE 66.71414748 63.4840503
PWP_WARNE-SR_PYM 66.71414748 63.4840503
C103-D865 171.6621655 168.4321003
D862B-PMP_EDMONS 171.6621655 168.4321003
PMP_EDMONS-C103 171.6621655 168.4321003
SR_PYM-D886 66.34205096 63.12430315
D886-PWP_CASTAI 66.08371728 62.8659699
PWP_CASTAI-D887 66.08371728 62.8659699
D887-SR_CAS 65.8837171 62.6659699
C98-D855 18.68333103 15.49752406
D752-D855 157.7585036 154.5860362
C8-WTP204 11.1863834 8.20115533
D692-D619 21.0870316 18.11862618
C47-D692 4.987438003 2.019032683
D855-C98 2.948026557 0
HSU401C53-A401 27.6972247 24.75663252
N2-N1 11.45064988 8.525521551
N1-N2 2.909262282 0.000626708
HGP08-U210 2.758320234 0.019366643
C153-C156 14.60094903 11.95462802
C156-WTP511 14.60094903 11.95462802
D31-D61 648.2002079 645.9033779
D61-C301 574.1111111 571.8142818
C48-C47 3.856772497 1.571581422
D649-HSU305D649 8.044713058 5.789660035
C313-C313B 29.1850803 26.96159061
C15-C313 24.81922715 22.59573775
C313B-C314 7.110882382 4.92907396
D507-HSU206D507 22.63862024 20.47871082
HSU206D507-A206 21.05391689 19.04520121
D42-D43 488.1520738 486.1907365
C12-D76B 24.51768342 22.58215773
C67-HSU204C67 24.793893 22.87836936
HSU303C30-HSR302 21.84227166 19.98028933
HSU305D649-A305 6.596664787 4.747521187
C69-C13 61.50130946 59.77415382
HSU204C67-A204 21.81862632 20.13296493
D77-D75 678.8182652 677.1783105
D75-C1 749.4542419 747.8142873
D66A-D66 725.4492731 723.8245471
N13-D66A 725.7947401 724.1700142
D76B-N13 721.9248721 720.3001471
D66-N14 752.7683776 751.1436529
N14-D30 786.7520156 785.1272912
D30-D31 654.1340779 652.5093556
D701-D800 2.251427187 0.669792188
C7-C8 1147.319722 1145.79009
C67-C7 1135.980717 1134.451088
C1-C4 748.2179941 746.7215023
C74-D752 2.575259538 1.081145449
SR_SKN-C153 60.70879245 59.22909242
D850-N12 80.45217429 79.0145718
C315-WTP304 6.464371612 5.040811262
D714-HARSC 16.24205625 14.82074005
C4-C69 750.8352162 749.4188606
C41-WTP210 5.325874823 3.971982198
C38-D98 82.53241114 81.22011558
D43-C67 1168.118259 1166.85168
C32-D42 478.1280184 476.8645313
D609-C48 1.51324129 0.259021716
C53-HSU403C53 45.35918535 44.10855896
N12-HSU408BN12 53.51405149 52.27116873
C74-HSU407C63 20.67207045 19.43108626
D672-D675 55.19289209 54.01219891
D675-D676 65.03195204 63.85125903
C104-C105 1.971935629 0.810711678
HSU408BN12-A408B 47.70748556 46.55160549
HSU407C63-A407 19.22502537 18.07091037
SR_OL-C120 15.79930722 14.69794415
C120-SR_OL 19.13180715 18.03044415
HSU403C53-A403 39.91608341 38.81553167
D98-HSU207D98 15.62842251 14.5422657
D74-C5 658.19441 657.109829
C87-D77 715.2067524 714.1225499
C5-C87 711.4139543 710.3297522
C37-C38 30.11519976 29.06270367
C53-WTP401 13.50921371 12.47653872
D624-C48 2.343531208 1.312559705
C308-D37 18.5807739 17.56513679
C33-C308 18.25191204 17.23627501
Avg. flow difference < -1 TAF/m (sorted small to large)
D624-HSU306D624 4.642579857 5.659813904
C49-WTP401 3.742574541 4.762225158
C8-C104 1349.292384 1350.328908
C37-HSU207C37 0.001771737 1.054267827
C65-HSU408BC65 6.146095074 7.203447451
D521-HSU206D521 9.106826652 10.17541229
HSU403C76-A403 0.005143002 1.105755993
D64-WTP204 3.362075038 4.521483896
C153-C154 46.10784397 47.2744644
N6-SR_SKN 94.48620779 95.65609002
C138-N6 64.46836709 65.63978749
C76-HSU403C76 0.00553011 1.18898493
HSU204C33-A204 10.15460734 11.36513442
C41-C42 8.573993216 9.788511462
HSU407C64-A407 22.41451607 23.72522962
C33-HSU204C33 11.53932632 12.91492555
C64-HSU407C64 24.10163052 25.51099939
C315-HARSC 3.757943747 5.179259948
HSU409AC68-A409A 9.913518882 11.41886889
C689-HSU409AC68 10.65969747 12.27835355
HSU305D662-A305 46.73235159 48.45542764
D664-D683 51.75851121 53.55406358
HSU303D731-HSR302 1.459898064 3.280752185
C64-C689 11.15244968 12.99103399
C12-C13 0.313141293 2.283852031
D662-HSU305D662 56.99067341 59.09198545
C67-C314 7.343647698 9.522223274
C104-D44 1347.320447 1349.518196
D44-D503 1371.395445 1373.593195
D503-D511 1371.395445 1373.593195
C15-C301 16.70667455 19.20692863
HSU401D606-A401 0.844646034 3.375156258
N8-WTP511 32.75375945 35.4000807
SR_SKN-N8 33.29111328 35.9374346
WTP210-U210 7.053346432 9.792300024
HXCMWD-ERES_CMWD 121.6311526 124.5512794
C161-WTP504 154.7601283 157.6802551
WTP504-U504 154.7601283 157.6802551
HXI504-HXCMWD 105.4017987 108.3219258
U504-HXI504 114.8451316 117.7652591
ERES_CMWD-SINK 4.444016902 7.364144817
D606-HSU401D606 1.030056143 4.116044253
C62-C64 35.45376283 38.70171634
HGP08-U204 10.01393551 13.40648498
C688-C62 36.43874119 40.0783141
C76-C688 121.4326631 125.2110164
D653B-D672 66.34002812 70.34515572
D16-D653A 57.93489497 61.9400226
D653A-D653B 57.93489497 61.9400226
C43-WTP210 1.727471609 5.820317826
HSU302D672-A302 9.140651507 13.39302447
HSU404AC52-A404A 14.81217809 19.42638299
D670-C43 7.39610892 12.24476806
HSU303D731-A303 4.975705492 9.890444649
C49-C76 123.1078828 128.0696912
C25-C31 332.5633859 337.6514731
D672-HSU302D672 11.14713595 16.33295681
C31-WTP201 1.66165803 6.851727642
D689-C10 1.542753609 7.013434163
D699-C10 4.303111494 9.975949731
C53-C54 26.28026123 32.04801696
C54-C52 16.6797006 22.44745637
C52-HSU404AC52 18.51522255 24.28297837
SR_GNT-SR_CRW 3.561948079 9.874774944
HSU302D689-A302 13.95159058 20.4900748
C116-C120 39.47307034 46.05707489
PWP_OWENS1-C116 15.13142157 21.71542621
SR_CRW-PWP_OWENS1 15.13142157 21.71542621
C120-SR_LA 30.65081318 37.23481789
SR_MIL-C49 127.6562018 134.3534422
C122-C161 30.44090823 37.23444592
PWP_OWENS2-C122 30.44090823 37.23444592
SR_LA-PWP_OWENS2 30.44090823 37.23444592
HSU302D662-A302 5.188653712 13.01207319
D731-HSU303D731 10.40432248 18.32854944
D689-HSU302D689 17.01413488 24.98789632
D5-D73 526.2667643 535.5880108
D662-HSU302D662 6.327626503 15.86838208
SR_WHI-D5 66.56458406 76.46066163
HSU306D645-A306 37.79101525 47.98350168
D645-HSU306D645 46.61482888 59.38876368
SR_SFAGG-WTP301 4.618895304 18.2675
C79-SR_SFAGG 12.9737563 27.13916667
D616-C10 4.192186262 19.41006823
HSU202C83-A202 2.537937501 21.74134421
D710-PMP_ONEILL 43.78680885 63.5608513
PMP_ONEILL-D814 43.78680885 63.5608513
C83-HSU202C83 2.884019835 24.7060731
HSU303C10-A303 8.532343801 30.93953517
C10-HSU303C10 10.03805131 36.39945212
D513-D521 32.6975663 88.09617922
D511-D55 852.1629026 932.8516343
D55-D507 995.1401354 1075.831792
D507-D509 972.5015154 1055.353081
S09I05 and pyvin link flows are attached
S09I05_pyvin_flow.zip
Check Reservoir Evaporation and Create a Postprocessor
Currently, reservoir evaporation is represented in amplitude in the network matrix.
There are two tasks:
- Verify that reservoir evaporation is correctly represented.
- Create a postprocessor that generates evaporation time-series. Based on the current configuration, it would be (1-amplitude)*storage, I think.
Help wanted for network checking preprocessor
@jdherman I have created a preprocessor that goes through the network and checks how many incoming and outgoing links for a given node. And it prints if there is none. This is something important for the mass balance flow constraint. Pyomo also does this but one node at a time. It stops when it sees there is no incoming or outgoing link. Except for SOURCE and SINK every node has ta have at least one inflow and outflow. With this preprocessor, we can check the whole network before running.
import csv
import numpy as np
# load network links
with open('networklinks.csv', 'rU') as l:
reader = csv.reader(l)
links = list(reader)
# load network nodes
with open('networknodes', 'rU') as n:
reader = csv.reader(n)
nodes = list(reader)
nrows_l = len(links) # # of rows in links
nrows_n = len(nodes) # # of rows in nodes
netcheck = np.zeros((nrows_n,2)) # incoming / outgoing links to / from a node
# check the numbers of incoming and outgoing links for each node
for i in range(0,nrows_n):
count1, count2 = 0, 0
for j in range(1,nrows_l): # start from the second row. First row is a header
if nodes[i][0] == links[j][1]: # check for incoming links
count1 += 1
if nodes[i][0] == links[j][0]: # check for outgoing links
count2 += 1
netcheck[i,0] = count1
netcheck[i,1] = count2
if count1 == 0:
print('no incoming link for ' + str(nodes[i]))
if count2 == 0:
print('no outgoing link for ' + str(nodes[i]))
# Save results to a .csv file
with open('networkcheck.csv', 'w') as csvfile:
writer = csv.writer(csvfile, delimiter=',')
writer.writerows(zip(nodes,netcheck))
The code below does the job, however, it saves in a weird way. It creates extra rows. First column below is node, second column is number of incoming links, and the third column is number of outgoing links.
Do you know an easy solution for this? (input and output files zipped and attached)
['D5.2000-10-31'] [ 8. 3.]
['D5.2000-11-30'] [ 8. 3.]
['D5.2000-12-31'] [ 8. 3.]
['D5.2001-01-31'] [ 8. 3.]
['D5.2001-02-28'] [ 8. 3.]
['D5.2001-03-31'] [ 8. 3.]
['D5.2001-04-30'] [ 8. 3.]
['D5.2001-05-31'] [ 8. 3.]
['D5.2001-06-30'] [ 8. 3.]
['D5.2001-07-31'] [ 8. 3.]
['D5.2001-08-31'] [ 8. 3.]
['D5.2001-09-30'] [ 8. 3.]
['FINAL'] [ 1. 1.]
['INBOUND.2000-10-31'] [ 1. 1.]
['INBOUND.2000-11-30'] [ 1. 1.]
['INBOUND.2000-12-31'] [ 1. 1.]
['INBOUND.2001-01-31'] [ 1. 1.]
['INBOUND.2001-02-28'] [ 1. 1.]
['INBOUND.2001-03-31'] [ 1. 1.]
['INBOUND.2001-04-30'] [ 1. 1.]
['INBOUND.2001-05-31'] [ 1. 1.]
['INBOUND.2001-06-30'] [ 1. 1.]
['INBOUND.2001-07-31'] [ 1. 1.]
['INBOUND.2001-08-31'] [ 1. 1.]
['INBOUND.2001-09-30'] [ 1. 1.]
['INFLOW.2000-10-31'] [ 1. 1.]
['INFLOW.2000-11-30'] [ 1. 1.]
['INFLOW.2000-12-31'] [ 1. 1.]
['INFLOW.2001-01-31'] [ 1. 1.]
['INFLOW.2001-02-28'] [ 1. 1.]
['INFLOW.2001-03-31'] [ 1. 1.]
['INFLOW.2001-04-30'] [ 1. 1.]
['INFLOW.2001-05-31'] [ 1. 1.]
['INFLOW.2001-06-30'] [ 1. 1.]
['INFLOW.2001-07-31'] [ 1. 1.]
['INFLOW.2001-08-31'] [ 1. 1.]
['INFLOW.2001-09-30'] [ 1. 1.]
['INITIAL'] [ 1. 1.]
['OUTBOUND.2000-10-31'] [ 3. 1.]
['OUTBOUND.2000-11-30'] [ 3. 1.]
['OUTBOUND.2000-12-31'] [ 3. 1.]
['OUTBOUND.2001-01-31'] [ 3. 1.]
['OUTBOUND.2001-02-28'] [ 3. 1.]
['OUTBOUND.2001-03-31'] [ 3. 1.]
['OUTBOUND.2001-04-30'] [ 3. 1.]
['OUTBOUND.2001-05-31'] [ 3. 1.]
['OUTBOUND.2001-06-30'] [ 3. 1.]
['OUTBOUND.2001-07-31'] [ 3. 1.]
['OUTBOUND.2001-08-31'] [ 3. 1.]
['OUTBOUND.2001-09-30'] [ 3. 1.]
['SINK'] [ 13. 0.]
['SOURCE'] [ 0. 25.]
['SR_SHA.2000-10-31'] [ 2. 11.]
['SR_SHA.2000-11-30'] [ 5. 11.]
['SR_SHA.2000-12-31'] [ 5. 11.]
['SR_SHA.2001-01-31'] [ 5. 11.]
['SR_SHA.2001-02-28'] [ 5. 11.]
['SR_SHA.2001-03-31'] [ 5. 11.]
['SR_SHA.2001-04-30'] [ 5. 11.]
['SR_SHA.2001-05-31'] [ 5. 11.]
['SR_SHA.2001-06-30'] [ 5. 11.]
['SR_SHA.2001-07-31'] [ 5. 11.]
['SR_SHA.2001-08-31'] [ 5. 11.]
['SR_SHA.2001-09-30'] [ 5. 8.]
Debug flow finder
@jdherman, @whiteellie I am trying to write a code that imports output flows (flow.csv), and goes through each column (flow time-series), and stores time-series if there is a debug flow (if flow > 0). There are two debug flows: DBUGSRC and DBUGSINK. DBUGSRC is on the left hand side of link name (if splitted with '-') and DBUGSNK is on the right hand side. The code attached finds debug links (if a link name includes DBUGSRC or DBUGSNK), however I can't iterate through columns to see if flow is > 0. Any help would be greatly appreciated. The code (debug_flow_finder.py) and flow (flow.csv) files are zipped.
Getting rid of debug flows and running w/o debug mode
My latest one-year run (Oct 2002 - Sep 2003) had some debug flows. But results are promising. It occurred in a few places and the problem seems to end-of-period storage problem for reservoirs. One debug flow on SR_BUL is quite big, others are small. I am just creating this issue. I need to run 82-year model and make calibration on that run not 1 year if needed. Since CALVIN and Hobbes test could be run w/o debug mode, we need to take a closer look what is causing it before modifying fixed time-series and minimum in-stream flow requirements.
1-year run debug flows:
Link: DBUGSRC-SR_BUL, Time: 2003-09-30, Flow: 165.997728
Link: DBUGSRC-SR_CLE, Time: 2003-09-30, Flow: 1.42277304048493
Link: DBUGSRC-SR_CR1, Time: 2003-09-30, Flow: 0.00999999977648258
Link: DBUGSRC-SR_CR2, Time: 2003-09-30, Flow: 0.00999999977648258
Link: DBUGSRC-SR_HTH, Time: 2003-09-30, Flow: 0.869099103516036
Mass balance issue for some nodes
@jdherman I was able to identify previously that desal node had a balance issue ucd-cws/calvin-network-tools#37 (comment)
However, after running the network checking preprocessor, looks like we have more nodes with the same problem. These nodes C146, C148, C304, C86, D43a, Desal have mass balance problem. I don't know the reason at the moment. Justin has some explanation in the link above, but still unsolved. I run this for only the 1-year period, assuming other periods have the same problem.
no incoming link for ['C146.2001-10-31']
no incoming link for ['C146.2001-11-30']
no incoming link for ['C146.2001-12-31']
no incoming link for ['C146.2002-01-31']
no incoming link for ['C146.2002-02-28']
no incoming link for ['C146.2002-03-31']
no incoming link for ['C146.2002-04-30']
no incoming link for ['C146.2002-05-31']
no incoming link for ['C146.2002-06-30']
no incoming link for ['C146.2002-07-31']
no incoming link for ['C146.2002-08-31']
no incoming link for ['C146.2002-09-30']
no incoming link for ['C148.2001-10-31']
no incoming link for ['C148.2001-11-30']
no incoming link for ['C148.2001-12-31']
no incoming link for ['C148.2002-01-31']
no incoming link for ['C148.2002-02-28']
no incoming link for ['C148.2002-03-31']
no incoming link for ['C148.2002-04-30']
no incoming link for ['C148.2002-05-31']
no incoming link for ['C148.2002-06-30']
no incoming link for ['C148.2002-07-31']
no incoming link for ['C148.2002-08-31']
no incoming link for ['C148.2002-09-30']
no incoming link for ['C304.2001-10-31']
no incoming link for ['C304.2001-11-30']
no incoming link for ['C304.2001-12-31']
no incoming link for ['C304.2002-01-31']
no incoming link for ['C304.2002-02-28']
no incoming link for ['C304.2002-03-31']
no incoming link for ['C304.2002-04-30']
no incoming link for ['C304.2002-05-31']
no incoming link for ['C304.2002-06-30']
no incoming link for ['C304.2002-07-31']
no incoming link for ['C304.2002-08-31']
no incoming link for ['C304.2002-09-30']
no incoming link for ['C86.2001-10-31']
no incoming link for ['C86.2001-11-30']
no incoming link for ['C86.2001-12-31']
no incoming link for ['C86.2002-01-31']
no incoming link for ['C86.2002-02-28']
no incoming link for ['C86.2002-03-31']
no incoming link for ['C86.2002-04-30']
no incoming link for ['C86.2002-05-31']
no incoming link for ['C86.2002-06-30']
no incoming link for ['C86.2002-07-31']
no incoming link for ['C86.2002-08-31']
no incoming link for ['C86.2002-09-30']
no incoming link for ['D43a.2001-10-31']
no incoming link for ['D43a.2001-11-30']
no incoming link for ['D43a.2001-12-31']
no incoming link for ['D43a.2002-01-31']
no incoming link for ['D43a.2002-02-28']
no incoming link for ['D43a.2002-03-31']
no incoming link for ['D43a.2002-04-30']
no incoming link for ['D43a.2002-05-31']
no incoming link for ['D43a.2002-06-30']
no incoming link for ['D43a.2002-07-31']
no incoming link for ['D43a.2002-08-31']
no incoming link for ['D43a.2002-09-30']
no incoming link for ['Desal.2001-10-31']
no incoming link for ['Desal.2001-11-30']
no incoming link for ['Desal.2001-12-31']
no incoming link for ['Desal.2002-01-31']
no incoming link for ['Desal.2002-02-28']
no incoming link for ['Desal.2002-03-31']
no incoming link for ['Desal.2002-04-30']
no incoming link for ['Desal.2002-05-31']
no incoming link for ['Desal.2002-06-30']
no incoming link for ['Desal.2002-07-31']
no incoming link for ['Desal.2002-08-31']
no incoming link for ['Desal.2002-09-30']
Negative lower & upper bound constraints
There are negative fixed (constrained) flows, where lower bound = upper bound < 0, that violates upper bound constraint. These negative fixed flows are groundwater inflows where pumping is greater than recharge in a given month. Current model uses variables within non-negative real values. Upper bound constraint basically states that model variable should be less than upper bound. When upper bound is negative, this constraint is violated.
Benchmark Study Runs
@jdherman we would like to perform multiple (so many) runs to compare solver and total run times for each solver we use. If we create 5 different data sets (different # of decision variables) for each solver (4 total) and 10 run for each to create error bars (originally 30), it makes 200 runs. Only changing thing in command in batch file would be --solver and data.dat paths.
pyomo solve --solver=cbc --solver-suffix=dual --solver-options="threads=32" pyvin.py $DIR/data.dat --json --report-timing --stream-solver
Is there a way to run job script recursively? Maybe a for loop in batch file :)
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.