Comments (4)
Hey @HauHe! I think your original snakefile only included the preprocessing and solving? I do agree though, parallelizing the lp creation would be good. The only issue is that since we try to handle LPs that fail to build, it can be difficult because snakemake will flag them as missing input files for the solve step.
Maybe using the multiprocessing library would be a better option. But Ive never used it so I dont really know
from osemosys_step.
Well or we do it how I did it for solving in the old scripts and put the lo-generation into a python script that handles it and provides snakemake with an output file.
from osemosys_step.
@trevorb1, I'm tempted to just do this as I had it in the original scrips, i.e., have a python script handling the lp generation and call it from snakemake
from osemosys_step.
@HauHe, that could be a good solution!
A couple of thoughts if this is implement though:
- Right now, if the LP generation fails, we remove the model run from the current step and all subsequent steps. I think this may be a little tricky to implement in snakemake? (Although, you have done it before, so I trust your experience!)
- As you mentioned we can create an LP file, regardless of if it completed successfully or not, and use the
--keep-going
flag to deal with failed solves. However, we will also need to re-implement the logic to track which LPs failed and remove all downstream steps between snakemake and passing residual capacity. - Alternatively, we can use input functions probably to check if the LP is generated correctly or not, and define logic in Snakemake on what to do in each case. Im not sure what would be easier off the top of my head.
- Also, it seems like we are going towards a point of just snakemake'ing everything? I know that that the LP generation and solving are the big components, but maybe this update deserves further discussion around how to just snakemake all logic for each step?
from osemosys_step.
Related Issues (20)
- ERROR:utils:Step must be of type int or List[int] HOT 12
- Add check if snakemake is installed HOT 3
- Passing of NewCapacity doesn't work as expected HOT 1
- Large capacity additions for no reason HOT 6
- Error when no folder called `solves` in `logs` directory HOT 1
- Residual Capacity + NewCapacity HOT 6
- Package OSeMOSYS_step
- User determined foresight HOT 1
- Last year skipped when foresight horizon reaches last year HOT 1
- No final results generated if no step has options HOT 1
- check of feasibility for gurobi missing HOT 1
- Merge of ResidualCapacity and NewCapacity HOT 2
- Rounding of solver tolerated values leaves too many values HOT 1
- Historic capacity is dropped in merge of residual and New capacity HOT 2
- Option to run with perfect foresight and scenarios HOT 1
- Didn't consider the subdirectories under step directories in case scenarios exist HOT 1
- ilp logs are written into step directory, which then deleted HOT 1
- Function to create folder structure locally HOT 1
- [Packaging] Error: Snakefile not found
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from osemosys_step.