Comments (6)
Hi @pilgrim2go,
please have a look at the example code in the README.md
file. You should be able to import your troposphere templates and then setup a hierarchy by declaratively adding your stacks to the file and chaining the stacks using the outputs. There is a dependency between two stacks, if one stack uses an output of another stack as input parameter. If there is a dependency aloisius executes the operations on those stacks in a sequential manner. If there is no dependency between two stacks, the parallelisation should happen automatically :-)
Best
Andreas
from aloisius.
@adonig:
As far as I know, if you have 5 Stack and adjust the following line
in stack.py
_executor = ThreadPoolExecutor(max_workers=3)
You'll see unexpected result ( eg some threads lost).
Due to myy lack of concurrency experience, I can't show the reason why yet. But each ThreadPool per Stack looks weird to me at first.
Besides, I can't find any line of code can specify Stack dependency. Such feature is really needed.
from aloisius.
Oh, no no no. It is automatically set to the number of CPUs your computer has:
_executor = ThreadPoolExecutor(max_workers=multiprocessing.cpu_count())
That makes sense because your computer can run that many tasks in parallel.
Stack dependencies are currently specified implicitly over stack outputs and stack input parameters. Here's an example:
stack1 = Stack(...)
stack2 = Stack(..., Parameters={'input1': stack1.outputs['foo']})
stack3 = Stack(...)
In that scenario, the operations on stack1
and stack3
are executed in parallel, while the operation on stack2
has to wait until the operation on stack1
is finished (because otherwise the outputs of stack1
would not exist).
from aloisius.
If you don't use the outputs you could force blocking using aloisius.stacks.wait()
before executing the next one
from aloisius.
If there is use for it, we could think about adding a constructor parameter DependsOn
so it would be possible to write something like:
stack1 = Stack(...)
stack2 = Stack(..., DependsOn=[stack1])
from aloisius.
@adonig : Yes, I prefer DependsOn solution. That makes Stack definition more clearer and we can build complex dependency graph.
Also I 1thought multiprocessing.cpu_count() is for process ( vs thread). Since my cpu_count is 4 that why explicitly set number=4 is the same.
Btw, I ended up using your solution to do something simple ( my case).
sc = StackCollection()
s1 = Stack('vpc')
s2 = Stack('cluster', depends=s1).
sc.add(s1)
sc.add(s2)
and then I can do something like
sc.runall()
sc.delete()
or
sc.runall(queryset=QuerySet(__name__'vpc')
Code is copied from Alosius ( worked for me but not a generic solution)
from aloisius.
Related Issues (3)
- New release? HOT 2
- StackCollection documentation HOT 2
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from aloisius.