Comments (1)
@amexboy This is what I've come up with to solve the distributed state problem when pausing/unpausing and even stopping(#56) pipelines without letting the current step finish and reacting (almost) instantly:
- The
runs
table contains the info we are interested in, namely the pipeline, current container id and status. - When a node recieves the pause/unpause/stop request, it checks if its running the container from the
last_pid
from the runs table, if yes pauses/unpauses/kills it. if its not running it, sets the status column to paused/running based on the request. - The other nodes are then notified of this change and the node actually running it does the job.
- To achieve this async notification, I'm using this and the impl is as follows:
- Create a PLSQL function that responds with the current row as json that has been affected with an update on the runs table:
CREATE FUNCTION "notify_on_status_update"() RETURNS TRIGGER LANGUAGE "plpgsql" AS $$ BEGIN PERFORM pg_notify('pipeline_status', row_to_json(NEW)::TEXT); RETURN NEW; END; $$;
- Create a trigger which triggers this when an update happens:
CREATE TRIGGER "pipeline_status_change" AFTER UPDATE ON "runs" FOR EACH ROW EXECUTE PROCEDURE "notify_on_status_update"();
- Create a
LISTEN
connection from each of the nodes to the DB as described here - In the callback, do the same process of checking if the node has the container and handle it accordingly.
This way the distributed state is maintained, messaging is async and the DB is still the single source of state.
I am planning to implement the framework for this so that the pausing and stopping works and feedback and better ideas most welcome!
from bob.
Related Issues (20)
- Enable stopping jobs during container provisioning HOT 2
- Add status when pipeline cannot be paused HOT 1
- Cascade delete resources of a pipeline when a pipeline is deleted
- Make each type of runner listen to its own queue HOT 2
- Record the timestamp of pipeline status changes HOT 19
- Add get started guide for contributors HOT 7
- Add cluster info and stats API HOT 5
- non-container runner HOT 12
- Add webhook for bob to notice new/changed resources HOT 4
- Test Bob with Bob
- Add support for webhooks HOT 1
- Rename Bob HOT 2
- Allow deploy time config HOT 1
- Event streaming from api-server HOT 14
- Naming of identifier of a run is confusing HOT 3
- Allow testing specific ns HOT 2
- Improve REPL test development HOT 6
- Figure out necessary java modules HOT 3
- Add end to end tests
- Add metrics for the runner for picking up jobs
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from bob.