Comments (7)
Hey @odarriba! This is an excellent question and something that I have previously deliberated a lot. In the end, I decided to not include this in que
for multiple reasons:
-
Jobs are typically run only on a single machine, and in most of the cases, it's the main application server. I wanted to make
Que
super simple to use. -
Not everyone wants to handle failures in the same way. And while it's common to retry them, Que's default mode of operation is to not do anything.
-
Distribution is hard and no single solution applies to all scenarios. I didn't want to lock
que
to a specific node distribution and job processing model. -
Same as the 3rd point, not all consensus algorithms are created equal and apply to same situations.
How/when/if an application should be replicated to another node is up to the developer. See the Distributed Applications guide on the Erlang website. All that said, for a simple node failover model, this should be pretty straightforward. You can take a look at how singleton
implements this (though this means you'll have to start que in runtime: false
mode and manually set up the distribution strategy).
from que.
Thinking about this more (and seeing how quickly your post got many 👍), I might have to reconsider my position on this.
I'm open to the idea of integrating this into Que if we're able to come up with sane defaults that are easy to change and do not negatively affect the developer experience. PRs/Issues are welcome if you have any ideas on how to approach this.
from que.
I might suggest adding a node parameter to the que table so that queue entries are associated with the node they were created on. My cluster handles about half a billion device reports per day and spans multiple nodes. There are certain tasks I'd like to queue up but they'd overwhelm any single node responsible for handling them.
from que.
Or possibly use Mnesia local table type although I'm not entirely sure with all of the details of that table type.
from que.
@odarriba because I needed some additional functionality I made a fork this morning.
I have added the ability to set a priority level on jobs such that all :pri0
jobs will be processed before any :pri1
, :pri2
, or :pri3
ones (in that order).
Additionally the schema has been updated to include current node. So that when restarting only jobs queued on a specific node will be requeued.
I have not yet added load balancing logic since I don't need it for my immediate needs but you can simulate it well enough using something like.
cluster = [:"node1@domain", :"node2@domain", ...]
for number <- 1..100 do
Que.remote_add(Enum.random(cluster), TestQue.TestWorker, number)
end
Or if using the original version of the code.
timeout = 50_000
cluster = [:"node1@domain", :"node2@domain", ...]
for number <- 1..100 do
:rpc.call(Enum.random(cluster), Que, :add, [TestQue.TestWorker, number], timeout)
end
use :rpc.cast if you don't care about confirmation.
@sheharyarn you have written some fantastically readable code here ^_^ keep up the good work. If you'd like to incorporate the priority logic chat with me sometime. I'll continue to refine it either way on my fork since I need to very efficiently process tens of thousands of jobs a minute with prioritization.
from que.
Thank you for the suggestions @noizu (and sorry for the late reply!). I took a quick look at your fork and it looks very interesting. Over the next few weeks, I'll try to come up with a plan for adding distributed job execution support, and will post back here with an update.
from que.
from que.
Related Issues (15)
- Worker is an Invalid Worker HOT 5
- perform completes before monitor is installed causing failure HOT 3
- Que failing with Distillery release and Docker HOT 3
- setting :mnesia for Pow breaks non-persistent que
- Merge Request - Noizu Fork HOT 3
- Inserting into queue is slow when queue is long
- Priority Queues HOT 2
- The pattern 'false' can never match the type 'true' HOT 7
- When running que.setup: (Memento.MnesiaException) Mnesia operation failed HOT 1
- Noob question HOT 1
- Delayed Jobs HOT 5
- How to properly shutdown app with Mnesia backed queue HOT 10
- Can't compile que on Erlang 20.2 and Elixir 1.6rc1 HOT 6
- Compilation issue on Elixir 1.7 HOT 3
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from que.