Comments (7)
Hello @fritzy, I understand the use case for this, and there is a symmetrical feature in Amazon SQS for example, but at the same time there is no global retry counter
being Disque distributed. This problem could be kinda fixed by using a non perfect counter in each node that uses QUEUED messages in order to take the counter updated. However there is another problem due to the distributed nature of Disque, we also need to coordinate nodes so that usually we get just a dead letter for each job that was not able to be processed, otherwise all the nodes having a copy may put the job into the dead letter queue multiple times.
This also seems to be kinda solvable using some protocol that does not ensure unicity but that approximates it. Basically it is not a trivial problem, but with some design we may find ways to solve it.
A few more thoughts:
- This means to enlarge the job structure even more, it's already 120 bytes.
- Maybe to specify the queue name for dead letters explicitly is not strictly needed? We may just prefix the queue name with something like dead_letter:
queue_name
. Not sure about that, saves memory but is less flexible, maybe some user may want all the dead letters to go into the same queue. - It is possible to address this in a much more memory efficient way at the cost of other things: we are going to have an iterator to iterate jobs given specific attributes. We may ask to iterate all the jobs inside the node where the approximated_retry_attempt is greater than N. However this makes the problem a per node problem, while all the rest of the Disque API is distributed and not linked to a specific node, so I'm not a big fan of this.
- The obvious counter argument for this feature is that this should be up to workers: when they can't process a job they may add a dead letter themselves for people to inspect. This is how Resque works I guess? However this means certain kinds of failures are harder to catch, for example the worker crashing as soon as it starts to process a given message, without having a chance to send a dead letter.
No clear ideas here in my side :-) Need to think / work on this more time. Any feedback?
from disque.
Three thoughts:
Generated names are fine.
NSQ allows you to listen for requeue events (I believe). If you could do this, a watcher process could manually move it elsewhere, assuming it could prevent a worker from picking it back up.
The accuracy of the counter isn't terribly important. Really we're just looking for a threshold of pain -- this job is not likely to succeed, so we pass it off somewhere else. So if we said 5 failures, and we didn't notice until 10 failures, it's a bit wasteful, and takes up resources, but the number of times itself isn't really that important.
from disque.
Isn't one global dead job queue enough? It's up to workers to process dead jobs, and report them. Using disque as a "database" for dead jobs doesn't seem like a good idea. Even if it only means to sort them by the originating queue.
I think it could be solved by allowing to add metadata to each message, just as RabbitMQ does. We need it other usecases too. For example for failed jobs we want to add stacktrace and error name to the jobs and put it in dead jobs queue. At the same time we'd like to avoid modyfying body of the original job.
from disque.
@fritzy @sheerun thanks for your comments. A few random arguments to keep the discussion going.
- It's interesting to note that the dead letter could be implemented client side if we offer, via GETJOB, a way to fetch also this approximated "retry counter" so that the worker itself may add the job into the dead letter. The only issue I see with this compared to the other approach of handling this inside the server, is that if we have a failure where the worker immediately crashes once it tries to process the job, it will not be able to add the dead letter.
- The Pub/Sub thing would be nice and extremely easy to implement, just a command where you listen for events like: job expired before being processed, job retry limit reached, ... However like any fire and forget thing it is not reliable. But, maybe it's not really needed to be reliable if we re-issue the event every time the job is re-queued with a retry count greater than the retry limit. Eventually some listener will notice. So this could be an option.
I'm a bit more biased towards the client-side solution if you think this could work, for reasons of simplicity of the Disque server and for the reasons @sheerun says: usually we want to augment the failed job with more information that the worker provides.
So a feature sketch for you to evaluate:
- Take the approximated count of the number of times a job was reissued in some interesting way. Not super reliable during failures, but reasonably reliable otherwise would be cool. We can use the QUEUED or WILLQUEUE cluster message I guess or something like that.
- Provide a GETJOB option to also fetch the retry count for the job, so that the worker can act ASAP if needed and post a dead letter.
Sounds good?
from disque.
Don't get me wrong. I think putting dead jobs to dead job queue (or job_events queue described below) after timeout (no ack), is quite important. Without this there's no guarantee some jobs aren't dropped, because worker hangs or doesn't implement dead-job logic.
I think one clean solution would be to:
- Not to handle retry counts at disque level at all
- Introduce
unacknowledged
/timeout
state for a job. - Create some special
job_events
queue that both disque and clients can push to. Clients can subscribe to job events selectively in a pub/sub manner (e.g. on transition tounacknowledged
disque adds a job tojob_events
queue stating exactly that). - Clients can process events from
job_events
and re-schedule dead jobs as needed with increased re-try counter (stored in the metadata). I still think introducing metadata to jobs is quite important.
from disque.
On the other hand the body of jobs in job_events
queue could serve as jobs metadata. Clients could read it and store somewhere if needed.
from disque.
Closing this in favor of #68 in order to start a fresh discussion around a different design proposal. My top priority was to explore alternative designs giving the same practical features without to resort to actual dead letter implementation.
from disque.
Related Issues (20)
- Status of the project HOT 14
- Received duplicated jobs
- Client supplied job ids?
- is this project still alive? HOT 1
- GETJOB from ALL?
- HELLO response with node missing IP address HOT 2
- Is the PING command intended for end-user use?
- WHY IT LOST JOBS? HOT 2
- QLEN inconsistent results HOT 5
- Increase Memory Size for disque-server
- Should I exec "FASTJOB" or "ACKJOB" after I exec "GETJOB" ? HOT 4
- When the author decide to publish the stable version ? HOT 1
- Document default REPLICATION in ADDJOB
- Missing info about SHOW command in the README
- Duplicate consumer when add_job param of repli no equal the cluster node numbers
- Project status update! HOT 7
- Does anyone know of an armv7 compiled version? HOT 1
- job received by multiple clients
- 有编译好的disque吗
- make next
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from disque.