assaf / ironium

Job queues and scheduled jobs for Node.js, Beanstalkd and/or Iron.io.

👏 Hey there, I’m Assaf.

🏖 Right now I'm “inbetween startups“, just a fancy way of saying I do some consulting/advising, while planning the next move.

☕️ I blog at https://labnotes.org/ about software design and development, people and management, culture and technology.

Feel free to subscribe.

ironium's Issues

DoS when not calling callback on schedules job

So first I create a scheduled job that never calls callback() (simulation of crash).

var ironium = require('ironium');

ironium.scheduleJob('second', '1s', function(callback) {
  console.log("I run every second");
});

ironium.start();

I run server for 10 seconds, stop it, add callback() to the handler, and start it again.

The result is DoS of previously scheduled jobs that were meant to run every 1s at max:

I run every second
I run every second
I run every second
I run every second
I run every second
I run every second

If I set interval to 1s, I'd expect this job will run every 1s at max. So if callback is not called, further schedules are dropped. Is there a way to achieve it?

Is ironium dead?

Hi guys, just posting a random question to know if this project is/will be maintained.

Unable to test if queue.delayJob has been processed

Ironium version: 4.1.3
Test suite: mocha 3.2.0

Problem:
I'm trying to test queue.delayJob processes a job after X amount of delay. I'm using timekeeper to stub Date and Date.now. After invoking queue.delayJob and Ironium.runJobs() in the test suite, these test(s) fail because the job is not processed. Beanstalkd checks the delay of a job against hte system time, but Timekeeper doesn't stub the system time.

Potential solution(s)?

• Use another library to stub the system time (eg, ionix) instead of Timekeeper. I started down this path and it was a bit complicated to implement.
• Set Ironium's delay based on the environment: for example, set it to 1s in test, in dev/prod set to the actual delay time.
• Create a method in Ironium to inspect the queue, and find the job with the delay in one test. In another test, run the job without delay and check the if the job was processed correctly.

Beanstalkd webhook url is localhost only?

Looking at the configuration.js, it appears that the assumption is that any webhook usage for beanstalkd would always access localhost. I'm not clear if this is an oversight, or indeed even an issue, but it strikes me as incorrect. Shouldn't the host in the WEBHOOK_LOCAL that's used for beanstalkd config use whatever beanstalkd host is specified?

Browsing the rest of the codebase it wasn't immediately apparent if the webhook url was actually necessary at all though.

Would appreciate some guidance on this, even if just to tell me it's really a non-issue :)

Long running jobs (> 10m)

Is there a way to deal with long running jobs, that would take longer then 10 minutes to complete?

Is it possible to specify max retry?

Error queuing to tasks: Unknown error

My logs occasionally contain this error. Looking at the code it seems to be a connection error when posting a job to iron.io. I think those jobs simply get lost and I am not quite sure how to handle those cases. I am actually a bit uncertain if the problem is the iron.io API or something being stuck in ironium or node, because it would seem very unusual that the iron.io API is unreachable for the period of time these errors occur.

Error queuing to tasks: Unknown error
at /pipeline/source/node_modules/ironium/lib/queues.js:173:20
at /pipeline/source/node_modules/iron_mq/lib/client.js:235:18
at APIClient.Client.parseResponse (/pipeline/source/node_modules/iron_core/lib/client.js:251:18)
at /pipeline/source/node_modules/iron_mq/lib/api_client.js:165:16
at Request._callback (/pipeline/source/node_modules/iron_core/lib/client.js:180:20)
at Request.self.callback (/pipeline/source/node_modules/iron_core/node_modules/request/request.js:344:22)
at emitTwo (events.js:100:13)
at Request.emit (events.js:185:7)
at Request.<anonymous> (/pipeline/source/node_modules/iron_core/node_modules/request/request.js:1239:14)
at emitOne (events.js:95:20)
at Request.emit (events.js:182:7)
at IncomingMessage.<anonymous> (/pipeline/source/node_modules/iron_core/node_modules/request/request.js:1187:12)
at emitNone (events.js:85:20)
at IncomingMessage.emit (events.js:179:7)
at endReadableNT (_stream_readable.js:906:12)
at nextTickCallbackWith2Args (node.js:474:9)
at process._tickDomainCallback (node.js:429:17)
at process.wrappedFunction (/pipeline/source/node_modules/newrelic/lib/transaction/tracer/index.js:245:51)

Bulk loading jobs

I'm looking at using Iron.io to do bulk processing for 100s of thousands of micro-jobs. Is there a way to bulk-upload jobs as well as letting a worker grab multiple jobs at a time?

Nice work on this library btw! Been enjoying poking around your code/docs.

Scheduled jobs with iron.io don't work properly

I've created a simple worker that uses iron.io as backend:

var ironium = require('ironium');
var config = require('./iron.json');
ironium.configure(config);

ironium.scheduleJob('second', '1s', function(next) {
  console.log('scheduling');
  next();
});

ironium.start();

but when I run it, scheduling is displayed only once. I see processed message in ironmq, but only one first job is consumed. No next job is scheduled.

Errors in workers are caught and discarded

Which makes debugging them not very fun. I'm assuming one of your promises are catching them? Could we rethrow caught errors from workers?

Reduce the number of iron.io api request

I have had 360k API request on iron.io and there have only been 149 messages. Any idea what's causing all those requests?

beanstalkd version?

What server version is this tested against?
I'm loving the syntax you provide in this library, saves me from wrapping fivebeans on my own, however i keep seeing: beanstalkd: prot.c:710 in check_err: read(): Connection reset by peer

And i'm wonderinf if there could be a version mismatch error?

next param to queueJob doesn't exist

Hey,

The readme says:

customer.on('save', function(next) {
  if (this.isNew)
    sendWelcomeEmail.queueJob(this.id, next);
  else
    next();
});

but the queueJob signature don't have callback as second param. maybe you forgot to fix docs.

Error queuing to tasks: DRAINING

This is happening running against iron.io.

I am running on version 3.0.1 but this was happening on 2.7.0 before. Basically the queue doesn't work at all - neither sending nor receiving messages.

I have also deleted the MQ2 queue to make sure it's not a compatibility issue. First restarted my ironium application and expected it to setup the queue automatically, which didn't work. Then I created the queue by hand, but it didn't work either.

Here goes the stack:

Error: Error queuing to tasks: DRAINING
  at Queue._delayJob$ (/pipeline/source/node_modules/ironium/lib/queues.js:549:19)
  at tryCatch (/pipeline/source/node_modules/ironium/node_modules/babel-runtime/regenerator/runtime.js:67:40)
  at GeneratorFunctionPrototype.invoke [as _invoke] (/pipeline/source/node_modules/ironium/node_modules/babel-runtime/regenerator/runtime.js:315:22)
  at GeneratorFunctionPrototype.prototype.(anonymous function) [as throw] (/pipeline/source/node_modules/ironium/node_modules/babel-runtime/regenerator/runtime.js:100:21)
  at GeneratorFunctionPrototype.invoke (/pipeline/source/node_modules/ironium/node_modules/babel-runtime/regenerator/runtime.js:136:37)
  at run (/pipeline/source/node_modules/ironium/node_modules/babel-runtime/node_modules/core-js/library/modules/es6.promise.js:89:39)
  at /pipeline/source/node_modules/ironium/node_modules/babel-runtime/node_modules/core-js/library/modules/es6.promise.js:100:28
  at flush (/pipeline/source/node_modules/ironium/node_modules/babel-runtime/node_modules/core-js/library/modules/$.microtask.js:17:13)
  at doNTCallback0 (node.js:407:9)
  at process._tickDomainCallback (node.js:377:13)
  at process.wrappedFunction (/pipeline/source/node_modules/newrelic/lib/transaction/tracer/index.js:271:51)

purge specific queue job

Is it possible to have multiple workers connect to the same queue?

When I run the worker application in a docker container and scale the container to run in let's say 5 instances. Will I be able to process 5 times as many messages from the queue or will each message be processed 5 times?