Discussion: running multiple jobs about stevejobs

[luixal]

Last week, I tried to make a PR adding some config features like setting the collection name and I found in the way the collection is created here: this runs before the main app is run.

Why commenting this here? Because implementing the package in an OOP style would solve both things: settings could be passed when creating the main object and then, the start function would be called. Also, you could create several object instances if you want to run multiple jobs at the same time.

Something like this:

queue1 = new JobQueue({
  collectionName: 'my-job-collection',
  timer: 5 * 1000,
  startupDelay: 5 * 1000,
  activityDelay: 5 * 60 * 1000
});

queue1.addJob(...);
queue1.addJob(...);

queue2 = new JobQueue({
  collectionName: 'my-job-collection',
  timer: 10 * 1000,
  startupDelay: 10 * 1000,
  activityDelay: 10 * 60 * 1000
});

queue2.addJob(...);
queue2.addJob(...);

// in case it makes sense, the start makes sense, the start/stop functions could be called juts manually instead of automatically:
queue1.start();

For example: I have a custom nightQueue that runs the same jobs every night and it gets started by queue1 every night. If I don't need something like that, I only use one global instance of the JobQueue.

What do you think about moving implementation to OOP?

[msavin]

@luixal I like the idea of allowing different collection names to be set, could be handy for debugging too. As for going all OOP, I'm not sure if it would be worth it, and the package is OOP-ish as is. What I really like is that you can just write Jobs.register({...}) and Jobs.run({...}) from anywhere in your app and expect things to work.

[msavin]

In other news, the package was just re-implemented to start an instance of JobsRunner for each registered queue. This means that Jobs now runs multiple queues simultaneously, while keeping the one job at a time approach with-in each queue.

I'm enjoying this approach because:

• jobs run in order and predictably.
• you can still run through thousands of jobs quickly and easily
• cpu impact is minimal

I wonder, maybe this could be a nice distinction of the package? or should we ultimately try to run multiple jobs simultaneously across multiple servers?

[luixal]

I wonder, maybe this could be a nice distinction of the package? or should we ultimately try to run multiple jobs simultaneously across multiple servers?

Not sure about this. I think that running across multiple servers implies to have a state control (or some kind of locks) to avoid different instances running the same job at the same time.

Right now, I'm not sure how would I implement something like this based in Meteor. My first approach would be kind a singleton: a common job dispatcher that runs jobs in different runners across instances.

[msavin]

Quick update on this: the 2.0 is out, and with it, all the functions relating to how jobs are executed are in the operator folder, so maybe in the future, we can upgrade it. For now, I will close this ticket as it relates to the 1.0 release.