Coder Social home page Coder Social logo

stevejobs's Introduction

Steve Jobs

The Simple Jobs Queue That Just Works

Run scheduled tasks with Steve Jobs, the simple jobs queue made just for Meteor. With tight MongoDB integration and fibers-based timing functions, this package is quick, reliable and effortless to use.

  • Jobs run on one server at a time
  • Jobs run predictably and consecutively
  • Jobs, their history and returned data are stored in MongoDB
  • Failed jobs are retried on server restart
  • No third party dependencies

The new 4.0 features repeating jobs, async support and more! It can run hundreds of jobs in seconds with minimal CPU impact, making it a reasonable choice for many applications. To get started, check out the documentation and the quick start below.

Developer Friendly GUI and API

In addition to a simple API, the Steve Jobs package offers an in-app development tool. After installing the main package, run the package command below and press Control + J in your app to open it.

meteor add msavin:sjobs-ui-blaze

Note: this package may be a little bit flakey, and might not work if audit-argument-checks is present. Unfortunately, I had lost the source code, and will probably rebuild the package once there is a good reason to do so.

Quick Start

First, install the package, and import if necessary:

meteor add msavin:sjobs
import { Jobs } from 'meteor/msavin:sjobs'

Then, write your background jobs like you would write your methods:

Jobs.register({
    "sendReminder": function (to, message) {
        const instance = this;

        const call = HTTP.put("http://www.magic.com/email/send", {
            to: to,
            message: message,
            subject: "You've Got Mail!",
        })

        if (call.statusCode !== 200) {
            instance.reschedule({
                in: {
                    minutes: 5
                }
            });
        } else {
            return call.data;
        }
    }
});

Finally, schedule a background job like you would call a method:

Jobs.run("sendReminder", "[email protected]", "The future is here!");

One more thing: the function above will schedule the job to run on the moment that the function was called, however, you can delay it by passing in a special configuration object at the end:

Jobs.run("sendReminder", "[email protected]", "The future is here!", {
    in: {
        days: 3,
    }, 
    on: {
        hour: 9,
        minute: 42
    },
    priority: 9999999999
});

The configuration object supports date, in, on, priority, singular, unique, and data, all of which are completely optional. For more information, see the Jobs.run documentation.

Repeating Jobs

Compared to a CRON Job, the Steve Jobs package gives you much more control over how and when the job runs. To get started, you just need to create a job that replicates itself.

Jobs.register({
    "syncData": function () {
        const instance = this;
        const call = HTTP.put("http://www.magic.com/syncData")

        if (call.statusCode === 200) {
            instance.replicate({
                in: {
                    hours: 1
                }
            });
            
            // to save storage, you can remove the document
            instance.remove();
        } else {
            instance.reschedule({
                in: {
                    minutes: 5
                }
            });
        }
    }
});

Then, you need to "kickstart" the queue by creating the first job to run. By using the singular flag, you can ensure that Meteor will only create the job if there is no pending or failed instance of it.

Meteor.startup(function () {
    Jobs.run("syncData", {
        singular: true
    })    
})

Documentation

Jobs.register and Jobs.run are all you need to get started, but that's only the beginning of what the package can do. To explore the rest of the functionality, jump into the documentation:


Steve Jobs is an MIT-licensed project, brought to you by Meteor Candy.

stevejobs's People

Contributors

banjerluke avatar chris-gong avatar justuswilhelm avatar kpervin avatar macrozone avatar mozfet avatar msavin avatar stevenqzhang avatar

Stargazers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

Watchers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

stevejobs's Issues

"failed" or "failure" ?

Hi Max,

I've been giving V3 a workout over the weekend (I had a horrid day on Friday with hosting issues that also triggered massive amounts of problems with jobs and multi-instance restarts after network failures - so decided to cut everything over and improve the error trapping and processing).....

However, in a number of places in the code you call getJob with the following....

   allow: ["pending", "failed"],
   message: '<appropriate message if it does not find a job>'

And it seems to me as though "failed" should be "failure" (or the state written to the DB should be failed instead of failure) as some of the routines are not behaving as expected and "failed/failure" jobs will not be found... specifically the cancel function (but I can see it on others as well).

I've basically cut everything over and am committed to using this now - so it is about to become a PRODUCTION issue. I can just go into the DB and delete these failed jobs but I would like my in-app jobs console to work correctly using the API.

can i expose doc as parameter when job execute?

before:

var jobResult = Jobs.private.registry[doc.name].apply(null, doc.arguments);

after:

var jobResult = Jobs.private.registry[doc.name].apply(null, [doc]);

why?

  1. i can copy the job in another collection,because i‘m ready to remove a job when it exec successful, so the queue become high-efficiency,but some logic program relay the job arguments and others, so i can use doc copy in another collection.

  2. when run job and need rerun, i can reschedule the job with (jobId, config)

How to work with non-fiber asynchronous flows?

Hey,

Just bringing up this issue again, prior reference here: #16

Behaviour with asynchronous code doesn't seem to work for me. Take this simple example:

import fetch from 'node-fetch'

const job = () => {
  console.log('Running job in queue')
  return new Promise((resolve, reject) => {
    fetch('https://reqres.in/api/users?page=2')
      .then(res => res.json())
      .then(json => {
        setTimeout(() => {
          resolve(json)
        }, 4000)
      })
      .catch(error => {
        reject(error)
      })
  })
}

const queue = function () {
  console.log('Running queue')
  const _this = this
  job()
    .then(result => {
      console.info('got a result', result)
      _this.success(result)
      return
    })
    .catch(error => {
      console.warn('got a error', error)
      _this.failure(error)
      return
    })
}

Jobs.register({ queue })
Jobs.run('queue')

When I start my Meteor server, I then see:

I20180218-18:18:39.886(0)? Running queue
I20180218-18:18:39.909(0)? Running job in queue
I20180218-18:18:39.990(0)? Running queue
I20180218-18:18:39.991(0)? Running job in queue
I20180218-18:18:40.081(0)? Running queue
I20180218-18:18:40.082(0)? Running job in queue
I20180218-18:18:40.170(0)? Running queue
I20180218-18:18:40.171(0)? Running job in queue
I20180218-18:18:40.262(0)? Running queue
I20180218-18:18:40.263(0)? Running job in queue

... and so on... Until I stop the server. So it's stuck in a loop.

Pretty sure the this context is not an issue here — but I could be wrong!

SPAM emails

Something is sending SPAM emails about this package.

They go to email addresses of accounts registered with Github.

It'll be great if that not happens

Cannot read property 'insert' of undefined

For my first run, I copied and pasted the example in the docs.
I created a file called worker.js with the following contents:

import { Jobs } from 'meteor/msavin:sjobs';



Jobs.register({
    "sendReminder": function (to, message) {
        var call = HTTP.put("http://www.magic.com/sendEmail", {
            to: to,
            message: message
        })

        if (call.statusCode === 200) {
            this.success(call.result);
        } else {
            this.reschedule({
                in: {
                    minutes: 5
                }
            });
        }
    }
});


Jobs.run("sendReminder", "[email protected]", "The future is here!", {
    in: {
        days: 3,
    }, 
    on: {
        hour: 9,
        minute: 42
    },
    priority: 9999999999
});

As a result my Meteor app is crashing with the following stacktrace:

W20180611-01:37:40.752(-4)? (STDERR) TypeError: Cannot read property 'insert' of undefined
W20180611-01:37:40.752(-4)? (STDERR) at add (packages/msavin:sjobs/server/imports/actions/add/index.js:31:35)
W20180611-01:37:40.752(-4)? (STDERR) at Object.Jobs.run (packages/msavin:sjobs/server/api.js:55:22)
W20180611-01:37:40.752(-4)? (STDERR) at workers.js (imports/api/workers.js:25:6)
W20180611-01:37:40.752(-4)? (STDERR) at fileEvaluate (packages/modules-runtime.js:343:9)
W20180611-01:37:40.762(-4)? (STDERR) at require (packages/modules-runtime.js:238:16)
W20180611-01:37:40.763(-4)? (STDERR) at main.js (server/main.js:1:524)
W20180611-01:37:40.763(-4)? (STDERR) at fileEvaluate (packages/modules-runtime.js:343:9)
W20180611-01:37:40.763(-4)? (STDERR) at require (packages/modules-runtime.js:238:16)
W20180611-01:37:40.764(-4)? (STDERR) at /Users/rapidmon17/Projects/hello/.meteor/local/build/programs/server/app/app.js:1266:1
W20180611-01:37:40.764(-4)? (STDERR) at /Users/rapidmon17/Projects/hello/.meteor/local/build/programs/server/boot.js:411:36
W20180611-01:37:40.764(-4)? (STDERR) at Array.forEach ()
W20180611-01:37:40.764(-4)? (STDERR) at /Users/rapidmon17/Projects/hello/.meteor/local/build/programs/server/boot.js:220:19
W20180611-01:37:40.764(-4)? (STDERR) at /Users/rapidmon17/Projects/hello/.meteor/local/build/programs/server/boot.js:471:5
W20180611-01:37:40.764(-4)? (STDERR) at Function.run (/Users/rapidmon17/Projects/hello/.meteor/local/build/programs/server/profile.js:510:12)
W20180611-01:37:40.764(-4)? (STDERR) at /Users/rapidmon17/Projects/hello/.meteor/local/build/programs/server/boot.js:470:11

defer vs on

wouldn't it be better to rename on to defer? I had to read twice to grasp the semantics when it's actually trivial, it's deferring a job to a future date. If not calling it defer then maybe run on, still easier/faster to grasp than just on.

The task manager Omnifocus is a good example, different thing but same semantics in principle, they also call it defer when a task is shifted to start at some future date...

Error I cannot explain

Hello I am getting this error when trying to run a job. But I can't get any idea why this happens. Do you see anything here?

I20171227-14:26:26.755(4)? Exception while invoking method 'trigger_WELCOME_AND_VERIFY' TypeError: Cannot read property 'name' of undefined
I20171227-14:26:26.755(4)?     at Jobs.private.run (packages/msavin_sjobs.js:206:48)
I20171227-14:26:26.755(4)?     at Object.Jobs.run (packages/msavin_sjobs.js:705:26)
I20171227-14:26:26.755(4)?     at DDPCommon.MethodInvocation.runJobTrigger_LIGHTBOX_MUST_HAVE_ELEMENTS (imports/02-Notifications/server/methods.js:18:10)
I20171227-14:26:26.755(4)?     at maybeAuditArgumentChecks (packages/ddp-server/livedata_server.js:1768:12)
I20171227-14:26:26.755(4)?     at DDP._CurrentMethodInvocation.withValue (packages/ddp-server/livedata_server.js:1686:15)
I20171227-14:26:26.755(4)?     at Meteor.EnvironmentVariable.EVp.withValue (packages/meteor.js:1134:15)
I20171227-14:26:26.755(4)?     at resolve (packages/ddp-server/livedata_server.js:1684:36)
I20171227-14:26:26.756(4)?     at new Promise (<anonymous>)
I20171227-14:26:26.757(4)?     at Server.applyAsync (packages/ddp-server/livedata_server.js:1683:12)
I20171227-14:26:26.757(4)?     at Server.apply (packages/ddp-server/livedata_server.js:1622:26)
I20171227-14:26:26.757(4)?     at Server.call (packages/ddp-server/livedata_server.js:1604:17)
I20171227-14:26:26.757(4)?     at EventEmitter.GettingStarted.js.Emitter.on (imports/02-Notifications/server/listeners/GettingStarted.js:21:10)
I20171227-14:26:26.757(4)?     at emitNone (events.js:105:13)
I20171227-14:26:26.757(4)?     at EventEmitter.emit (events.js:207:7)
I20171227-14:26:26.757(4)?     at DDPCommon.MethodInvocation.trigger_GETTING_STARTED (imports/02-Notifications/server/methods.js:8:13)
I20171227-14:26:26.757(4)?     at maybeAuditArgumentChecks (packages/ddp-server/livedata_server.js:1768:12)
I20171227-14:26:26.758(4)?  => awaited here:
I20171227-14:26:26.758(4)?     at Promise.await (/Users/hayksafaryan/.meteor/packages/promise/.0.10.0.3vpk2l.pdr8h++os+web.browser+web.cordova/npm/node_modules/meteor-promise/promise_server.js:60:12)
I20171227-14:26:26.758(4)?     at Server.apply (packages/ddp-server/livedata_server.js:1635:14)
I20171227-14:26:26.758(4)?     at Server.call (packages/ddp-server/livedata_server.js:1604:17)
I20171227-14:26:26.758(4)?     at EventEmitter.GettingStarted.js.Emitter.on (imports/02-Notifications/server/listeners/GettingStarted.js:21:10)
I20171227-14:26:26.758(4)?     at emitNone (events.js:105:13)
I20171227-14:26:26.759(4)?     at EventEmitter.emit (events.js:207:7)
I20171227-14:26:26.759(4)?     at DDPCommon.MethodInvocation.trigger_GETTING_STARTED (imports/02-Notifications/server/methods.js:8:13)
I20171227-14:26:26.759(4)?     at maybeAuditArgumentChecks (packages/ddp-server/livedata_server.js:1768:12)
I20171227-14:26:26.759(4)?     at DDP._CurrentMethodInvocation.withValue (packages/ddp-server/livedata_server.js:1686:15)
I20171227-14:26:26.759(4)?     at Meteor.EnvironmentVariable.EVp.withValue (packages/meteor.js:1134:15)
I20171227-14:26:26.759(4)?     at resolve (packages/ddp-server/livedata_server.js:1684:36)
I20171227-14:26:26.759(4)?     at new Promise (<anonymous>)
I20171227-14:26:26.759(4)?     at Server.applyAsync (packages/ddp-server/livedata_server.js:1683:12)
I20171227-14:26:26.759(4)?     at Server.apply (packages/ddp-server/livedata_server.js:1622:26)
I20171227-14:26:26.759(4)?     at Server.call (packages/ddp-server/livedata_server.js:1604:17)
I20171227-14:26:26.760(4)?     at EventEmitter.WelcomeAndVerify.js.Emitter.on (imports/02-Notifications/server/listeners/WelcomeAndVerify.js:39:10)
I20171227-14:26:26.760(4)?  => awaited here:
I20171227-14:26:26.760(4)?     at Promise.await (/Users/hayksafaryan/.meteor/packages/promise/.0.10.0.3vpk2l.pdr8h++os+web.browser+web.cordova/npm/node_modules/meteor-promise/promise_server.js:60:12)
I20171227-14:26:26.760(4)?     at Server.apply (packages/ddp-server/livedata_server.js:1635:14)
I20171227-14:26:26.760(4)?     at Server.call (packages/ddp-server/livedata_server.js:1604:17)
I20171227-14:26:26.760(4)?     at EventEmitter.WelcomeAndVerify.js.Emitter.on (imports/02-Notifications/server/listeners/WelcomeAndVerify.js:39:10)
I20171227-14:26:26.760(4)?     at emitNone (events.js:105:13)
I20171227-14:26:26.760(4)?     at EventEmitter.emit (events.js:207:7)
I20171227-14:26:26.761(4)?     at DDPCommon.MethodInvocation.trigger_WELCOME_AND_VERIFY (imports/02-Notifications/server/methods.js:12:13)
I20171227-14:26:26.761(4)?     at maybeAuditArgumentChecks (packages/ddp-server/livedata_server.js:1768:12)
I20171227-14:26:26.761(4)?     at DDP._CurrentMethodInvocation.withValue (packages/ddp-server/livedata_server.js:719:19)
I20171227-14:26:26.761(4)?     at Meteor.EnvironmentVariable.EVp.withValue (packages/meteor.js:1134:15)
I20171227-14:26:26.761(4)?     at DDPServer._CurrentWriteFence.withValue (packages/ddp-server/livedata_server.js:717:46)
I20171227-14:26:26.761(4)?     at Meteor.EnvironmentVariable.EVp.withValue (packages/meteor.js:1134:15)
I20171227-14:26:26.761(4)?     at Promise (packages/ddp-server/livedata_server.js:715:46)
I20171227-14:26:26.761(4)?     at new Promise (<anonymous>)
I20171227-14:26:26.761(4)?     at Session.method (packages/ddp-server/livedata_server.js:689:23)
I20171227-14:26:26.761(4)?     at packages/ddp-server/livedata_server.js:559:43

Jobs: invalid argument was ignored: priority

Jobs.register({
  sayHello: function (who) {
    const text = `hello ${who}`;
    console.log(text);
    return text;
  },
  sayHi: function(who){
    const text = `hi ${who}`;
    console.log(text);
    return text;
  }
});
Jobs.run(sayHi,'aaa' {
        in: {
          days: 5
        },
        priority: 10
      })

then run with log: Jobs: invalid argument was ignored: priority

How to run the job every 15 minutes?

I want to run my job for every 15mins, how could i do this.I wrote my job configuration like below

Jobs.register({
testjob: function () {
console.log("calling testjob");
this.success("sucess");
return;
}
});
// schedule the job
Jobs.run("testjob",{
in: {
minute: 15
}
});

But it's not working, can you please help me on this

Correct way to run recurring job

I want to implement a monthly billing function using a job. The billing should occur every 30 days after the user subscribed to a service until canceled.

From the documentation, I understand I cannot mark a job as successful and then reschedule it (reschedule is only for failed jobs). Should I replicate the job after setting this.success in the register job function, or what is the best practice for a repeating job?

[Feature Request] - Jobs.stop - stop all instances...

Given Dominator is a central point, it may be possible to implement a flag in the Dominator record that, if set, means the instance will stop itself. With that, it would be possible to pass a "stop all" flag to Jobs.stop in a multi-instance environment (to be able to stop queues across ALL servers by setting the flag in the dominator record).

Similarly, all instances could be restarted again by changing the flag back.

It's an "emergency" thing - in case jobs are going crazy.

Jobs.configure fails when setting the timer

Jobs.configure({
	timer: 1000
});

Error: Match error: Unknown key in field timer
W20180317-17:34:57.953(1)? (STDERR) at check (packages/check/match.js:35:15)
W20180317-17:34:57.954(1)? (STDERR) at Object.Jobs.configure (packages/msavin:sjobs/server/api.js:10:2)

reschedule does not work if there is only a single job of that name

I'm new to the package and was just doing some testing (only a a few hours on it - looking to replace the vsivsi job collection package).

I used the example sendReminder and added in a reschedule instead of success and set it for 10 seconds.

If I run it with a clean jobs_data collection then it never triggers again.

However, if I shut the server down and run it up again (and now there are two sendReminder records) then they both run at the expected interval.

NOTE: if I then delete one of the two records out while the job is still running, it then stops and does not trigger the job any more.

I had a look at the code quickly and there seems to be a blocker in there if the same _id was just run in a Queue.

https://github.com/msavin/SteveJobs...meteor.schedule.background.tasks.jobs.queue/blob/84df2fa3cebb9b50b6b3b7ce3ffc3400c5c10b12/package/server/imports/operator/queue/index.js#L86

This would make sense if there were two records for that queue. One would execute and then the other - alternating between the two. Maybe this query also needs to look at the status to see if it is pending ?

Quite Confusing Delay Schedule

Hi Max, good job!

But can you please elaborate on this?:

The date object will be updated in the order that is specified.
 For example, if you set the job to run in 1 year, an on year 2037, the year will be 2037. 
However, if you set the job to run on year 2037, and in 1 year, the year will be 2038.

Do u mean that these jobs are not the same?
1

Jobs.add("sendReminderEmail", "[email protected]", "The future is here!", {
    in: {
        days: 1,
        hours: 13
    }, 
    on: {
        minute: 13,
        year: 2037
    }
});

2

Jobs.add("sendReminderEmail", "[email protected]", "The future is here!", {
    on: {
        days: 1,
        hours: 13
    }, 
    in: {
        minute: 13,
        year: 2037
    }
});

Didn't try this though but it confused me while reading the instructions.

Thanks in advance.

How to schedule jobs to repeat at a particular time?

I want to run the job at particular time like everyday at 12AM, how can i achieve this, below are my sample code

[Jobs.register({
everydayjob: function () {
console.log("calling everydayjob");
this.success("sucess");
return;
}
});
// schedule the job
Jobs.run("everydayjob",{
in: {
hours: 24
}
});

How can i configure at run the job at particular time . Thanks in advance

Updating jobs

During local development I'm noticing that it creates a new copy of a job whenever the server restarts. I expected subsequent restarts to simply update the existing job (that has the same job id). Is there some mechanism here that I am not understanding?

I expect that the implementations of my job will change over time, how do I ensure that new jobs that I register with new code overwrite existing versions instead of creating a new, different job?

It appears the calling Job.run at server startup creates a new entry in the jobs_data table, and then eventually all of these copies of the job run on the defined run schedule.

Cannot import the module using ES6 ?

import { Jobs } from 'meteor/msavin:jobs' is failing on startup ...

W20180115-17:02:37.789(1)? (STDERR) Error: Cannot find module 'meteor/msavin:jobs'

Any idea why ? (I added it using atmosphere)


EDIT : typo on readme, import { Jobs } from 'meteor/msavin:sjobs';

s missing.

[FEATURE REQUEST] add 'serverId' to Jobs.configure({options}) - use instead of Random.id() if not null

The use case for this is in a situation like Meteor Galaxy where each instance is already allocated an ID and it would be useful to be able to correlate the Galaxy instance ID with the instance that is running the queues.

With that information it would be very easy to restart the instance and, hence, force another instance to take over.

[Also - it seems that the Dominator adds a record to jobs_dominator for each registered job but it only sets the lastPing on one of them. I would probably only expect one record for the current server running the queues. Or a separate one for each job name (each updating separately)]

Discussion: prioritizing jobs

Currently, the package runs jobs based on which is due the soonest. However, we might want to give some jobs priority over others. For example, if an application is sending out email newsletters over a queue, perhaps paying customers get to skip the wait.

There's three ways to implement this:
1- have a expedite: true field
2- allow people to set arbitrary numbers for priority (ie. 1000, 900, 1)
3- have some kind of priority system (for example, numbers rated 1-10 have different effects)

I'm leaning on #1 for simplicity, or #2 for flexibility. I noticed some queue's have an approach like #3, but unless I am missing something, I do not see the benefit.

Code execution error

The package in it's current state does not work as indicated on the readme.
Following the basic example gives me this error:

TypeError: Cannot read property 'insert' of undefined
at add (packages/msavin:sjobs/server/imports/actions/add/index.js:31:35)
at Object.Jobs.run (packages/msavin:sjobs/server/api.js:55:22)
at jobs.js (imports/server/startup/jobs.js:28:6)
at fileEvaluate (packages/modules-runtime.js:333:9)
at require (packages/modules-runtime.js:228:16)
at index.js (imports/server/startup/index.js:1:171)
at fileEvaluate (packages/modules-runtime.js:333:9)
at require (packages/modules-runtime.js:228:16)
at main.js (server/main.js:1:14)
at fileEvaluate (packages/modules-runtime.js:333:9)

Jobs.reschedule does not accept Date objects

I'm trying to reschedule an existing job for a given date, so I'm using the following code:

Jobs.reschedule(<jobId>, {
    date: new Date(<date specification>),
});

However, when this is run I get the following error:

Error: Match error: Expected plain object in field date
    at exports.check (packages/check.js:55:15)
    at Object.Jobs.reschedule (packages/msavin:sjobs/server/api.js:129:2)
    ...

Should I be specifying the date as something other than a Date object? When I create the job in a similar way everything seems to work as it should:

Jobs.run('<name>', <arg>, {
    date: new Date(<date specification>),
});

Debugging story/development mode?

Request: could there be some sort of scenario covered where we could run the jobs in dev mode, either by accelerating the clock manually, or having all the jobs run at a "debug" interval in addition to actual interval used in prod?

call this.set(key, value) get error!

TypeError: Cannot set property 'count' of undefined
at toolbelt.set (packages/msavin:sjobs/server/imports/actions/execute/toolbelt.js:27:4)

// first, update the document
		var update = Utilities.collection.update(docId, {
			$set: patch
		})
	
		// second, patch the cached document if write is successful
		if (update) {
			this.document.data[key] = value
		}

because this.document.data is undefined!

same reason, self.get(key), if key is not exist, get error!
TypeError: Cannot read property 'count1' of undefined
at toolbelt.get (packages/msavin:sjobs/server/imports/actions/execute/toolbelt.js:48:10)

return this.document.data[key] || null;

because this.document.data is undefined!

Future Features

I don't know if this is open to community consideration but... what the hell! I'm going to write down my opinions over the possible new features commented in the readme, just to get more opinnions :)

  1. Create a way to run jobs across multiple servers. [...] --> maybe different collections? so each server can handle its own? Would need to implement kind of a "proxy" in the API but seems better for performance, doesn't it?
  2. Create a way for the server to pause jobs if the CPU usage is high. Every little bit helps, right? --> i would implement this after 5, just to allow to run important jobs in any case
  3. Create a way to run jobs as a microservice. --> maybe that would be another project that uses this package and adds a REST API?
  4. Create a way to repeat jobs. (Perhaps this should be another package?) --> as there are many formats to represent schedules and repetitions, I would go with different packages, yep 👍
  5. Create a way to prioritize certain jobs --> nothing to add here :)
  6. Add support for hooks --> nothing to add here neither :)
  7. Add Jobs.delay() to delaying tasks --> maybe a field in the job object? a Jobs.delay() to delay all jobs and a Jobs.delay(jobId), Jobs.delay(priorityLevel), ... also related to 5?
  8. Add support for setting manual timezones on jobs --> using momentJS should be "easy", isn't it?

bonus track:

  1. Add a template you can use as UI in a project's admin are (control job queue, jobs and get execution logs) --> this would totally be another package :)
  2. Add some basic functions like: Jobs.isStarted() and Jobs.getStats() or just a Jobs.getStatus() for getting if the queue is stopped/started and jobs pending/failed/succeeded/canceled/running. This would be related to 1 and 5.

I'm totally interested in 4 and quite interested in 7 and 8. Also in 9 and 10 logically. I can put some spare time in this too :)

Setting activityGap per "kind" of job?

On my application some jobs ( or type of jobs ) can have a different activityGaps, so i believe it would be important to have activeGap set per document and not as a "global" option if that makes sense?

Server dominator/arbitration is not being maintained

I've implemented V3.0.1 (with the 'failed/failure' fix) but that is not my issue (nothing is failing right now)....

The server arbitration (dominator) does not appear to be working properly.

Here is a screenshot of my dominator table (3 instances running).....

image

Here is a screenshot of my getEmails job running (out of control)....

image

It always creates 3 copies of the getEmails job and sets them as pending - and then it only seems to resolve one of them and then goes on to create 3 more pending jobs.

MaxListenersExceededWarning

Hi Max,

Since working with the Jobs package I've been getting

(node:8326) MaxListenersExceededWarning: Possible EventEmitter memory leak detected. 11 pipe listeners added. Use emitter.setMaxListeners() to increase limit

Have you seen this before? Any thoughts what may trigger this in my scheduled methods?

Cheers,

Dirk

Did not check() all arguments during publisher 'jobsDebugger' error when attempting to load Gui debugger

Not sure why, but the exception I get is this:

Exception from sub jobsDebugger id sTZfK5nZtaHLusoqT Error: Did not check() all arguments during publisher 'jobsDebugger'
I20180131-22:57:12.996(-8)?     at ArgumentChecker.throwUnlessAllArgumentsHaveBeenChecked (packages/check.js:483:13)
I20180131-22:57:12.996(-8)?     at Object._failIfArgumentsAreNotAllChecked (packages/check.js:131:16)
I20180131-22:57:12.996(-8)?     at maybeAuditArgumentChecks (packages/ddp-server/livedata_server.js:1765:18)
I20180131-22:57:12.996(-8)?     at DDP._CurrentPublicationInvocation.withValue (packages/ddp-server/livedata_server.js:1043:15)

Discussion: running multiple jobs

Currently, the queue just runs one job at a time based in order of the time the job is due. Whenever a server starts up, it could start the JobsRunner, which gets one job document from MongoDB and runs the corresponding code.

See: https://github.com/msavin/SteveJobs-meteor-jobs-queue/blob/master/package/server/runner.js

In the next version, maybe we can start one JobsRunner for each job we have registered. For example, the code below could start two runners, sendReminderEmail and insertRecord:

Jobs.register({
    sendReminderEmail: function (to, content) {
        Email.send({
            to: to,
            from: "[email protected]",
            subject: "Your Reminder",
            content: content,
        })
    },
    insertRecord: function (data) {
        Collection.insert({
            date: new Date(),
            data: data
        });
    }
});

The package would still process jobs one at a time and in the order received, while letting different jobs run at the same time.

how to deal with async and await in register?

Jobs.register({
  rest: async function(url){
    const res =  await fetch(url, {
      method: 'POST'
    });
    return res;
  }
});

i just want fetch a restful api result in job, how can i return the result in job.

feature question

Hi,
I found this package is very interesting but I'm not sure if this package will fulfill my requirement.
We have a dedicated server to do video compressing work. Our requirement is:

  • Server can run up to X threads at same time. Each thread will be compressing one video file using ffmpeg. The actual video compressing code in ffmpeg is async.
  • Thread is considered as done and deleted when it finishes the video processing.
  • Server checks how many threads are running in every Y seconds interval. If total running threads are less than X, server can create new thread to process video file.

The tricky part here is new job cannot run until current total running job count is less than X. I didnt find how to achieve this in this package.

Repeating a job

Thanks for yet another very nice package @msavin!

I would love to use SteveJobs for my background tasks but I would desperately need one thing: repetition. I need to be able to repeat a job every xy amount of time indefinitely - like checking a remote server for certain updates every 24h.

Could you implement such a feature?

Thousands of jobs resulting in "RangeError Maximum call stack size exceeded"

Thanks for this great package :).

I'm having an issue when I schedule a large number of jobs to run immediately. After processing several thousand of the jobs the server throws "RangeError: Maximum call stack size exceeded"

More details: I'm trying to run about 4000 jobs, in the same queue. Each job involves copying a file from one folder on the server to another, inserting details for it into a collection (using ostrio:files Meteor package), then performing a couple of other ordinary collection inserts (some via collection hooks). If I split up the jobs into two batches of about 2000 files it works fine on all the files (ie it doesn't seem to be anything about a specific file). The last couple of times I tested it it failed after processing 2753 jobs exactly (but different amounts of elapsed time).

Maybe a recursive call structure is used to process jobs sequentially, eventually resulting in a stack overflow when many jobs are queued?

The job code is below.

Jobs.register({
  'image.importLocal': (dataSetId, srcDirPath, fileName) => {
    const srcFilePath = srcDirPath + '/' + fileName;
    const destFilePath = Meteor.settings.dataSets + '/' + dataSetId + '/image/' + fileName;

    try {
      fse.copySync(srcFilePath, destFilePath, { overwrite: false, errorOnExist: true });
      Image.addFile(destFilePath,
        {
          fileName: fileName,
          type: mime.getType(fileName),
          meta: {dataSetId: dataSetId },
        },
        (error, fileObj) => {
          if (error) {
            console.error("imageImportLocal: error adding file: " + srcFilePath + ": " + error);
            fse.remove(destFilePath);
          }
        }
      );
    }
    catch (err) {
      console.warn("imageImportLocal: skipping file: " + srcFilePath + ": " + err.message);
    }
  }
});

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❤️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.