hokify / agenda Goto Github PK

View Code? Open in Web Editor NEW

52.0 52.0 22.0 5.99 MB

Lightweight job scheduling for Node.js

Home Page: https://hokify.github.io/agenda/

License: Other

JavaScript 0.22% TypeScript 99.78%

agenda's People

Contributors

Stargazers

Watchers

Forkers

uzlopak rviar gmh5225 guoard golatoye ghostker luizcarlos1405 bompus ponomarevlad goswift-repo kndonetm nesto-software dnvyrn alejandro-martinez whisthub ubinatus joyspring

agenda's Issues

`runOrRetry()` method in `JobProcessor` class "leaks" timers

the method creates a promise race between the execution of the actual job and a new timer, to check wheater the job is still running/expired after a set amount of time.

but this timer is never cleared after the job fails/succeeds, or after the agenda.stop() method is called, so each new job will "leak" a new timer that will be cleared only after it is expired, and these can stack up pretty quickly.

link to code

Concurrency issue

We have scheduled 5000 jobs at the same time but job is not running on time

[REQUEST] - trigger event emit when Agenda lockedAt

Hello Team,

We used multi worker for schedule jobs, We can not know that job locked by what worker. So, You can add a emit when the job begin locked and API to update worker name to job database?

Thank Team.

Execute something at a specific time of job

I wanted to know if there is any feature that allows me to identify when the job arrives at a specific time and run something after that, example:
i create a job that runs in 30 seconds, but when there are 5 seconds left (25s) it would run another task before the main one.
Or would i have to create two tasks?

(sorry if my english is bad)

It supports startDate and endDate?

Hey does this agenda supports start date and end date for a job ? Something like this:
Every 45 days at 8:00 AM CST starting 30-Aug-2016, run till 30-Aug-2019

agendash not working with this fork

Hi,

Thanks for forking this, it's solved some issues I was having with the original package.

One thing I can't get to work is agendash via Fastify which was working previously.

The error I get is about the collection being undefined.

(node:41832) UnhandledPromiseRejectionWarning: TypeError: Cannot read property 'collection' of undefined
    at getOverview (C:\ProtempoCoding\job.scraping.manager\node_modules\agendash\lib\controllers\agendash.js:125:43)
    at api (C:\ProtempoCoding\job.scraping.manager\node_modules\agendash\lib\controllers\agendash.js:262:7)
    at Object.<anonymous> (C:\ProtempoCoding\job.scraping.manager\node_modules\agendash\lib\middlewares\fastify.js:27:33)
    at preHandlerCallback (C:\ProtempoCoding\job.scraping.manager\node_modules\fastify\lib\handleRequest.js:126:28)
    at preValidationCallback (C:\ProtempoCoding\job.scraping.manager\node_modules\fastify\lib\handleRequest.js:109:5)
    at handler (C:\ProtempoCoding\job.scraping.manager\node_modules\fastify\lib\handleRequest.js:72:7)
    at handleRequest (C:\ProtempoCoding\job.scraping.manager\node_modules\fastify\lib\handleRequest.js:20:5)
    at runPreParsing (C:\ProtempoCoding\job.scraping.manager\node_modules\fastify\lib\route.js:451:5)
    at Object.routeHandler [as handler] (C:\ProtempoCoding\job.scraping.manager\node_modules\fastify\lib\route.js:409:7)
    at Router.lookup (C:\ProtempoCoding\job.scraping.manager\node_modules\find-my-way\index.js:378:14)
(node:41832) UnhandledPromiseRejectionWarning: Unhandled promise rejection. This error originated either by throwing inside of an async function without a catch block, or by rejecting a promise which was not handled with .catch(). To terminate the node process on unhandled promise rejection, use the CLI flag `--unhandled-rejections=strict` (see https://nodejs.org/api/cli.html#cli_unhandled_rejections_mode). (rejection id: 2)

It worked fine on the standard agenda setup. I'm thinking it's likely to do with the increased mongodb sdk version?

Here is some example code.

import { Agenda } from "@hokify/agenda";
import config from "config";

const Agendash = require("agendash");
const Fastify = require("fastify");

const mongoConnectionStringAgenda: string = config.get("agenda.db.uri");
const agenda = new Agenda({
  db: { address: mongoConnectionStringAgenda, collection: "agendaJobs" },
});
const fastify = new Fastify();
fastify.register(Agendash(agenda, { middleware: "fastify" }));


agenda.define(
  "start company jobs",
  async (job) => {
console.log(job)
  },
  { priority: 20 }
);

// Get jobs on first run
async function init() {
  // Start web server for monitoring
  await fastify.listen(3002);
  console.log("Listening on port 3002");
  // Start agenda process
  await agenda.start();

  console.log("Agenda started");
  await agenda.purge();
  // Start jobs
  agenda.now("start company jobs", {});

}
init();

Thanks

AgendaTS with other DB

Hey,

we are currently looking to adapt Agenda (TS) or Bree for us to use.
The bad news is that we have to use Oracle as a backend :(

I looked at your approach to to abstract the DB into JobDbRepository.ts
We now have two options: A) Roll our own db for Bree or use B) AgendaTS and open up JobDbRepository.ts and extend it with TypeORM for example.

Would you be interested in a pull request for this?
Do you have any guidelines or recommendations for doing so?

Cheers,
Gerwin

defaultConcurrency is working like maxConcurrency

Setting default concurrency prevents executing more jobs than the defaultConcurrency value set, in short defaultConcurrency is working like maxConcurrency. This observation is assuming that defaultConcurrency considers tasks with same name as a specific task.
Please help me understand if I am doing something wrong or is this a bug.

[FEATURE REQUEST] Graceful shutdown support

Hi.
What you've done in this fork is so awesome. Keep up the good work.

I think this fork doesn't support the stop method yet.
https://github.com/agenda/agenda#stop

Do you think adding it to this fork would require a lot of effort?

What is the Most Efficient Way to use Agenda in Production

I am using the NodeJS Agenda Scheduler and I wish to know a few things before going to production.

There are 3 scenarios

A User called Peter wants to schedule a Message to be sent by 5PM on the Second of July
2022
Another User called Mary wants to schedule a Message to be sent by 10PM on the 15th of
August 2022.

For every new request to schedule a new message to be sent in the future I am creating a new Job for it using the function below

defineNewJob(messageId, jobDate) {
    let jobName = `send_${messageId}`;
    try {
        agenda.define(jobName, jobHandler);
        await agenda.schedule(jobDate, jobName, { message_id: messageId, removeOnCompleted: true });
    } catch (e) {
        console.log(`Error Scheduling Job`);
    }
}

This means that I might end up with over 1 million messages from different users all to be scheduled in the future at completely different times and stored in the MongoDB database

I noticed that when I restart my server, Agenda has lost all I scheduled so I need to read back the data from my database and call the function above again with each of the returned messages from my database. Is this an efficient way to do things? I don't think so.

Please currently the scheduling works flawlessly at the specified dates and time. I'm only concerned about the efficiency of this in production with a lot of users scheduling all kinds of messages at different dates and times.

So, Is this fine in production or do I need to make changes?

Multiple Instances Generate Duplicate Events (Mongo Changestream)

New to agenda, but I want to use its job queue to update a materialized view when a collection is updated. Since our production app has multiple instances, I would like to utilize the locking mechanism to prevent duplication.

Using:

"@hokify/agenda": "^6.3.0",
"mongoose": "^7.0.2"

with Atlas v. 6.0.8.

Testing with local environment: macOS Ventura 13.4

Setting up Agenda instance:

export const agenda = new Agenda({
  name: 'agenda-v-' + process.env.PORT, 
  defaultConcurrency: 1,
  lockLimit: 1, 
  defaultLockLimit: 1, 
  defaultLockLifetime: 10000
})

export const init = async (io) => {
  try {
    const db = await connectToDb();
    
    db.once('open', async () => {
      console.log('Connected to MongoDB...');
      agenda.mongo(db, 'agendaJobs');

      agenda.define('test-lock', async (job) => {
        try {
          console.log('Job %s starting', job.attrs.name);
          const { attrs: {data: {name: eventId}} } = job; 
          job.unique({'data.name': eventId, 'name': job.attrs.name});
          await job.save();
          const testLock = await TestLock.create({name: eventId})
        } catch (error) {
          console.log('ERROR => ', error)
        }
      },  { lockLifetime: 1000, concurrency: 1, lockLimit: 1 });

      agenda
        .on("ready", () => console.log("Agenda started!"))
        .on("error", () => console.log("Agenda [iTerm2 Session Jul 18, 2023 at 10:56:41 AM.txt](https://github.com/hokify/agenda/files/12081986/iTerm2.Session.Jul.18.2023.at.10.56.41.AM.txt) error!"));
    })

When the change stream emits an event, I am creating a job for test-lock to be processed immediately.

// inside change handler
agenda.start(); 
agenda.now('test-lock', {name: incoming._id._data});

Documents from MongoDB

{"_id":{"$oid":"64b6a9e46ac9497b3e2d6629"},"name":"test-lock","data":{"name":"8264B6A9E3000000022B022C0100296E5A10049783BCD202DC47179E6885FE7FDBBF5F46645F69640064620EA7EE92B6F86A163EAC410004"},"type":"normal","priority":{"$numberInt":"0"},"nextRunAt":null,"lastModifiedBy":"agenda-v-3006","lockedAt":null,"failCount":null,"failReason":null,"failedAt":null,"lastFinishedAt":{"$date":{"$numberLong":"1689692644667"}},"lastRunAt":{"$date":{"$numberLong":"1689692644406"}},"progress":null}

{"_id":{"$oid":"64b6a9e4adcf584a52f92410"},"name":"test-lock","data":{"name":"8264B6A9E3000000022B022C0100296E5A10049783BCD202DC47179E6885FE7FDBBF5F46645F69640064620EA7EE92B6F86A163EAC410004"},"type":"normal","priority":{"$numberInt":"0"},"nextRunAt":null,"lastModifiedBy":"agenda-v-3005","lockedAt":null,"failCount":null,"failReason":null,"failedAt":null,"lastFinishedAt":{"$date":{"$numberLong":"1689692644667"}},"lastRunAt":{"$date":{"$numberLong":"1689692644406"}},"progress":null}

Attaching logs for both instances below.

I'm not sure if this is a bug or if I am misunderstanding the documentation.

instance1.txt
instance2.txt

Error using with CosmosDB (MongoAPI)

I encountered this error while running with CosmosDB, in the original repo package i didn't got this error, but sometimes jobs stop getting processed mysteriously, and i don't know why. Then i stumbled upon this fork i thought it would work better, but instead i got this error. Please advise

MongoServerError: Message: {"Errors":["Encountered exception while executing function. Exception = Error: {\"Errors\":[\"The order by query does not have a corresponding composite index that it can be served from.\"]}\r\nStack trace: Error: {\"Errors\":[\"The order

Can I schedule up to 20 Jobs at once

Thank you for porting this library.

I had several issues with the original agenda library so I switched to this as recommended by you in one of your comments.

I have a very important question though which is, Can I schedule up to 20 different jobs at a time?

I tried more than 20 in the original library and none of them ran. And this happened in production. I was so frustrated as customers were raining curses and insults on me already. They were extremely angry.

So, here is what I mean,

Imagine I have 20 different dates as follows:


let dates = [new Date("someDate"),...] //Some random dates up to 20 dates in the array;

for(let date of dates){
         let jobName = generateUniqueJobName();
         agenda.define(jobName, module.exports.jobHandler);
         agenda.schedule(date, jobName, {params: someRandomParams, removeOnCompleted: true });
}

Will the above code work? For now I'm scheduling just 10 at a time, but I really need to get up to 20 at once.

Is this possible?

Thank you

Missed agenda drain method.

There is drain method in agenda version 4.4 but seems it's not migrated here. Any specific reason?

UnhandledPromiseRejectionWarning on Job Timeout

First of all, thank you for forking Agenda and keeping the project alive! There are a lot of issues with locking which seem to be fixed now.

When a Job runs longer than the configured LockLifeTime without returning a Promise or calling the done handler, it runs in a timeout. In the original Agenda this was a big issue because the job was still in the local locked cache and was never cleared. In this fork an exception is thrown and the job is cleared from the internal lock cache so it can be retried again. While this is great, the thrown exception cannot be handled. The job itself does not fail, so the listener on fail is never triggered. I tried a wrapper function around the fn function of the job, which had an additional try...catch just to be sure, but it was not caught there either.

It would be great if there was a listener for timeout jobs, like the one for failed jobs. Or the job could fail like it would if there was an error. This way it could be handled and would also work with a retry like a failed job would.

Thanks!

Jobs not running definition.fn is not a function

Jobs are not running can anyone please help me with the problem is coming for both agenda and @hokify/agenda package in my Nodejs project i am using mongo atlas using the latest version of the package i can only see this fail reason in my DB "definition.fn is not a function"

import {Agenda} from '@hokify/agenda';
import { MongoDBConnection } from './db.js';
const connection = await MongoDBConnection();
const agenda = new Agenda({ mongo: connection });

const ScheduleUserExpiration = async (DateTime, email) => {
    // DateTime is a Time like const DateTime = new Date() + 1 day etc...
    const expirationTime = await GetTime(DateTime);

    const jobNameTemplate = 'ScheduleUserExpiration-{email}';

    const jobName = jobNameTemplate.replace('{email}', email);
    agenda.define(jobName, { concurrency: 1 }, async (job) => {
        await NotifeeUser(email);
    });

    const jobOptions = {
        maxAttempts: 3,
        backoff: '1s',
    };

    (async function () {
        await agenda.start();
        await agenda.schedule(expirationTime, jobName, jobOptions);
    })();
};

Support for `shouldSaveResult`

The upstream version of agenda has a cool feature where it automatically saves the result you return from the job in the database. Not sure how difficult it is to support, but would be nice to have.

Jobs are not running sometimes

Thanks for porting this amazing library.

Please,I'm having a really serious issue in production.

Which is that sometimes Agenda doesn't run the Jobs and some other times it runs. This is really embarrassing as my users are so so angry already.

So, my question is at any random point in time is there a function I can call on agenda to know if it is still running or it has stopped?

Any ideas would be really appreciated.

An error occurred: done is not a function

It was working beautifully...FOR MONTHS.....all of a sudden everything is broken.
I'm getting An error occurred: done is not a function.
It also crashes server.
Anyone encounter this issue??
Where even to begin??

@hokify/agenda ^6.3.0
nodejs v18.13.0

[SANDBOX] Example

Hi everyone, can you provide a better example https://github.com/hokify/agenda?tab=readme-ov-file#sandboxed-worker---use-child-processes as this one tbh is pretty difficult to understand on how to connect things all together, and I've been struggling with it, no example found

`success` events emitted for failed job

documentation states that success and success:job name events will be emitted only when a job succeeds, but if the job is failed manually by calling job.fail(err).save(); the messages will be emitted anyway.

const agenda = new Agenda();

const handler = (job) => job.fail('error').save();
agenda.define('test', handler);

agenda.on('success', () => console.log('success'));

await agenda.start();
await agenda.now('test');

// will print 'success' even if the job has failed

link to code

Infinite loop job execution

Im trying to use agenda with a simple job to repeat every minute but the job keeps starting over and over again, mI doing something wrong or it may be a bug?
This is my setup

mongodb
processEvery: 1 * 60 * 1000,

{
    "_id": {
        "$oid": "61ae514a962be964c45c8ef0"
    },
    "name": "test-job",
    "priority": 0,
    "type": "normal",
    "nextRunAt": {
        "$date": "2021-12-06T18:19:44.000Z"
    },
    "repeatInterval": "* */1 * * * *",
    "startDate": null,
    "endDate": null,
    "skipDays": null,
    "disabled": false,
}

job processor

this.agenda.define('test-job', async (job) => {
        const testJob = new TestJob();

        await testJob.execute();

        if (job.attrs?.data?.removeJobAfterRun) {
          await job.remove();
        }
      });

job function

export class TestJob {
  public async execute(): Promise<void> {
    console.log('--- TEST JOB START ---');

    await new Promise((r) => {
      setTimeout(r, 5000);
    });

    console.log('--- TEST JOB END ---');
  }
}

LOG
when job finishs the execution it starts immediately again

  agenda:jobProcessor:findAndLockNextJob found a job available to lock, creating a new job on Agenda with id [61ae514a962be964c45c8ef0] +0ms
  agenda:jobProcessor:shouldLock job [test-job] lock status: shouldLock = true undefined >= 0 0 >= 0 +0ms
  agenda:jobProcessor:jobQueueFilling [test-job:61ae514a962be964c45c8ef0] job locked while filling queue +0ms
  agenda:jobProcessor:shouldLock job [test-job] lock status: shouldLock = true 1 >= 0 1 >= 0 +0ms
  agenda:jobProcessor:findAndLockNextJob looking for lockable jobs for test-job (lock dead line = Mon Dec 06 2021 15:09:24 GMT-0300 ) +0ms     
  agenda:jobProcessor:jobProcessing [test-job:61ae514a962be964c45c8ef0] there is a job to process (priority = 0) undefined +0ms
  agenda:jobProcessor:jobProcessing [test-job:61ae514a962be964c45c8ef0] nextRunAt is in the past, run the job immediately +0ms
  agenda:jobProcessor:runOrRetry [test-job:61ae514a962be964c45c8ef0] processing job +0ms
  agenda:job [test-job:61ae514a962be964c45c8ef0] setting lastRunAt to: 2021-12-06T18:19:26.534Z +0ms
  agenda:nextRunAt [test-job:61ae514a962be964c45c8ef0] computing next run via interval [* */1 * * * *] +0ms
  agenda:job [test-job:61ae514a962be964c45c8ef0] nextRunAt set to [2021-12-06T18:19:27.000Z] +21ms
  agenda:db attempting to save a job +3s
  agenda:db [job 61ae514a962be964c45c8ef0] set job props:
  agenda:db {
  agenda:db   name: 'test-job',
  agenda:db   priority: 0,
  agenda:db   type: 'normal',
  agenda:db   nextRunAt: 2021-12-06T18:19:27.000Z,
  agenda:db   repeatInterval: '* */1 * * * *',
  agenda:db   startDate: null,
  agenda:db   endDate: null,
  agenda:db   skipDays: null,
  agenda:db   disabled: false,
  agenda:db   lockedAt: 2021-12-06T18:19:24.540Z,
  agenda:db   lastRunAt: 2021-12-06T18:19:26.534Z,
  agenda:db   lastModifiedBy: ''
  agenda:db } +0ms
  agenda:db current time stored as 2021-12-06T18:19:26.558Z +2ms
  agenda:db job already has _id, calling findOneAndUpdate() using _id as query +0ms
  agenda:db processDbResult() called with success, checking whether to process job immediately or not +188ms
  agenda:jobProcessor:process [test-job:+0ms] job would have ran by nextScanAt, processing the job immediately
  agenda:jobProcessor:shouldLock job [test-job] lock status: shouldLock = true 1 >= 0 1 >= 0 +0ms
  agenda:job [test-job:61ae514a962be964c45c8ef0] starting job +194ms
  agenda:job [test-job:61ae514a962be964c45c8ef0] process function being called +0ms
--- TEST JOB START ---
  agenda:jobProcessor:lockOnTheFly cannot lock job [test-job] on the fly +0ms
  agenda:jobProcessor:lockOnTheFly no jobs to current lock on the fly, returning +0ms
--- TEST JOB END ---
  agenda:job [test-job:61ae514a962be964c45c8ef0] has succeeded +5s
  agenda:db attempting to save a job +5s
  agenda:db [job 61ae514a962be964c45c8ef0] set job props:
  agenda:db {
  agenda:db   name: 'test-job',
  agenda:db   priority: 0,
  agenda:db   type: 'normal',
  agenda:db   nextRunAt: 2021-12-06T18:19:27.000Z,
  agenda:db   repeatInterval: '* */1 * * * *',
  agenda:db   startDate: null,
  agenda:db   endDate: null,
  agenda:db   skipDays: null,
  agenda:db   disabled: false,
  agenda:db   lockedAt: undefined,
  agenda:db   lastRunAt: 2021-12-06T18:19:26.534Z,
  agenda:db   lastFinishedAt: 2021-12-06T18:19:31.752Z,
  agenda:db   lastModifiedBy: ''
  agenda:db } +1ms
  agenda:db current time stored as 2021-12-06T18:19:31.754Z +1ms
  agenda:db job already has _id, calling findOneAndUpdate() using _id as query +0ms
  agenda:db processDbResult() called with success, checking whether to process job immediately or not +188ms
  agenda:jobProcessor:process [test-job:+0ms] job would have ran by nextScanAt, processing the job immediately
  agenda:jobProcessor:shouldLock job [test-job] lock status: shouldLock = true 1 >= 0 1 >= 0 +0ms
  agenda:job [test-job:61ae514a962be964c45c8ef0] was saved successfully to MongoDB +191ms
  agenda:job [test-job:61ae514a962be964c45c8ef0] job finished at [2021-12-06T18:19:31.752Z] and was unlocked +1ms
  agenda:jobProcessor:runOrRetry [test-job:61ae514a962be964c45c8ef0] processing job successfull +0ms
  agenda:jobProcessor:shouldLock job [test-job] lock status: shouldLock = true 0 >= 0 0 >= 0 +0ms
  agenda:jobProcessor:lockOnTheFly found job [test-job:61ae514a962be964c45c8ef0] that can be locked on the fly +0ms
  agenda:jobProcessor:jobProcessing [test-job:61ae514a962be964c45c8ef0] there is a job to process (priority = 0) undefined +0ms
  agenda:jobProcessor:jobProcessing [test-job:61ae514a962be964c45c8ef0] nextRunAt is in the past, run the job immediately +0ms
  agenda:jobProcessor:runOrRetry [test-job:61ae514a962be964c45c8ef0] processing job +0ms
  agenda:job [test-job:61ae514a962be964c45c8ef0] setting lastRunAt to: 2021-12-06T18:19:32.131Z +187ms
  agenda:nextRunAt [test-job:61ae514a962be964c45c8ef0] computing next run via interval [* */1 * * * *] +6s
  agenda:job [test-job:61ae514a962be964c45c8ef0] nextRunAt set to [2021-12-06T18:19:33.000Z] +2ms
  agenda:jobProcessor:lockOnTheFly no jobs to current lock on the fly, returning +0ms
  agenda:db attempting to save a job +192ms
  agenda:db [job 61ae514a962be964c45c8ef0] set job props:
  agenda:db {
  agenda:db   name: 'test-job',
  agenda:db   priority: 0,
  agenda:db   type: 'normal',
  agenda:db   nextRunAt: 2021-12-06T18:19:33.000Z,
  agenda:db   repeatInterval: '* */1 * * * *',
  agenda:db   startDate: null,
  agenda:db   endDate: null,
  agenda:db   skipDays: null,
  agenda:db   disabled: false,
  agenda:db   lockedAt: 2021-12-06T18:19:31.943Z,
  agenda:db   lastModifiedBy: '',
  agenda:db   lastRunAt: 2021-12-06T18:19:32.131Z,
  agenda:db   lastFinishedAt: 2021-12-06T18:19:31.752Z
  agenda:db } +0ms
  agenda:db current time stored as 2021-12-06T18:19:32.135Z +1ms
  agenda:db job already has _id, calling findOneAndUpdate() using _id as query +1ms
  agenda:db processDbResult() called with success, checking whether to process job immediately or not +186ms
  agenda:jobProcessor:process [test-job:+0ms] job would have ran by nextScanAt, processing the job immediately
  agenda:jobProcessor:shouldLock job [test-job] lock status: shouldLock = true 1 >= 0 1 >= 0 +0ms
  agenda:job [test-job:61ae514a962be964c45c8ef0] starting job +190ms
  agenda:job [test-job:61ae514a962be964c45c8ef0] process function being called +1ms
--- TEST JOB START ---
  agenda:jobProcessor:lockOnTheFly cannot lock job [test-job] on the fly +0ms
  agenda:jobProcessor:lockOnTheFly no jobs to current lock on the fly, returning +0ms
--- TEST JOB END ---
  agenda:job [test-job:61ae514a962be964c45c8ef0] has succeeded +5s
  agenda:db attempting to save a job +5s
  agenda:db [job 61ae514a962be964c45c8ef0] set job props:
  agenda:db {
  agenda:db   name: 'test-job',
  agenda:db   priority: 0,
  agenda:db   type: 'normal',
  agenda:db   nextRunAt: 2021-12-06T18:19:33.000Z,
  agenda:db   repeatInterval: '* */1 * * * *',
  agenda:db   startDate: null,
  agenda:db   endDate: null,
  agenda:db   skipDays: null,
  agenda:db   disabled: false,
  agenda:db   lockedAt: undefined,
  agenda:db   lastModifiedBy: '',
  agenda:db   lastRunAt: 2021-12-06T18:19:32.131Z,
  agenda:db   lastFinishedAt: 2021-12-06T18:19:37.332Z
  agenda:db } +0ms
  agenda:db current time stored as 2021-12-06T18:19:37.334Z +2ms
  agenda:db job already has _id, calling findOneAndUpdate() using _id as query +0ms
  agenda:db processDbResult() called with success, checking whether to process job immediately or not +186ms
  agenda:jobProcessor:process [test-job:+0ms] job would have ran by nextScanAt, processing the job immediately
  agenda:jobProcessor:shouldLock job [test-job] lock status: shouldLock = true 1 >= 0 1 >= 0 +0ms
  agenda:job [test-job:61ae514a962be964c45c8ef0] was saved successfully to MongoDB +189ms
  agenda:job [test-job:61ae514a962be964c45c8ef0] job finished at [2021-12-06T18:19:37.332Z] and was unlocked +1ms
  agenda:jobProcessor:runOrRetry [test-job:61ae514a962be964c45c8ef0] processing job successfull +0ms
  agenda:jobProcessor:shouldLock job [test-job] lock status: shouldLock = true 0 >= 0 0 >= 0 +0ms
  agenda:jobProcessor:lockOnTheFly found job [test-job:61ae514a962be964c45c8ef0] that can be locked on the fly +0ms
  agenda:jobProcessor:jobProcessing [test-job:61ae514a962be964c45c8ef0] there is a job to process (priority = 0) undefined +0ms
  agenda:jobProcessor:jobProcessing [test-job:61ae514a962be964c45c8ef0] nextRunAt is in the past, run the job immediately +0ms
  agenda:jobProcessor:runOrRetry [test-job:61ae514a962be964c45c8ef0] processing job +0ms
  agenda:job [test-job:61ae514a962be964c45c8ef0] setting lastRunAt to: 2021-12-06T18:19:37.710Z +188ms
  agenda:nextRunAt [test-job:61ae514a962be964c45c8ef0] computing next run via interval [* */1 * * * *] +6s
  agenda:job [test-job:61ae514a962be964c45c8ef0] nextRunAt set to [2021-12-06T18:19:38.000Z] +2ms
  agenda:jobProcessor:lockOnTheFly no jobs to current lock on the fly, returning +0ms
  agenda:db attempting to save a job +192ms
  agenda:db [job 61ae514a962be964c45c8ef0] set job props:
  agenda:db {
  agenda:db   name: 'test-job',
  agenda:db   priority: 0,
  agenda:db   type: 'normal',
  agenda:db   nextRunAt: 2021-12-06T18:19:38.000Z,
  agenda:db   repeatInterval: '* */1 * * * *',
  agenda:db   startDate: null,
  agenda:db   endDate: null,
  agenda:db   skipDays: null,
  agenda:db   disabled: false,
  agenda:db   lockedAt: 2021-12-06T18:19:37.521Z,
  agenda:db   lastModifiedBy: '',
  agenda:db   lastRunAt: 2021-12-06T18:19:37.710Z,
  agenda:db   lastFinishedAt: 2021-12-06T18:19:37.332Z
  agenda:db } +0ms
  agenda:db current time stored as 2021-12-06T18:19:37.714Z +2ms
  agenda:db job already has _id, calling findOneAndUpdate() using _id as query +0ms
  agenda:db processDbResult() called with success, checking whether to process job immediately or not +186ms
  agenda:jobProcessor:process [test-job:+0ms] job would have ran by nextScanAt, processing the job immediately
  agenda:jobProcessor:shouldLock job [test-job] lock status: shouldLock = true 1 >= 0 1 >= 0 +0ms
[SCHEDULER] Job test-job starting
2021-12-06T18:19:37.901Z [WEBSCRAP] INFO: Job test-job starting
  agenda:job [test-job:61ae514a962be964c45c8ef0] starting job +189ms
  agenda:job [test-job:61ae514a962be964c45c8ef0] process function being called +1ms
--- TEST JOB START ---
  agenda:jobProcessor:lockOnTheFly cannot lock job [test-job] on the fly +0ms
  agenda:jobProcessor:lockOnTheFly no jobs to current lock on the fly, returning +0ms
--- TEST JOB END ---
  agenda:job [test-job:61ae514a962be964c45c8ef0] has succeeded +5s
  agenda:db attempting to save a job +5s
  agenda:db [job 61ae514a962be964c45c8ef0] set job props:
  agenda:db {
  agenda:db   name: 'test-job',
  agenda:db   priority: 0,
  agenda:db   type: 'normal',
  agenda:db   nextRunAt: 2021-12-06T18:19:38.000Z,
  agenda:db   repeatInterval: '* */1 * * * *',
  agenda:db   startDate: null,
  agenda:db   endDate: null,
  agenda:db   skipDays: null,
  agenda:db   disabled: false,
  agenda:db   lockedAt: undefined,
  agenda:db   lastModifiedBy: '',
  agenda:db   lastRunAt: 2021-12-06T18:19:37.710Z,
  agenda:db   lastFinishedAt: 2021-12-06T18:19:42.914Z
  agenda:db } +0ms
  agenda:db current time stored as 2021-12-06T18:19:42.916Z +2ms
  agenda:db job already has _id, calling findOneAndUpdate() using _id as query +0ms
  agenda:db processDbResult() called with success, checking whether to process job immediately or not +189ms
  agenda:jobProcessor:process [test-job:+0ms] job would have ran by nextScanAt, processing the job immediately
  agenda:jobProcessor:shouldLock job [test-job] lock status: shouldLock = true 1 >= 0 1 >= 0 +0ms
  agenda:job [test-job:61ae514a962be964c45c8ef0] was saved successfully to MongoDB +192ms

Concurrency not working properly

Hi,
I have defined a job with concurrency: 1.

agenda.define(
    "job_1",
    job1Processor,
    { priority: 10, concurrency: 1, lockLifetime: minutesToMs(100) }
);

But it can happened that agenda is processing more than 1 job at the same time.
Do anyone has the same problem ?

job dynamic define

I use pm2 start 4 process, and I create job by http request, and I define job when create job, but this job only define the current process. when job start, if another process get this job lock, but don't define job, the job will execute fail。

So I want to know is job can dynamic define?

I want to listen start event, then I define job , but I find origin code get definition before emit start event, So if there hava a prefer way？

 this.on('start', async (job: any) => {
            if(!this.definitions[job.attrs.name]){
                this.define(job.attrs.name, this.jobHandler); 
            }
}

       const definition = this.agenda.definitions[this.attrs.name];
        this.attrs.lastRunAt = new Date();
        log('[%s:%s] setting lastRunAt to: %s', this.attrs.name, this.attrs._id, this.attrs.lastRunAt.toISOString());
        this.computeNextRunAt();
        await this.save();
        try {
            this.agenda.emit('start', this);
            this.agenda.emit(`start:${this.attrs.name}`, this);
            log('[%s:%s] starting job', this.attrs.name, this.attrs._id);
            if (!definition) {
                log('[%s:%s] has no definition, can not run', this.attrs.name, this.attrs._id);
                throw new Error('Undefined job');
            }

As above， If the definition can declaration after this.agenda.emit('start', this) ?

Dynamic Job name

Hello.
First of all, congrats for the library.
Is there a chance to set the job name in a dynamic way?
I mean, the job behaviour is the same, so I can put the definition in a method that I can invoke when starting the node principal process.
But the name has to be different because I have to execute that job multiple times, one for each message I retrieve from a data source.
For example, say I read a table, each record has an identifier (generated by a third party system), I need to start (and schedule) the JOB for each record (according to some conditions). The job name CANNOT be the same otherwise the record created in mongodb is always the same. Job type should be single because the node process can fail and can be restarted (pod).

This is why I am asking if it's possible to set the job name dinamically.
Does it make sense?

Thanks a lot
Alessandro