redislabs / redis-quartz

A Quartz Scheduler JobStore that uses Redis for persistent storage.

RedisJobStore

A Quartz Scheduler JobStore that uses Redis for persistent storage.

Configuration

To get Quartz up and running quickly with RedisJobStore, use the following example to configure your quartz.properties file:

# setting the scheduler's misfire threshold, in milliseconds
org.quartz.jobStore.misfireThreshold: 60000

# setting the scheduler's JobStore to RedisJobStore
org.quartz.jobStore.class: com.redislabs.quartz.RedisJobStore

# setting your redis host
org.quartz.jobStore.host: <your_redis_host>

# setting your redis port
org.quartz.jobStore.port: <your_redis_port>

# setting your redis password (optional)
org.quartz.jobStore.password: <your_redis_password>

# setting a 'releaseTriggersInterval' will trigger a mechanism for releasing triggers of non-alive schedulers in a given interval, in milliseconds
org.quartz.jobStore.releaseTriggersInterval: 600000

# setting a 'instanceIdFilePath' will release triggers of previous schedulers on startup
org.quartz.jobStore.instanceIdFilePath: /etc/quartz

External Libraries

RedisJobStore uses the jedis, gson and jedis-lock libraries, so you'll have to download them and add them to your project's classpath or define the relevant Maven dependencies:

<dependency>
	<groupId>redis.clients</groupId>
	<artifactId>jedis</artifactId>
	<version>2.0.0</version>
</dependency>

<dependency>
 	<groupId>com.google.code.gson</groupId>
 	<artifactId>gson</artifactId>
 	<version>2.2.4</version>
</dependency>

<dependency>
  <groupId>com.github.jedis-lock</groupId>
  <artifactId>jedis-lock</artifactId>
  <version>1.0.0</version>
</dependency>

Limitations

RedisJobStore attempts to be fully compliant with all of Quartz's features, but currently has some limitations that you should be aware of:

• Only SimpleTrigger and CronTriggerare supported.
• For any GroupMatcher, only a StringOperatorName.EQUALS operator is supported. You should note that if your scheduler is designed to compare any group of jobs, triggers, etc. with a pattern-based matcher.
• RedisJobStore is designed to use multiple schedulers, but it is not making any use of the org.quartz.scheduler.instanceName. The only limitation here is that you should maintain the uniquness of your trigger_group_name:trigger_name, and your job_group_name:job_name and you'll be good to go with multiple schedulers.
• A Scheduler should be started once on a machine, also to ensure releasing locked triggers of previously crashed schedulers.
• Data atomicity- RedisJobStore is not using any transaction-like mechanism, but ensures synchronization with global lockings. As a result, if a connection issue occurs during an operation, it might be partially completed.
• JobDataMap values are stored and returned as Strings, so you should implement your jobs accordingly.
• RedisJobStore is firing triggers only by their fire time, without any cosideration to their priorities at all.

Known Issues

1. Quartz's standard JobStores are sometimes considering triggers without a next fire time as tirggers in a WAITING state. As RedisJobStore is using redis Sorted Sets to maintain triggers states, using their next fire time as the score, it will consider these triggers as stateless.

Redis Schema

To better understand the workflow and the behavior of a Quartz Scheduler using a RedisJobStore, you may want to review the redis schema in which the RedisJobStore is making a use of at: /schema/schema.txt

License

The MIT License

redis-quartz's Issues

Is redis-quartz using Pub/sub mechanism to implement global locking?

As per the blog: https://redislabs.com/blog/run-quartz-the-java-job-scheduler-on-top-of-redis/, redis-quartz should be using pub/sub mechanism to implement global locking. Instead I see the library using JedisLock to implement this (which is in-turn using setNX and timeout for locking).

An additional element used in Redis JobStore is global locking with Redis' publish and subscribe mechanism (Pub/Sub), in which JobStore listens to an ‘unlock’ message.

I also see a class UnlockListener is defined in the RedisJobStore but is not being used anywhere.

GPL and Java

By the standard interpretation of the GPL with regard to Java libraries, I can't use this data store in a non-GPL application: https://www.gnu.org/licenses/lgpl-java.html

I would love to use this job store, but my application is not GPL-licensed and will not become GPL-licensed.

Next Fire Time is null

First of all thanks for this sample, i used as a reference and modified a bit, trying to make it as simple as possible, in my case triggers are stored as Hash keys as "trigger:grpName:cronTriggerTest"

hence in "acquireNextTriggers" this is what i do:

for (String triggerHashKey : stringRedisConnection.keys("trigger:grpName:cronTriggerTest") ) {

// do retrievetrigger ...

but this trigger always has next fire time as NULL and hence triggers never gets fired as we have a check here that:
if ((trigger != null) && trigger.getNextFireTime() == null) {
/** Trigger has no next fire time. Unset its WAITING state. **/
continue;
}
}

Note: i am also NOT using any trigger state logic here.

issue when job name and group name contains ':' char

i have a job name as " org:1010:user:7529bd47-7f0e-4be3-8552-2fa5b3acc35bdata", group name as "org:1010",

with this combo, the jobstore throwing this exception inside while(retry).
java.lang.NullPointerException
at com.redislabs.quartz.RedisJobStore.applyMisfire(RedisJobStore.java:2103)
at com.redislabs.quartz.RedisJobStore.acquireNextTriggers(RedisJobStore.java:1652)
at org.quartz.core.QuartzSchedulerThread.run(QuartzSchedulerThread.java:272)

Support for Redis on Cloud Foundry ?

Hi

Does it support an instance of Redis running on Cloud Foundry where we only have a service name and cannot rely on the Host and Port as they can change ?

thanks

Redis 3.x Clustering Support?

Are there plans to add Redis 3.x clustering support?

Some Issues with the Implementation

Here are some of the problems I've experienced with this implementation:

• faulty Redis lock implementation. Unexpected shutdown may cause it hang forever on next restart.
• Unexpected shutdown when a repeating is acquired. The trigger will neve fire again on the next restart.
• Trigger JobDataMap do not persist at all. According to Quartz docs they trigger dta map is supposed to be merged with job data map when trigger value will override the job values.

adding existing triggers back to scheduler after restart

Thanks so much for open souring the redis JobStore. i am planning to use this for a project, i did little bit of testing and realized that on scheduler restart, the existing triggers are not getting added back to scheduler. looks like we need something like org.quartz.impl.jdbcjobstore.JobStoreSupport.MisfireHandler for redis job store.

Is this project still maintained?

Is this project still maintained? If not what is the recommended way to integrate redis with quartz?
Thanks.

Why does setTriggerState/unsetTriggerState get new connection?

I'm interested to know why these methods always get a new Jedis instance from the pool, even when the calling method already has an instance. Is there a particular reason not to use the same one?

JobDataMap does not persist when associated with triggers

HI, I am using redisStore for Quartz.
I see that if I pass data using JobDetail.JobDataMap.put("key","value") I can access it when my job fires doing context.getJobDetail().getJObDataMap()

But if i try to pass JobDataMap via the trigger that i used to trigger the job, trigger1.getJobDataMap().put("key","value"), then i cannot access the value when my job fires bu doing context.getTrigger().getJobDataMap().getString("key");

the above works if i use RAMStore instead of RedisStore.

Please help resolve this issue. it is urgent.
Please let me know if i am missing something here.

Thanks.

Issue with threads being blocked.

Hello,

I'm running into an issue where the RedisJobStore blocks for long periods of time (>1 minute) during execution of storeJobAndTrigger(...). It seems to be when we are acquiring the JedisLock.

I've created a small test that recreates the issue, and I've included the code below. Hopefully it should be pretty clear.

Test code:

public class TestHanging extends TestEnv {
    private static final Logger LOGGER = Logger.getLogger(TestJedisHanging.class);

    private static JedisPoolConfig config = new JedisPoolConfig();
    private static JedisPool pool = new JedisPool(config, "localhost", 6381, Protocol.DEFAULT_TIMEOUT);
    private static JedisLock lockPool = new JedisLock(pool.getResource(), "TestLock", 10 * 60 * 1000);

    @Test
    public void testSendScheduledMessagesBig() throws InterruptedException {

        TestHelper.threadTesting(5, () -> {
            for (int i = 0; i < 200; i++) {
                doSomething();
            }
        });
    }

    public void doSomething() throws InterruptedException {
        try (Jedis jedis = pool.getResource()) {
            lockPool.acquire();
            LOGGER.fatal("HI");
        } finally {
            LOGGER.fatal("BYE");
            lockPool.release();
        }
    }
}

Some log output:

2018-01-21 22:22:43 FATAL TestJedisHanging:33 - HI
2018-01-21 22:22:43 FATAL TestJedisHanging:35 - BYE
2018-01-21 22:23:43 FATAL TestJedisHanging:33 - HI
2018-01-21 22:23:43 FATAL TestJedisHanging:35 - BYE
2018-01-21 22:24:43 FATAL TestJedisHanging:33 - HI
2018-01-21 22:24:43 FATAL TestJedisHanging:35 - BYE
2018-01-21 22:25:43 FATAL TestJedisHanging:33 - HI
2018-01-21 22:25:43 FATAL TestJedisHanging:35 - BYE
2018-01-21 22:26:43 FATAL TestJedisHanging:33 - HI
2018-01-21 22:26:43 FATAL TestJedisHanging:35 - BYE
2018-01-21 22:26:43 FATAL TestJedisHanging:33 - HI
2018-01-21 22:26:43 FATAL TestJedisHanging:33 - HI
2018-01-21 22:26:43 FATAL TestJedisHanging:35 - BYE
2018-01-21 22:26:43 FATAL TestJedisHanging:35 - BYE
2018-01-21 22:27:43 FATAL TestJedisHanging:33 - HI
2018-01-21 22:27:43 FATAL TestJedisHanging:33 - HI
2018-01-21 22:27:43 FATAL TestJedisHanging:35 - BYE
2018-01-21 22:27:43 FATAL TestJedisHanging:35 - BYE
2018-01-21 22:28:43 FATAL TestJedisHanging:33 - HI
2018-01-21 22:28:43 FATAL TestJedisHanging:35 - BYE

In the test, I have 5 threads that run the simple function of acquiring the JedisLock, writing some log output, then releasing it.

I've observed that execution gets blocked for 1 minute intervals, before my lock/unlock function runs for a seemingly random number of times.

Do you have any idea why this might be happening?

Finally block does not release jedis resources back to the pool

Using try with resources just closes the resource defined in the try part and does not return it back to the pool. This leads to pool.getResource() to timeout once the pools connections are exhausted.

try (Jedis jedis = pool.getResource()) { lockPool.acquire(); // code } finally { lockPool.release(); }

jobs not getting triggered at correct intervals

HI
I have 2 jobs with separate triggers,
trigger : group1.triggerA For Job key refreshgroup1.refreshjobA will run at: Tue Oct 18 15:23:12 IST 2016 and repeat: 10 times, every 10 seconds
trigger : group2.triggerB For Job key refreshgroup1.refreshjobB will run at: Tue Oct 18 15:23:14 IST 2016 and repeat: 10 times, every 7 seconds

But in the output i see that Job A doesnot repeat after 10 sec, Sometime the interval is 20 sec, sometimes it is 14 sec

Same problem for Job B. the interval is not 7 sec as it is configured to be. What am i missing here?

Thanks for your help.

------- Starting Scheduler ----------------
------- Started Scheduler -----------------
------- Waiting 90 seconds... -------------
RefreshJob: execute called Tue Oct 18 15:23:19 IST 2016 : for A
RefreshJob: execute called Tue Oct 18 15:23:26 IST 2016 : for B
RefreshJob: execute called Tue Oct 18 15:23:32 IST 2016 : for B
RefreshJob: execute called Tue Oct 18 15:23:39 IST 2016 : for A
RefreshJob: execute called Tue Oct 18 15:23:45 IST 2016 : for B
RefreshJob: execute called Tue Oct 18 15:23:52 IST 2016 : for A
RefreshJob: execute called Tue Oct 18 15:23:59 IST 2016 : for B
RefreshJob: execute called Tue Oct 18 15:24:06 IST 2016 : for A
RefreshJob: execute called Tue Oct 18 15:24:14 IST 2016 : for B
RefreshJob: execute called Tue Oct 18 15:24:21 IST 2016 : for B
RefreshJob: execute called Tue Oct 18 15:24:28 IST 2016 : for A
RefreshJob: execute called Tue Oct 18 15:24:35 IST 2016 : for B
RefreshJob: execute called Tue Oct 18 15:24:42 IST 2016 : for A