ConsulLockHandler local job locking about bootique-job

[atomashpolskiy]

I can't reproduce the issue. I also verified that a new session is created for each lock handler regardless of whether it shares the same JVM process with other handlers. Session creation is performed once per Bootique runtime (io.bootique.job.consul.ConsulSession#getOrCreateSession) and boils down to REST session/create call to Consul agent, which fills the node ID parameter in the request and forwards the call to the server [1]. Upon receiving session creation requests Consul server always creates a new session with a random ID [2]. This unique ID is then used by our lock handler to lock a key-value node in Consul (also implemented as a REST call, which ends up in Consul's transactional key-value store [3]).

I can't see any reason for the behavior that you observe other than the two jobs being run with a signficant time gap:

scheduler_1.runOnce(new LockJob(), callsCount);
scheduler_2.runOnce(new LockJob(), callsCount);

But this is super-strange, given that a thread-pool executor with multiple worker threads is used in your test. Are you sure that you can reproduce the issue?

[stariy95]

Test failure (from PR #70) is reproducible on my machine too (MacOS).
@atomashpolskiy can you please review test itself, is it correct?

Here is log:

[main] INFO 🐳 [consul:latest] - Container consul:latest started
[main] INFO org.springframework.scheduling.concurrent.ThreadPoolTaskScheduler - Initializing ExecutorService 
[bootique-job-1] INFO io.bootique.job.consul.lock.ConsulLockHandler - Attempting to lock 'TestJobs/lock'
[bootique-job-2] INFO io.bootique.job.consul.lock.ConsulLockHandler - Attempting to lock 'TestJobs/lock'
[bootique-job-2] ERROR io.bootique.job.consul.lock.ConsulLockHandler - Failed to release lock, manual intervention might be needed: TestJobs/lock
[pool-2-thread-1] INFO org.springframework.scheduling.concurrent.ThreadPoolTaskScheduler - Shutting down ExecutorService

java.lang.AssertionError: 
Expected :1
Actual   :2

[atomashpolskiy]

I was not able to find any problems with the test yesterday. I'll take another look shortly. Btw, I've ran the test on Ubuntu. Will check on my Mac..

…

On Fri, 31 Aug 2018, 11:59 Nikita Timofeev, ***@***.***> wrote: Test failure (from PR #70 <#70>) is reproducible on my machine too (MacOS). @atomashpolskiy <https://github.com/atomashpolskiy> can you please review test itself, is it correct? Here is log: [main] INFO 🐳 [consul:latest] - Container consul:latest started [main] INFO org.springframework.scheduling.concurrent.ThreadPoolTaskScheduler - Initializing ExecutorService [bootique-job-1] INFO io.bootique.job.consul.lock.ConsulLockHandler - Attempting to lock 'TestJobs/lock' [bootique-job-2] INFO io.bootique.job.consul.lock.ConsulLockHandler - Attempting to lock 'TestJobs/lock' [bootique-job-2] ERROR io.bootique.job.consul.lock.ConsulLockHandler - Failed to release lock, manual intervention might be needed: TestJobs/lock [pool-2-thread-1] INFO org.springframework.scheduling.concurrent.ThreadPoolTaskScheduler - Shutting down ExecutorService java.lang.AssertionError: Expected :1 Actual :2 — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#71 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/ADk0hh-mmc8LAWSP1rEQcxPxsBXpNoL7ks5uWPrrgaJpZM4WD_9A> .

[stariy95]

Cool, thanks! Different OS can be the case here (at least Docker can perform differently).
AFAIK Maxim run tests on Mac too.

[atomashpolskiy]

Turns out, I was looking at and talking about testConsulClusterJobLocking. Sorry :)

Regarding the testConsulLocalJobLocking test: from my point of view this behavior (lock reentrancy, basically) is to be expected. And in any case, why would you ever need two job instances running simultaneously in the same Bootique runtime?

[atomashpolskiy]

I guess that the fool-proof solution would be to always decorate serial jobs with an additional local lock :)

[MegaPapa]

Suppose, we have api for running job, without local locking we can start several executions of a one job (with one bqruntime).
P.S. bootique-zookeeper has clustered and local locking mechanism.