leandromoreira / redlock-rb

Redlock is a redis-based distributed lock implementation in Ruby. More than 20M downloads.

Redlock - A ruby distributed lock using redis.

Distributed locks are a very useful primitive in many environments where different processes require to operate with shared resources in a mutually exclusive way.

There are a number of libraries and blog posts describing how to implement a DLM (Distributed Lock Manager) with Redis, but every library uses a different approach, and many use a simple approach with lower guarantees compared to what can be achieved with slightly more complex designs.

This is an implementation of a proposed distributed lock algorithm with Redis. It started as a fork from antirez implementation.

Compatibility

• It works with Redis server versions 6.0 or later.
• Redlock >= 2.0 only works with RedisClient client instance.

Installation

Add this line to your application's Gemfile:

gem 'redlock'

And then execute:

$ bundle

Or install it yourself as:

$ gem install redlock

Documentation

RubyDoc

Usage example

Acquiring a lock

NOTE: All expiration durations are in milliseconds.

  # Locking
  lock_manager = Redlock::Client.new([ "redis://127.0.0.1:7777", "redis://127.0.0.1:7778", "redis://127.0.0.1:7779" ])
  first_try_lock_info = lock_manager.lock("resource_key", 2000)
  second_try_lock_info = lock_manager.lock("resource_key", 2000)

  p first_try_lock_info
  # => {validity: 1987, resource: "resource_key", value: "generated_uuid4"}

  p second_try_lock_info
  # => false

  # Unlocking
  lock_manager.unlock(first_try_lock_info)

  second_try_lock_info = lock_manager.lock("resource_key", 2000)

  p second_try_lock_info
  # => {validity: 1962, resource: "resource_key", value: "generated_uuid5"}

There's also a block version that automatically unlocks the lock:

lock_manager.lock("resource_key", 2000) do |locked|
  if locked
    # critical code
  else
    # error handling
  end
end

There's also a bang version that only executes the block if the lock is successfully acquired, returning the block's value as a result, or raising an exception otherwise. Passing a block is mandatory.

begin
  block_result = lock_manager.lock!("resource_key", 2000) do
    # critical code
  end
rescue Redlock::LockError
  # error handling
end

Extending a lock

To extend the life of the lock:

begin
  lock_info = lock_manager.lock("resource_key", 2000)
  while lock_info
    # Critical code

    # Time up and more work to do? Extend the lock.
    lock_info = lock_manager.lock("resource key", 3000, extend: lock_info)
  end
rescue Redlock::LockError
  # error handling
end

The above code will also acquire the lock if the previous lock has expired and the lock is currently free. Keep in mind that this means the lock could have been acquired and released by someone else in the meantime. To only extend the life of the lock if currently locked by yourself, use the extend_only_if_locked parameter:

lock_manager.lock("resource key", 3000, extend: lock_info, extend_only_if_locked: true)

Querying lock status

You can check if a resource is locked:

resource = "resource_key"
lock_info = lock_manager.lock(resource, 2000)
lock_manager.locked?(resource)
#=> true

lock_manager.unlock(lock_info)
lock_manager.locked?(resource)
#=> false

Any caller can call the above method to query the status. If you hold a lock and would like to check if it is valid, you can use the valid_lock? method:

lock_info = lock_manager.lock("resource_key", 2000)
lock_manager.valid_lock?(lock_info)
#=> true

lock_manager.unlock(lock_info)
lock_manager.valid_lock?(lock_info)
#=> false

The above methods are not safe if you are using this to time critical code, since they return true if the lock has not expired, even if there's only (for example) 1ms left on the lock. If you want to safely time the lock validity, you can use the get_remaining_ttl_for_lock and get_remaining_ttl_for_resource methods.

Use get_remaining_ttl_for_lock if you hold a lock and want to check the TTL specifically for your lock:

resource = "resource_key"
lock_info = lock_manager.lock(resource, 2000)
sleep 1

lock_manager.get_remaining_ttl_for_lock(lock_info)
#=> 986

lock_manager.unlock(lock_info)
lock_manager.get_remaining_ttl_for_lock(lock_info)
#=> nil

Use get_remaining_ttl_for_resource if you do not hold a lock, but want to know the remaining TTL on a locked resource:

# Some part of the code
resource = "resource_key"
lock_info = lock_manager.lock(resource, 2000)

# Some other part of the code
lock_manager.locked?(resource)
#=> true
lock_manager.get_remaining_ttl_for_resource(resource)
#=> 1975

# Sometime later
lock_manager.locked?(resource)
#=> false
lock_manager.get_remaining_ttl_for_resource(resource)
#=> nil

Redis client configuration

Redlock::Client expects URLs, or configurations or Redis objects on initialization. Redis objects should be used for configuring the connection in more detail, i.e. setting username and password.

servers = [ 'redis://localhost:6379', RedisClient.new(:url => 'redis://someotherhost:6379') ]
redlock = Redlock::Client.new(servers)

To utilize Redlock::Client with sentinels you can pass an instance of RedisClient or just a configuration hash as part of the servers array during initialization.

config = {
  name: "mymaster",
  sentinels: [
    { host: "127.0.0.1", port: 26380 },
    { host: "127.0.0.1", port: 26381 },
  ],
  role: :master
}
client = RedisClient.sentinel(**config).new_client
servers = [ config, client ]
redlock = Redlock::Client.new(servers)

Redlock supports the same configuration hash as RedisClient.

Redlock configuration

It's possible to customize the retry logic providing the following options:

  lock_manager = Redlock::Client.new(
                  servers, {
                  retry_count:   3,
                  retry_delay:   200, # milliseconds
                  retry_jitter:  50,  # milliseconds
                  redis_timeout: 0.1  # seconds
                 })

It is possible to associate :retry_delay option with Proc object. It will be called every time, with attempt number as argument, to get delay time value before next retry.

retry_delay = proc { |attempt_number| 200 * attempt_number ** 2 } # delay of 200ms for 1st retry, 800ms for 2nd retry, etc.
lock_manager = Redlock::Client.new(servers, retry_delay: retry_delay)

For more information you can check documentation.

Run tests

Make sure you have docker installed.

$ make

Disclaimer

This code implements an algorithm which is currently a proposal, it was not formally analyzed. Make sure to understand how it works before using it in your production environments. You can see discussion about this approach at reddit and also the Antirez answers for some critics.

Contributing

1. Fork it
2. Create your feature branch (git checkout -b my-new-feature)
3. Commit your changes (git commit -am 'Add some feature')
4. Push to the branch (git push origin my-new-feature)
5. Create a new Pull Request

redlock-rb's Issues

Fix extend and remove extend_life

Thanks @seamusabshere for pointing it out.

Race condition in #unlock

What if by the time we call get and del a new lock gets created?

https://github.com/leandromoreira/redlock-rb/blob/master/lib/redlock/client.rb#L146-L150

        if redis.call("get",KEYS[1]) == ARGV[1] then
          return redis.call("del",KEYS[1])
        else
          return 0
        end

provide documentation to new method extend_life

Doc the extend_life

need docs for ttl

I don't understand what the ttl does, with regard to locks and unlocks.

lock_manager.lock do
   expensive_work
end

If the expensive_work is done shorter than the ttl, the lock is still released immediately at end of block, right?

If the expensive_work takes longer than the ttl.... is the lock essentially released early? Is the ttl actually like a failsafe, in case the code crashed without unlocking? Which effectively makes the ttl the maximum amount of time the lock can be held for? Do I have this right?

Does the time in the ttl include the time spent waiting trying to acquire the lock, if relevant?

I couldn't find any clear explanation of this in the README or wiki or issues or anywhere. More docs? Or even an answer on this issue will perhaps be findable to more people?

Allow non-expiring locks?

It doesn't seem necessary to mandate a TTL when using the block form (#lock and #lock!). Naturally, you want it to hold the lock until the block finishes executing, however long that might be. The ensure in #lock will make sure it's unlocked if the block raises an error. Am I missing something?

I'd be happy to take a stab at a PR if we think this makes sense. My only questions are 1) how to indicate a non-expiring lock (zero or nil ttl?) and 2) how then to populate the validity field in lock_info.

Timeout error causing a locked resource

Hi, we are using redlock 0.2.2 with redis 3.3.3.
I checked the release notes of redlock 1.0 and newer redis.rb versions which didn't seems to fix the issue (see my analysis bellow).

We configured a short 'redis_timeout' (0.02 seconds) in a certain flow, since then we see a weird behaviour from Redis:
After Redis:Timeout error ('rescue in _read_from_socket') it seems the resource is already locked, no other indication to another flow which tries to lock the same resource.
The issue is fixed after max_lock_time has been reached.

A deeper dive into both Redis-rb and Redlock made us believe the issue lies in Redis/client.rb:123 - call(command):
We suspect the timeout occurred in the middle of the 'read; after the resource have already been locked by the 'write' command which happened before.

We cannot fix it in our side because we do not have the 'lock handle'.
It can be fixed inside redlock by catching TimeoutError in redlock/client.rb :127 and the calling unlock to make sure the resource will not be locked unexpectedly.

Is there a known workaround to solve this?

lock fail frequently

Scence：

Use redlock to lock product record when reduce the stock number. code like that:

product_lock = $redlock_client.lock(product.redlock_key, ENV.fetch('REDLOCK_TIME', 4000))
if product_lock
  product.stock -= buy_num
  product.save!

  $redlock_client.unlock(product_lock)
else
  Rails.logger.info("set_order_paid.redlock.fail product:#{product.id}")
end

---

When high concurrency level (like more than 2000 - 5000 request/s), product_lock fail frequently.
We don't want to use the database lock, because it will make deadlock sometime. And we also use the retry setting, but not go well.

retry_count: 5
retry_delay: 500

What should we do ? Thanks for your help!

Support for exponential backoff

My understanding is that currently, redlock will retry a request to Redis every retry_delay milliseconds, with some retry_jitter jitter in between each request.

However, it would be nice if, either instead of or in addition to jitter, a user of redlock could a set a parameter to allow for exponential backoff between the retries. A user could also say what they want the base and max number of milliseconds to wait before retries.

The use case is that we don't want to continually make requests to Redis at the same interval, if we know the lock is held up.

Plan for switching to RedisClient

Hi! Is there any plan or roadmap to move to https://github.com/redis-rb/redis-client that I can contribute to?

Lock extension via lock method allows extending expired locks

If your code allows the lock to expire and then you call lock to extend the life of the lock, it will allow it. It would be really handy to have it fail if the lock was allowed to expire. This has been added to the node and .NET versions of the algorithm, so there's some precedent.

Use of mathn and a single server prevents the ability to acquire a lock resource

If the mathn library is being used with redlock-db, and only a single server is being tested, then the @quorum variable will have a rational value of (3/2), rather than the integer value 1, which means that there will never be a quorum of servers that agree.

To avoid this, quorum can be initialized more explicitly like so:

@quorum = (servers.length / 2).to_i + 1

See this ruby-lang bug for more information:

https://bugs.ruby-lang.org/issues/2121

unlock method returns NoMethodError: undefined method `[]' for false:FalseClass

the unlock method returns NoMethodError: undefined method '[]' for false:FalseClass

my code
lock_manager = Redlock::Client.new(REDIS_LOCKING[:servers])
lock_info = lock_manager.lock(retailer_id.to_s, REDIS_LOCKING[:ttl])
.....
lock_manager.unlock(lock_info)

---

error from sentry

Question/Feature Request: How to check if lock has expired?

I am running some tasks that could have a fairly wide variability in execution times due to reasons outside of my application's control. Despite this, I need to guarantee that the task executes exclusively. I've read through the discussion at #94, and I understand that mutual exclusion is only guaranteed for the requested TTL. I think the compromises made with respect to this guarantee are sensible, and in my use case, I would indeed like the locks to expire if some execution is taking too long.

So as recommended by this and this, I need to implement an app level way of polling the lock at defined critical points in potentially long tasks, and aborting execution if the lock has been released. However, as far as I can see the from the docs and source code, this gem doesn't really provide a way to check if a lock is still valid.

From what I've gathered from the source code, it looks like given a redis instance, I could check this from the application side like this:

redis.get(lock_info[:resource]) == lock_info[:value]

Is this something that you would consider including directly in the gem? Perhaps with an API like this:

lock_manager.locked?(lock_info) # true/false

Further, I am considering that I may also want to track the TTL, so that a task can decide to abort early if the remaining execution would be estimated to take longer than the remaining TTL. This would allow for more granular control:

lock_manager.get_ttl(lock_info) # nil if lock expired, remaining time in ms if not yet.

It also seems that this could also be achieved using the redis PTTL command on the lock_info[:resource] key.

What do you think about these additions? I think they would make the gem API better suited for fine grained control, and provide a cleaner boundary between the gem's responsibilities and the application's.

I'm happy to attempt a PR if you think it's a good idea.

Redlock::Client.lock returns false when there is a connection issue

Hello, everyone. I want to use redlock to implement a rate limit so that duplicate requests within a certain time frame are dropped. According to the documentation, if an attempt to acquire a lock fails, the lock method returns false. At the same time, after rescuing exceptions, lock returns false too. Such behavior, in my opinion, is dangerous and an anti-pattern. If a client receives a false response from lock, there is no guarantee that this is because the lock has already been acquired. In the scenario described above , this behavior could result in the drop of all requests if the connection to the redis servers fails (once that Redis::BaseConnectionError is rescued).

How to use with Redis sentinel?

It's a bit unclear how to use this gem with Redis sentinel. I think I should pass the Redis master ip in the initializer (and not the sentinels-ips)?

And am I right that i need to update my code when the Redis master changes? (Since writes can only be done on master)

Migrate travis to container-based infrastructure

http://docs.travis-ci.com/user/migrating-from-legacy/?utm_source=legacy-notice&utm_medium=banner&utm_campaign=legacy-upgrade

extend_life interprets TTL as seconds rather than milliseconds (0.1.8, fixed on master)

I'm rather new with redlock-rb so bear with me 😄

While doing extensive tests I think I stumbled upon a bug on version 0.1.8 (I'm using redis 3.0.7 and ruby 2.2.3 on that specific test).

I believe when extend_life: true is passed, 0.1.8 interprets the TTL value as seconds, rather than milliseconds. Master (commit 7d9c91a) will interpret the value as milliseconds (as I'd have expected).

I'm running this:

redis.flushall
ap "==================="
lock_info = lock_manager.lock('test', 2000)
5.times do
  ap redis.pttl('test')
  sleep(0.1)
end
ap "==================="
lock_info = lock_manager.lock('test', 3000, extend: lock_info, extend_life: true)
5.times do
  ap redis.pttl('test')
  sleep(0.1)
end

Output on 0.1.8

"==================="
1999
1898
1797
1696
1594
"==================="
3000000
2999898
2999797
2999696
2999594

Output on master

"==================="
1999
1899
1797
1695
1594
"==================="
2999
2897
2783
2681
2579

CI is not working

https://travis-ci.org/github/leandromoreira/redlock-rb/jobs/748334352

CHANGELOG or UPGRADING?

New major version dropped, and not sure where to look for breaking changes.

RETRY_COUNT is a misleading variable

In the comments for the initialize method of the RedLock client retry_count is defined as: being how many times it'll try to lock a resource (default: 3).

IMO, this is misleading. When I read retry_count I assume that it will try to lock the resource at least once before running the retry logic.

I think you should rename this variable to try_count if you do not want to make significant code changes our keep it as retry_count but then change the code to actually try to acquire the lock once before going into the retry logic.

Does v1.3.1 not have any code changes?

Hi. I checked the changing of v1.3.1 with GitHub's comparing changes page. But I couldn't see any changes of codes.
1.3.0...1.3.1

According to the release page, v1.3.1 would be including changes of #110. But I couldn't see that changes. I also checked the source codes of the gem file, but it seems that codes doesn't have changes of #110.

Is there any mistakes for the release? I'm sorry if I misunderstood. Thanks.

Redis error message

We're using redlock in Hyku and Hyrax and I keep seeing the following:

Passing 'script' command to redis as is; administrative commands cannot be effectively namespaced and should be called on the redis connection directly; passthrough has been deprecated and will be removed in redis-namespace 2.0 (at /usr/local/rbenv/versions/2.4.1/lib/ruby/gems/2.4.0/gems/redlock-0.2.0/lib/redlock/client.rb:127:in `load_scripts')

I see the same issue has been logged against sidekiq, with some discussion of the solution there:
sidekiq/sidekiq#3366

Question: Redis 2.4 unsupported?

When using redlock with Redis 2.4, I'm seeing:

Redis::CommandError
        ERR unknown command 'script'

It looks like redlock is using script commands and features that were implemented in 2.6, and would not be compatible with prior versions. Is that correct?

Redlock::Client#lock raises error when one of the Redis instances is broken

As far as I understand Redlock, it is about distributing locks across many instances in order to avoid single point of failure.

Here is the scenario which I don't get.

1. Spin up three redis instances.
2. Create redlock client connected to those three.
3. Call client.lock("lock1", 10000) which creates locks on all three instances successfully
4. Shut down one redis instance.
5. Try to acquire lock with client.lock("lock2", 10000) which raises:
   SocketError: getaddrinfo: Name or service not known

Isn't this supposed to be handled by Redlock? So that I don't get the error raised. I thought it is assumed that one of the instance would crash.

Pls tell me if I'm missing something.

I have used redlock gem with a single instance so far, though I want to migrate those locks to other. Hence I'm trying to figure out, how does redlock works when one of Redis instances crashes.

Breaking change in `lock_instances` from `0.1.8`

The line in current master:
https://github.com/leandromoreira/redlock-rb/blob/master/lib/redlock/client.rb#L166

is a breaking change from 0.1.8:
https://github.com/leandromoreira/redlock-rb/blob/0.1.8/lib/redlock/client.rb#L176

Old behavior: if :extend key exists but is nil, SecureRandom.uuid will be returned.
New behavior: if :extend key exists but is nil, nil will be returned, because fetch returns the default only if the key is missing.

Is this by design?

[Question] about the `NX` flag

Hello

At risk to look like a complet brainless person.
Why don't you use the flag NX when you try get a lock ?
I understand why NX is a problem when you want to extand a lock.
But after reading redis paper, NX seems mandatory for all new lock.

Did i miss something critical ?

Are you using this gem in production?

Hi folks!

Are you using this gem in production?

Is it trustworthy to use it?

Feature request: allow `lock!` to be extended

I would like to extend a lock!

Would you consider making lock! return the lock_info and also allow it to be called without a block, e.g.

def lock!(*args)
  lock(*args) do |lock_info|
    raise LockError, 'failed to acquire lock' unless lock_info
    if block_given?
      yield lock_info
    else
      lock_info
    end
  end
end

Change the block behavior to return the yield return

Hi,

when I was trying to use the lock passing a block, I noticed the method doesn't return the block return.

Before the use of lock, I had a method like this:

def do_somenthing
  call_something
end

And I tryed to do something this:

def do_somenthing
  Redlock::Client.new(redis_config).lock do |lock_info|
    if lock_info
      call_something
    else
       # whatever
    end
  end
end 

But it was not possible because the lock method returns a boolean when passing a block.

What do you think of changing this behavior?

misleading typo in README

retry_timeout should be redis_timeout

  lock_manager = Redlock::Client.new(
    servers, {
      retry_count:   3,
      retry_delay:   200, # milliseconds
      retry_jitter:  50,  # milliseconds
      retry_timeout: 0.1  # seconds
    })

Connection issue

With the new 1.1.0 release I have this issue now:

Redis::CannotConnectError: Error connecting to Redis on 127.0.0.1:6379 (Errno::ECONNREFUSED)
/var/app/current/vendor/bundle/ruby/2.6.0/gems/redis-4.1.3/lib/redis/client.rb:362:in `rescue in establish_connection'
/var/app/current/vendor/bundle/ruby/2.6.0/gems/redis-4.1.3/lib/redis/client.rb:343:in `establish_connection'
/var/app/current/vendor/bundle/ruby/2.6.0/gems/redis-4.1.3/lib/redis/client.rb:106:in `block in connect'
/var/app/current/vendor/bundle/ruby/2.6.0/gems/redis-4.1.3/lib/redis/client.rb:306:in `with_reconnect'
/var/app/current/vendor/bundle/ruby/2.6.0/gems/redis-4.1.3/lib/redis/client.rb:105:in `connect'
/var/app/current/vendor/bundle/ruby/2.6.0/gems/redis-4.1.3/lib/redis/client.rb:381:in `ensure_connected'
/var/app/current/vendor/bundle/ruby/2.6.0/gems/redis-4.1.3/lib/redis/client.rb:231:in `block in process'
/var/app/current/vendor/bundle/ruby/2.6.0/gems/redis-4.1.3/lib/redis/client.rb:319:in `logging'
/var/app/current/vendor/bundle/ruby/2.6.0/gems/redis-4.1.3/lib/redis/client.rb:230:in `process'
/var/app/current/vendor/bundle/ruby/2.6.0/gems/redis-4.1.3/lib/redis/client.rb:125:in `call'
/var/app/current/vendor/bundle/ruby/2.6.0/gems/redis-4.1.3/lib/redis.rb:2543:in `block in script'
/var/app/current/vendor/bundle/ruby/2.6.0/gems/redis-4.1.3/lib/redis.rb:52:in `block in synchronize'
/var/app/current/vendor/bundle/ruby/2.6.0/gems/redis-4.1.3/lib/redis.rb:52:in `synchronize'
/var/app/current/vendor/bundle/ruby/2.6.0/gems/redis-4.1.3/lib/redis.rb:2542:in `script'
/var/app/current/vendor/bundle/ruby/2.6.0/gems/redlock-1.1.0/lib/redlock/client.rb:154:in `load_scripts'
/var/app/current/vendor/bundle/ruby/2.6.0/gems/redlock-1.1.0/lib/redlock/testing.rb:31:in `load_scripts'
/var/app/current/vendor/bundle/ruby/2.6.0/gems/redlock-1.1.0/lib/redlock/client.rb:132:in `initialize'
/var/app/current/vendor/bundle/ruby/2.6.0/gems/redlock-1.1.0/lib/redlock/client.rb:42:in `new'
/var/app/current/vendor/bundle/ruby/2.6.0/gems/redlock-1.1.0/lib/redlock/client.rb:42:in `block in initialize'
/var/app/current/vendor/bundle/ruby/2.6.0/gems/redlock-1.1.0/lib/redlock/client.rb:38:in `map'
/var/app/current/vendor/bundle/ruby/2.6.0/gems/redlock-1.1.0/lib/redlock/client.rb:38:in `initialize'
/var/app/current/vendor/bundle/ruby/2.6.0/gems/generic_app-6.2.10/config/initializers/redlock.rb:8:in `new'

Testing mode code in the initializer is:

# generic_app-6.2.10/config/initializers/redlock.rb
if Rails.env.test? or ENV['REDIS_URL'].blank?
  # Needed for redlock mock
  require 'redlock/testing'
end

redis_opts = {url: ENV['REDIS_URL'], namespace: "redlock_#{Rails.application.class.name}_#{Rails.env}"}
redis_opts.merge!(driver: Redis::Connection::Memory) if defined?(Redis::Connection::Memory)
LOCK_MANAGER = Redlock::Client.new([Redis.new(redis_opts)])

if Rails.env.test?
  LOCK_MANAGER.testing_mode = :bypass
end

Why is this causing an error when running in bypass mode?

Handling timeout error from RedisClient

Hi,
We receive Redis::TimeoutError, from Redlock::Client:127 which are not handled by Redlock::Client which causes 'deadlocked' instances.
For now we catch the exception in order to handle it however the to fix the deadlock we need a feature request from redlock.

I would like to change the rescue at line 129 from 'CannotConnectError' to 'BaseConnectionError' which will cause the exception to be caught and then the instance will be unlocked.
Best regards,
Asaf

Feature suggestion: Extending lock life

Hi,
One of the benefits of using mlanett-redis-lock is that is you can extend the lock life from internally in the lock, this is useful when iterating over an unknown amount of information/data.
However we use RedLock because we don't have to do blocks which is important to us.

I'd like to include this into RedLock, am I allowed to fork and add this?

--Karl.

Feature request: support Redis as a ConnectionPool

We have a pool of Redis connections shared among our threaded webserver that we'd like to use to pass into Redlock but there's no way to initialize with a pool. Please consider adding support for this!

Test failed when using with fakeredis

env

$ gem list | grep redlock
redlock (1.0.0)
$ gem list | grep fakeredis
fakeredis (0.7.0)
$ gem list | grep redis
fakeredis (0.7.0)
redis (3.3.3)

After I used redlock and tried to run my test I got the folloing error.

error

"ERR unknown command 'script'"
[
    [  0] "/Users/feng.zhang/.rbenv/versions/2.3.3/lib/ruby/gems/2.3.0/gems/fakeredis-0.7.0/lib/fakeredis/command_executor.rb:15:in `write'",
    [  1] "/Users/feng.zhang/.rbenv/versions/2.3.3/lib/ruby/gems/2.3.0/gems/redis-3.3.3/lib/redis/client.rb:271:in `block in write'",
    [  2] "/Users/feng.zhang/.rbenv/versions/2.3.3/lib/ruby/gems/2.3.0/gems/redis-3.3.3/lib/redis/client.rb:250:in `io'",
    [  3] "/Users/feng.zhang/.rbenv/versions/2.3.3/lib/ruby/gems/2.3.0/gems/redis-3.3.3/lib/redis/client.rb:269:in `write'",
    [  4] "/Users/feng.zhang/.rbenv/versions/2.3.3/lib/ruby/gems/2.3.0/gems/redis-3.3.3/lib/redis/client.rb:228:in `block (3 levels) in process'",
    [  5] "/Users/feng.zhang/.rbenv/versions/2.3.3/lib/ruby/gems/2.3.0/gems/redis-3.3.3/lib/redis/client.rb:222:in `each'",
    [  6] "/Users/feng.zhang/.rbenv/versions/2.3.3/lib/ruby/gems/2.3.0/gems/redis-3.3.3/lib/redis/client.rb:222:in `block (2 levels) in process'",
    [  7] "/Users/feng.zhang/.rbenv/versions/2.3.3/lib/ruby/gems/2.3.0/gems/redis-3.3.3/lib/redis/client.rb:367:in `ensure_connected'",
    [  8] "/Users/feng.zhang/.rbenv/versions/2.3.3/lib/ruby/gems/2.3.0/gems/redis-3.3.3/lib/redis/client.rb:221:in `block in process'",
    [  9] "/Users/feng.zhang/.rbenv/versions/2.3.3/lib/ruby/gems/2.3.0/gems/redis-3.3.3/lib/redis/client.rb:306:in `logging'",
    [ 10] "/Users/feng.zhang/.rbenv/versions/2.3.3/lib/ruby/gems/2.3.0/gems/redis-3.3.3/lib/redis/client.rb:220:in `process'",
    [ 11] "/Users/feng.zhang/.rbenv/versions/2.3.3/lib/ruby/gems/2.3.0/gems/redis-3.3.3/lib/redis/client.rb:120:in `call'",
    [ 12] "/Users/feng.zhang/.rbenv/versions/2.3.3/lib/ruby/gems/2.3.0/gems/redis-3.3.3/lib/redis.rb:2385:in `block in script'",
    [ 13] "/Users/feng.zhang/.rbenv/versions/2.3.3/lib/ruby/gems/2.3.0/gems/redis-3.3.3/lib/redis.rb:58:in `block in synchronize'",
    [ 14] "/Users/feng.zhang/.rbenv/versions/2.3.3/lib/ruby/2.3.0/monitor.rb:214:in `mon_synchronize'",
    [ 15] "/Users/feng.zhang/.rbenv/versions/2.3.3/lib/ruby/gems/2.3.0/gems/redis-3.3.3/lib/redis.rb:58:in `synchronize'",
    [ 16] "/Users/feng.zhang/.rbenv/versions/2.3.3/lib/ruby/gems/2.3.0/gems/redis-3.3.3/lib/redis.rb:2384:in `script'",
    [ 17] "/Users/feng.zhang/.rbenv/versions/2.3.3/lib/ruby/gems/2.3.0/gems/redlock-1.0.0/lib/redlock/client.rb:144:in `load_scripts'",
    [ 18] "/Users/feng.zhang/.rbenv/versions/2.3.3/lib/ruby/gems/2.3.0/gems/redlock-1.0.0/lib/redlock/client.rb:122:in `initialize'",
    [ 19] "/Users/feng.zhang/.rbenv/versions/2.3.3/lib/ruby/gems/2.3.0/gems/redlock-1.0.0/lib/redlock/client.rb:40:in `new'",
    [ 20] "/Users/feng.zhang/.rbenv/versions/2.3.3/lib/ruby/gems/2.3.0/gems/redlock-1.0.0/lib/redlock/client.rb:40:in `block in initialize'",
    [ 21] "/Users/feng.zhang/.rbenv/versions/2.3.3/lib/ruby/gems/2.3.0/gems/redlock-1.0.0/lib/redlock/client.rb:38:in `map'",
    [ 22] "/Users/feng.zhang/.rbenv/versions/2.3.3/lib/ruby/gems/2.3.0/gems/redlock-1.0.0/lib/redlock/client.rb:38:in `initialize'",
    [ 23] "/Users/feng.zhang/ase/fout_x/manager/app/clients/redlock_client.rb:10:in `new'",
    [ 24] "/Users/feng.zhang/ase/fout_x/manager/app/clients/redlock_client.rb:10:in `initialize'",
    [ 25] "/Users/feng.zhang/.rbenv/versions/2.3.3/lib/ruby/2.3.0/singleton.rb:142:in `new'",
    [ 26] "/Users/feng.zhang/.rbenv/versions/2.3.3/lib/ruby/2.3.0/singleton.rb:142:in `block in instance'",
    [ 27] "/Users/feng.zhang/.rbenv/versions/2.3.3/lib/ruby/2.3.0/singleton.rb:140:in `synchronize'",
    [ 28] "/Users/feng.zhang/.rbenv/versions/2.3.3/lib/ruby/2.3.0/singleton.rb:140:in `instance'",
    [ 29] "/Users/feng.zhang/ase/fout_x/manager/app/workers/create_location_worker.rb:32:in `block in perform'",
    [ 30] "/Users/feng.zhang/.rbenv/versions/2.3.3/lib/ruby/gems/2.3.0/gems/rgeo-geojson-0.4.3/lib/rgeo/geo_json/entities.rb:141:in `each'",
    [ 31] "/Users/feng.zhang/.rbenv/versions/2.3.3/lib/ruby/gems/2.3.0/gems/rgeo-geojson-0.4.3/lib/rgeo/geo_json/entities.rb:141:in `each'",
    [ 32] "/Users/feng.zhang/ase/fout_x/manager/app/workers/create_location_worker.rb:26:in `perform'",
    [ 33] "/Users/feng.zhang/ase/fout_x/manager/app/workers/set_cv_worker.rb:49:in `perform'",
    [ 34] "/Users/feng.zhang/.rbenv/versions/2.3.3/lib/ruby/gems/2.3.0/gems/sidekiq-5.0.0/lib/sidekiq/testing.rb:292:in `execute_job'",
    [ 35] "/Users/feng.zhang/.rbenv/versions/2.3.3/lib/ruby/gems/2.3.0/gems/sidekiq-5.0.0/lib/sidekiq/testing.rb:287:in `block in process_job'",
    [ 36] "/Users/feng.zhang/.rbenv/versions/2.3.3/lib/ruby/gems/2.3.0/gems/sidekiq-5.0.0/lib/sidekiq/middleware/chain.rb:128:in `block in invoke'",
    [ 37] "/Users/feng.zhang/.rbenv/versions/2.3.3/lib/ruby/gems/2.3.0/gems/sidekiq-5.0.0/lib/sidekiq/middleware/chain.rb:133:in `invoke'",
    [ 38] "/Users/feng.zhang/.rbenv/versions/2.3.3/lib/ruby/gems/2.3.0/gems/sidekiq-5.0.0/lib/sidekiq/testing.rb:286:in `process_job'",
    [ 39] "/Users/feng.zhang/.rbenv/versions/2.3.3/lib/ruby/gems/2.3.0/gems/sidekiq-5.0.0/lib/sidekiq/testing.rb:82:in `block in raw_push'",
    [ 40] "/Users/feng.zhang/.rbenv/versions/2.3.3/lib/ruby/gems/2.3.0/gems/sidekiq-5.0.0/lib/sidekiq/testing.rb:78:in `each'",
    [ 41] "/Users/feng.zhang/.rbenv/versions/2.3.3/lib/ruby/gems/2.3.0/gems/sidekiq-5.0.0/lib/sidekiq/testing.rb:78:in `raw_push'",
    [ 42] "/Users/feng.zhang/.rbenv/versions/2.3.3/lib/ruby/gems/2.3.0/gems/sidekiq-5.0.0/lib/sidekiq/client.rb:69:in `push'",
    [ 43] "/Users/feng.zhang/.rbenv/versions/2.3.3/lib/ruby/gems/2.3.0/gems/sidekiq-5.0.0/lib/sidekiq/worker.rb:139:in `client_push'",
    [ 44] "/Users/feng.zhang/.rbenv/versions/2.3.3/lib/ruby/gems/2.3.0/gems/sidekiq-5.0.0/lib/sidekiq/worker.rb:88:in `perform_async'",
    [ 45] "/Users/feng.zhang/ase/fout_x/manager/app/controllers/conversions_controller.rb:81:in `create'",
    [ 46] "/Users/feng.zhang/.rbenv/versions/2.3.3/lib/ruby/gems/2.3.0/gems/actionpack-5.0.3/lib/action_controller/metal/basic_implicit_render.rb:4:in `send_action'",
    [ 47] "/Users/feng.zhang/.rbenv/versions/2.3.3/lib/ruby/gems/2.3.0/gems/actionpack-5.0.3/lib/abstract_controller/base.rb:188:in `process_action'",
    [ 48] "/Users/feng.zhang/.rbenv/versions/2.3.3/lib/ruby/gems/2.3.0/gems/actionpack-5.0.3/lib/action_controller/metal/rendering.rb:30:in `process_action'",
    [ 49] "/Users/feng.zhang/.rbenv/versions/2.3.3/lib/ruby/gems/2.3.0/gems/actionpack-5.0.3/lib/abstract_controller/callbacks.rb:20:in `block in process_action'",
    [ 50] "/Users/feng.zhang/.rbenv/versions/2.3.3/lib/ruby/gems/2.3.0/gems/activesupport-5.0.3/lib/active_support/callbacks.rb:126:in `call'",
    [ 51] "/Users/feng.zhang/.rbenv/versions/2.3.3/lib/ruby/gems/2.3.0/gems/activesupport-5.0.3/lib/active_support/callbacks.rb:506:in `block (2 levels) in compile'",
    [ 52] "/Users/feng.zhang/.rbenv/versions/2.3.3/lib/ruby/gems/2.3.0/gems/activesupport-5.0.3/lib/active_support/callbacks.rb:455:in `call'",
    [ 53] "/Users/feng.zhang/.rbenv/versions/2.3.3/lib/ruby/gems/2.3.0/gems/activesupport-5.0.3/lib/active_support/callbacks.rb:101:in `__run_callbacks__'",
    [ 54] "/Users/feng.zhang/.rbenv/versions/2.3.3/lib/ruby/gems/2.3.0/gems/activesupport-5.0.3/lib/active_support/callbacks.rb:750:in `_run_process_action_callbacks'",
    [ 55] "/Users/feng.zhang/.rbenv/versions/2.3.3/lib/ruby/gems/2.3.0/gems/activesupport-5.0.3/lib/active_support/callbacks.rb:90:in `run_callbacks'",
    [ 56] "/Users/feng.zhang/.rbenv/versions/2.3.3/lib/ruby/gems/2.3.0/gems/actionpack-5.0.3/lib/abstract_controller/callbacks.rb:19:in `process_action'",
    [ 57] "/Users/feng.zhang/.rbenv/versions/2.3.3/lib/ruby/gems/2.3.0/gems/actionpack-5.0.3/lib/action_controller/metal/rescue.rb:20:in `process_action'",
    [ 58] "/Users/feng.zhang/.rbenv/versions/2.3.3/lib/ruby/gems/2.3.0/gems/actionpack-5.0.3/lib/action_controller/metal/instrumentation.rb:32:in `block in process_action'",
    [ 59] "/Users/feng.zhang/.rbenv/versions/2.3.3/lib/ruby/gems/2.3.0/gems/activesupport-5.0.3/lib/active_support/notifications.rb:164:in `block in instrument'",
    [ 60] "/Users/feng.zhang/.rbenv/versions/2.3.3/lib/ruby/gems/2.3.0/gems/activesupport-5.0.3/lib/active_support/notifications/instrumenter.rb:21:in `instrument'",
    [ 61] "/Users/feng.zhang/.rbenv/versions/2.3.3/lib/ruby/gems/2.3.0/gems/activesupport-5.0.3/lib/active_support/notifications.rb:164:in `instrument'",
    [ 62] "/Users/feng.zhang/.rbenv/versions/2.3.3/lib/ruby/gems/2.3.0/gems/actionpack-5.0.3/lib/action_controller/metal/instrumentation.rb:30:in `process_action'",
    [ 63] "/Users/feng.zhang/.rbenv/versions/2.3.3/lib/ruby/gems/2.3.0/gems/actionpack-5.0.3/lib/action_controller/metal/params_wrapper.rb:248:in `process_action'",
    [ 64] "/Users/feng.zhang/.rbenv/versions/2.3.3/lib/ruby/gems/2.3.0/gems/activerecord-5.0.3/lib/active_record/railties/controller_runtime.rb:18:in `process_action'",
    [ 65] "/Users/feng.zhang/.rbenv/versions/2.3.3/lib/ruby/gems/2.3.0/gems/actionpack-5.0.3/lib/abstract_controller/base.rb:126:in `process'",
    [ 66] "/Users/feng.zhang/.rbenv/versions/2.3.3/lib/ruby/gems/2.3.0/gems/actionview-5.0.3/lib/action_view/rendering.rb:30:in `process'",
    [ 67] "/Users/feng.zhang/.rbenv/versions/2.3.3/lib/ruby/gems/2.3.0/gems/actionpack-5.0.3/lib/action_controller/metal.rb:190:in `dispatch'",
    [ 68] "/Users/feng.zhang/.rbenv/versions/2.3.3/lib/ruby/gems/2.3.0/gems/actionpack-5.0.3/lib/action_controller/test_case.rb:547:in `process'",
    [ 69] "/Users/feng.zhang/.rbenv/versions/2.3.3/lib/ruby/gems/2.3.0/gems/rails-controller-testing-1.0.2/lib/rails/controller/testing/template_assertions.rb:61:in `process'",
    [ 70] "/Users/feng.zhang/.rbenv/versions/2.3.3/lib/ruby/gems/2.3.0/gems/devise-4.3.0/lib/devise/test/controller_helpers.rb:33:in `block in process'",
    [ 71] "/Users/feng.zhang/.rbenv/versions/2.3.3/lib/ruby/gems/2.3.0/gems/devise-4.3.0/lib/devise/test/controller_helpers.rb:100:in `catch'",
    [ 72] "/Users/feng.zhang/.rbenv/versions/2.3.3/lib/ruby/gems/2.3.0/gems/devise-4.3.0/lib/devise/test/controller_helpers.rb:100:in `_catch_warden'",
    [ 73] "/Users/feng.zhang/.rbenv/versions/2.3.3/lib/ruby/gems/2.3.0/gems/devise-4.3.0/lib/devise/test/controller_helpers.rb:33:in `process'",
    [ 74] "/Users/feng.zhang/.rbenv/versions/2.3.3/lib/ruby/gems/2.3.0/gems/actionpack-5.0.3/lib/action_controller/test_case.rb:644:in `process_with_kwargs'",
    [ 75] "/Users/feng.zhang/.rbenv/versions/2.3.3/lib/ruby/gems/2.3.0/gems/actionpack-5.0.3/lib/action_controller/test_case.rb:397:in `post'",
    [ 76] "/Users/feng.zhang/.rbenv/versions/2.3.3/lib/ruby/gems/2.3.0/gems/rails-controller-testing-1.0.2/lib/rails/controller/testing/integration.rb:12:in `block (2 levels) in <module:Integration>'",
    [ 77] "/Users/feng.zhang/ase/fout_x/manager/spec/controllers/conversions_controller_spec.rb:146:in `block (6 levels) in <top (required)>'",
    [ 78] "/Users/feng.zhang/.rbenv/versions/2.3.3/lib/ruby/gems/2.3.0/gems/rspec-expectations-3.6.0/lib/rspec/matchers/built_in/change.rb:347:in `perform_change'",
    [ 79] "/Users/feng.zhang/.rbenv/versions/2.3.3/lib/ruby/gems/2.3.0/gems/rspec-expectations-3.6.0/lib/rspec/matchers/built_in/change.rb:133:in `matches?'",
    [ 80] "/Users/feng.zhang/.rbenv/versions/2.3.3/lib/ruby/gems/2.3.0/gems/rspec-expectations-3.6.0/lib/rspec/expectations/handler.rb:50:in `block in handle_matcher'",
    [ 81] "/Users/feng.zhang/.rbenv/versions/2.3.3/lib/ruby/gems/2.3.0/gems/rspec-expectations-3.6.0/lib/rspec/expectations/handler.rb:27:in `with_matcher'",
    [ 82] "/Users/feng.zhang/.rbenv/versions/2.3.3/lib/ruby/gems/2.3.0/gems/rspec-expectations-3.6.0/lib/rspec/expectations/handler.rb:48:in `handle_matcher'",
    [ 83] "/Users/feng.zhang/.rbenv/versions/2.3.3/lib/ruby/gems/2.3.0/gems/rspec-expectations-3.6.0/lib/rspec/expectations/expectation_target.rb:65:in `to'",
    [ 84] "/Users/feng.zhang/.rbenv/versions/2.3.3/lib/ruby/gems/2.3.0/gems/rspec-expectations-3.6.0/lib/rspec/expectations/expectation_target.rb:101:in `to'",
    [ 85] "/Users/feng.zhang/ase/fout_x/manager/spec/controllers/conversions_controller_spec.rb:145:in `block (5 levels) in <top (required)>'",
    [ 86] "/Users/feng.zhang/.rbenv/versions/2.3.3/lib/ruby/gems/2.3.0/gems/rspec-expectations-3.6.0/lib/rspec/matchers/built_in/change.rb:347:in `perform_change'",
    [ 87] "/Users/feng.zhang/.rbenv/versions/2.3.3/lib/ruby/gems/2.3.0/gems/rspec-expectations-3.6.0/lib/rspec/matchers/built_in/change.rb:49:in `matches?'",
    [ 88] "/Users/feng.zhang/.rbenv/versions/2.3.3/lib/ruby/gems/2.3.0/gems/rspec-expectations-3.6.0/lib/rspec/matchers/built_in/change.rb:55:in `does_not_match?'",
    [ 89] "/Users/feng.zhang/.rbenv/versions/2.3.3/lib/ruby/gems/2.3.0/gems/rspec-expectations-3.6.0/lib/rspec/expectations/handler.rb:78:in `does_not_match?'",
    [ 90] "/Users/feng.zhang/.rbenv/versions/2.3.3/lib/ruby/gems/2.3.0/gems/rspec-expectations-3.6.0/lib/rspec/expectations/handler.rb:72:in `block in handle_matcher'",
    [ 91] "/Users/feng.zhang/.rbenv/versions/2.3.3/lib/ruby/gems/2.3.0/gems/rspec-expectations-3.6.0/lib/rspec/expectations/handler.rb:27:in `with_matcher'",
    [ 92] "/Users/feng.zhang/.rbenv/versions/2.3.3/lib/ruby/gems/2.3.0/gems/rspec-expectations-3.6.0/lib/rspec/expectations/handler.rb:70:in `handle_matcher'",
    [ 93] "/Users/feng.zhang/.rbenv/versions/2.3.3/lib/ruby/gems/2.3.0/gems/rspec-expectations-3.6.0/lib/rspec/expectations/expectation_target.rb:78:in `not_to'",
    [ 94] "/Users/feng.zhang/.rbenv/versions/2.3.3/lib/ruby/gems/2.3.0/gems/rspec-expectations-3.6.0/lib/rspec/expectations/expectation_target.rb:106:in `not_to'",
    [ 95] "/Users/feng.zhang/ase/fout_x/manager/spec/controllers/conversions_controller_spec.rb:144:in `block (4 levels) in <top (required)>'",
    [ 96] "/Users/feng.zhang/.rbenv/versions/2.3.3/lib/ruby/gems/2.3.0/gems/rspec-core-3.6.0/lib/rspec/core/example.rb:254:in `instance_exec'",
    [ 97] "/Users/feng.zhang/.rbenv/versions/2.3.3/lib/ruby/gems/2.3.0/gems/rspec-core-3.6.0/lib/rspec/core/example.rb:254:in `block in run'",
    [ 98] "/Users/feng.zhang/.rbenv/versions/2.3.3/lib/ruby/gems/2.3.0/gems/rspec-core-3.6.0/lib/rspec/core/example.rb:500:in `block in with_around_and_singleton_context_hooks'",
    [ 99] "/Users/feng.zhang/.rbenv/versions/2.3.3/lib/ruby/gems/2.3.0/gems/rspec-core-3.6.0/lib/rspec/core/example.rb:457:in `block in with_around_example_hooks'",
    [100] "/Users/feng.zhang/.rbenv/versions/2.3.3/lib/ruby/gems/2.3.0/gems/rspec-core-3.6.0/lib/rspec/core/hooks.rb:464:in `block in run'",
    [101] "/Users/feng.zhang/.rbenv/versions/2.3.3/lib/ruby/gems/2.3.0/gems/rspec-core-3.6.0/lib/rspec/core/hooks.rb:604:in `block in run_around_example_hooks_for'",
    [102] "/Users/feng.zhang/.rbenv/versions/2.3.3/lib/ruby/gems/2.3.0/gems/rspec-core-3.6.0/lib/rspec/core/example.rb:342:in `call'",
    [103] "/Users/feng.zhang/.rbenv/versions/2.3.3/lib/ruby/gems/2.3.0/gems/rspec-rails-3.6.0/lib/rspec/rails/example/controller_example_group.rb:191:in `block (2 levels) in <module:ControllerExampleGroup>'",
    [104] "/Users/feng.zhang/.rbenv/versions/2.3.3/lib/ruby/gems/2.3.0/gems/rspec-core-3.6.0/lib/rspec/core/example.rb:447:in `instance_exec'",
    [105] "/Users/feng.zhang/.rbenv/versions/2.3.3/lib/ruby/gems/2.3.0/gems/rspec-core-3.6.0/lib/rspec/core/example.rb:447:in `instance_exec'",
    [106] "/Users/feng.zhang/.rbenv/versions/2.3.3/lib/ruby/gems/2.3.0/gems/rspec-core-3.6.0/lib/rspec/core/hooks.rb:375:in `execute_with'",
    [107] "/Users/feng.zhang/.rbenv/versions/2.3.3/lib/ruby/gems/2.3.0/gems/rspec-core-3.6.0/lib/rspec/core/hooks.rb:606:in `block (2 levels) in run_around_example_hooks_for'",
    [108] "/Users/feng.zhang/.rbenv/versions/2.3.3/lib/ruby/gems/2.3.0/gems/rspec-core-3.6.0/lib/rspec/core/example.rb:342:in `call'",
    [109] "/Users/feng.zhang/.rbenv/versions/2.3.3/lib/ruby/gems/2.3.0/gems/rspec-rails-3.6.0/lib/rspec/rails/adapters.rb:127:in `block (2 levels) in <module:MinitestLifecycleAdapter>'",
    [110] "/Users/feng.zhang/.rbenv/versions/2.3.3/lib/ruby/gems/2.3.0/gems/rspec-core-3.6.0/lib/rspec/core/example.rb:447:in `instance_exec'",
    [111] "/Users/feng.zhang/.rbenv/versions/2.3.3/lib/ruby/gems/2.3.0/gems/rspec-core-3.6.0/lib/rspec/core/example.rb:447:in `instance_exec'",
    [112] "/Users/feng.zhang/.rbenv/versions/2.3.3/lib/ruby/gems/2.3.0/gems/rspec-core-3.6.0/lib/rspec/core/hooks.rb:375:in `execute_with'",
    [113] "/Users/feng.zhang/.rbenv/versions/2.3.3/lib/ruby/gems/2.3.0/gems/rspec-core-3.6.0/lib/rspec/core/hooks.rb:606:in `block (2 levels) in run_around_example_hooks_for'",
    [114] "/Users/feng.zhang/.rbenv/versions/2.3.3/lib/ruby/gems/2.3.0/gems/rspec-core-3.6.0/lib/rspec/core/example.rb:342:in `call'",
    [115] "/Users/feng.zhang/.rbenv/versions/2.3.3/lib/ruby/gems/2.3.0/gems/rspec-core-3.6.0/lib/rspec/core/hooks.rb:607:in `run_around_example_hooks_for'",
    [116] "/Users/feng.zhang/.rbenv/versions/2.3.3/lib/ruby/gems/2.3.0/gems/rspec-core-3.6.0/lib/rspec/core/hooks.rb:464:in `run'",
    [117] "/Users/feng.zhang/.rbenv/versions/2.3.3/lib/ruby/gems/2.3.0/gems/rspec-core-3.6.0/lib/rspec/core/example.rb:457:in `with_around_example_hooks'",
    [118] "/Users/feng.zhang/.rbenv/versions/2.3.3/lib/ruby/gems/2.3.0/gems/rspec-core-3.6.0/lib/rspec/core/example.rb:500:in `with_around_and_singleton_context_hooks'",
    [119] "/Users/feng.zhang/.rbenv/versions/2.3.3/lib/ruby/gems/2.3.0/gems/rspec-core-3.6.0/lib/rspec/core/example.rb:251:in `run'",
    [120] "/Users/feng.zhang/.rbenv/versions/2.3.3/lib/ruby/gems/2.3.0/gems/rspec-core-3.6.0/lib/rspec/core/example_group.rb:627:in `block in run_examples'",
    [121] "/Users/feng.zhang/.rbenv/versions/2.3.3/lib/ruby/gems/2.3.0/gems/rspec-core-3.6.0/lib/rspec/core/example_group.rb:623:in `map'",
    [122] "/Users/feng.zhang/.rbenv/versions/2.3.3/lib/ruby/gems/2.3.0/gems/rspec-core-3.6.0/lib/rspec/core/example_group.rb:623:in `run_examples'",
    [123] "/Users/feng.zhang/.rbenv/versions/2.3.3/lib/ruby/gems/2.3.0/gems/rspec-core-3.6.0/lib/rspec/core/example_group.rb:589:in `run'",
    [124] "/Users/feng.zhang/.rbenv/versions/2.3.3/lib/ruby/gems/2.3.0/gems/rspec-core-3.6.0/lib/rspec/core/example_group.rb:590:in `block in run'",
    [125] "/Users/feng.zhang/.rbenv/versions/2.3.3/lib/ruby/gems/2.3.0/gems/rspec-core-3.6.0/lib/rspec/core/example_group.rb:590:in `map'",
    [126] "/Users/feng.zhang/.rbenv/versions/2.3.3/lib/ruby/gems/2.3.0/gems/rspec-core-3.6.0/lib/rspec/core/example_group.rb:590:in `run'",
    [127] "/Users/feng.zhang/.rbenv/versions/2.3.3/lib/ruby/gems/2.3.0/gems/rspec-core-3.6.0/lib/rspec/core/example_group.rb:590:in `block in run'",
    [128] "/Users/feng.zhang/.rbenv/versions/2.3.3/lib/ruby/gems/2.3.0/gems/rspec-core-3.6.0/lib/rspec/core/example_group.rb:590:in `map'",
    [129] "/Users/feng.zhang/.rbenv/versions/2.3.3/lib/ruby/gems/2.3.0/gems/rspec-core-3.6.0/lib/rspec/core/example_group.rb:590:in `run'",
    [130] "/Users/feng.zhang/.rbenv/versions/2.3.3/lib/ruby/gems/2.3.0/gems/rspec-core-3.6.0/lib/rspec/core/runner.rb:118:in `block (3 levels) in run_specs'",
    [131] "/Users/feng.zhang/.rbenv/versions/2.3.3/lib/ruby/gems/2.3.0/gems/rspec-core-3.6.0/lib/rspec/core/runner.rb:118:in `map'",
    [132] "/Users/feng.zhang/.rbenv/versions/2.3.3/lib/ruby/gems/2.3.0/gems/rspec-core-3.6.0/lib/rspec/core/runner.rb:118:in `block (2 levels) in run_specs'",
    [133] "/Users/feng.zhang/.rbenv/versions/2.3.3/lib/ruby/gems/2.3.0/gems/rspec-core-3.6.0/lib/rspec/core/configuration.rb:1894:in `with_suite_hooks'",
    [134] "/Users/feng.zhang/.rbenv/versions/2.3.3/lib/ruby/gems/2.3.0/gems/rspec-core-3.6.0/lib/rspec/core/runner.rb:113:in `block in run_specs'",
    [135] "/Users/feng.zhang/.rbenv/versions/2.3.3/lib/ruby/gems/2.3.0/gems/rspec-core-3.6.0/lib/rspec/core/reporter.rb:79:in `report'",
    [136] "/Users/feng.zhang/.rbenv/versions/2.3.3/lib/ruby/gems/2.3.0/gems/rspec-core-3.6.0/lib/rspec/core/runner.rb:112:in `run_specs'",
    [137] "/Users/feng.zhang/.rbenv/versions/2.3.3/lib/ruby/gems/2.3.0/gems/rspec-core-3.6.0/lib/rspec/core/runner.rb:87:in `run'",
    [138] "/Users/feng.zhang/.rbenv/versions/2.3.3/lib/ruby/gems/2.3.0/gems/rspec-core-3.6.0/lib/rspec/core/runner.rb:71:in `run'",
    [139] "/Users/feng.zhang/.rbenv/versions/2.3.3/lib/ruby/gems/2.3.0/gems/rspec-core-3.6.0/lib/rspec/core/runner.rb:45:in `invoke'",
    [140] "/Users/feng.zhang/.rbenv/versions/2.3.3/lib/ruby/gems/2.3.0/gems/rspec-core-3.6.0/exe/rspec:4:in `<top (required)>'",
    [141] "/Users/feng.zhang/.rbenv/versions/2.3.3/lib/ruby/gems/2.3.0/gems/activesupport-5.0.3/lib/active_support/dependencies.rb:287:in `load'",
    [142] "/Users/feng.zhang/.rbenv/versions/2.3.3/lib/ruby/gems/2.3.0/gems/activesupport-5.0.3/lib/active_support/dependencies.rb:287:in `block in load'",
    [143] "/Users/feng.zhang/.rbenv/versions/2.3.3/lib/ruby/gems/2.3.0/gems/activesupport-5.0.3/lib/active_support/dependencies.rb:259:in `load_dependency'",
    [144] "/Users/feng.zhang/.rbenv/versions/2.3.3/lib/ruby/gems/2.3.0/gems/activesupport-5.0.3/lib/active_support/dependencies.rb:287:in `load'",
    [145] "/Users/feng.zhang/.rbenv/versions/2.3.3/lib/ruby/gems/2.3.0/gems/spring-commands-rspec-1.0.4/lib/spring/commands/rspec.rb:18:in `call'",
    [146] "/Users/feng.zhang/.rbenv/versions/2.3.3/lib/ruby/2.3.0/rubygems/core_ext/kernel_require.rb:55:in `require'",
    [147] "/Users/feng.zhang/.rbenv/versions/2.3.3/lib/ruby/2.3.0/rubygems/core_ext/kernel_require.rb:55:in `require'",
    [148] "-e:1:in `<main>'"
]
ERR unknown command 'script' excluded from capture: Not configured to send/capture in environment 'test'
  Rendering errors/500.html.erb within layouts/application
  Rendered errors/500.html.erb within layouts/application (0.1ms)
  Template rendering was prevented by rspec-rails. Use `render_views` to verify re

Question / feature request: abort block execution if ttl reached

Given we use a gem like this to prevent the same block of code executing simultaneously, I think the .lock method should at least accept an option to abort the code execution if the ttl is reached.

As per this comment, "You have to make sure that your code won't take longer than TTL".

Right now, if the ttl is reached, the lock is released, but the current executing block is not halted, which means, as per that very comment, "all hell will be lose" ... and I agree it will, because code WILL run simultaneously, since the first code block is still running and we released the lock to allow for other code blocks to start their turn.

So my point is: shouldn't the gem help us preventing that all hell breaks lose? 😅 And maybe even make that the default behavior?

Maybe something like this:

block_result = lock_manager.lock!("resource_key", 2000, abort_if_expired: true) do
    sleep 3
    # Will raise an error because it's duration (3000 ms) is greater than the lock is held for (2000)
  end

Maybe even make that the default behavior? I don't understand what I'm missing here, because this is a very popular gem, linked from the official redis docs, and I think it would be very unsafe to use it without having a way to abort the block's execution if the lock gets released.

I could even try a PR, but I honestly didn't understand the extending a lock part. Are we supposed to rely on the validity key of the .lock hash and use it as a timer or something? 😬

Incompatibilities with fakeredis

@leandromoreira Is there a strong reason or preference for using the Redis::Client#call methods directly for the :set and :eval commands? Specifically these:

redlock/client.rb

def lock(resource, val, ttl)
  @redis.client.call([:set, resource, val, 'NX', 'PX', ttl])
end

def unlock(resource, val)
  @redis.client.call([:eval, UNLOCK_SCRIPT, 1, resource, val])
  # ...
end

I ask because this method of using the Client#call directly is causing compatibility issues with the fakeredis gem. Specifically, the #lock call is returning 0 when used with fakeredis, but it returns nil when used with a real Redis connection.

To be clear, this is entirely a bug with fakeredis, but I could not fix it by simply having fakeredis return nil instead of 0 when commands are falsy. That change ended up with dozens of red specs within fakeredis.

I propose this change to redlock-rb which is functionally equivalent, and builds green. It could also be argued that it is a more appropriate way to interact with the Redis gem, since I can find very very few cases of code that uses Redis::Client#call directly, but instead works with the Redis#set, Redis#eval, Redis#xxxx methods.

--- a/lib/redlock/client.rb
+++ b/lib/redlock/client.rb
@@ -77,11 +77,11 @@ module Redlock
       end

       def lock(resource, val, ttl)
-        @redis.client.call([:set, resource, val, 'NX', 'PX', ttl])
+        @redis.set(resource, val, nx: true, px: ttl)
       end

       def unlock(resource, val)
-        @redis.client.call([:eval, UNLOCK_SCRIPT, 1, resource, val])
+        @redis.eval(UNLOCK_SCRIPT, [resource], [val])
       rescue
         # Nothing to do, unlocking is just a best-effort attempt.
       end

Thoughts? I will happily submit a PR if you agree.

Unlock without the lock info

Hi there,

I'm using the gem in a distributed infra including multiple APIs where one endpoint would lock and a different endpoint would later unlock.

Is that possible to unlock with just providing the lock name and not the entire hash? I tried to test that but it didn't seem to work.

Thanks

Does redlock support Redis Cluster?

Hi,

I am trying to use redlock with redis with cluster mode enabled. The redlock client doesn't currently handle MOVED errors which are thrown by redis in cluster mode.

Does redlock support redis cluster? If not, is there a way around this or do you know of any other potential solutions for locking with a redis cluster?

Thanks!

doesn't recover from NOSCRIPT

Redis::CommandError: NOSCRIPT No matching script. Please use EVAL.

it should re-run load_script

lock_info not passed to lock!() block

the doc says:

begin
  block_result = lock_manager.lock!("resource_key", 2000) do |lock_info|
    # critical code
    lock_manager.lock("resource key", 3000, extend: lock_info, extend_life: true)
    # more critical code, only if lock was still hold
  end
rescue Redlock::LockError
  # error handling
end

but i'm getting always nil into lock_info, so the lock is not extended...
maybe to culprit is a yield in lock!() that does not pass the lock_info got from lock().

can you confirm?

Lock! Empty Resource Key Behavior (question)

Hi there,

I have a method that may or may not need to lock based on a few conditions. I've tried moving all of that logic to a separate method which takes a few arguments. I can call that method directly with no problem, however, I'm unable to turn that method into a block to call lock! since the method takes args. If that's possible, I'd love to hear how.

But I was curious about the resource key. If that were to be conditionally set, either to a unique string or empty string/nil/etc. What would the behavior be? Would it lock on the unique string and not acquire a lock for the empty (which would be ideal). Or would there still be an acquired lock for ''.

Thanks for any insight you can provide.

Example:

class JobClass
  def perform
    if some_value == some_other_value
      return lock!('resource_key', ttl: 2000) do
        # How can I pass `run_job` with the args it needs to this block
      end
    end

    # This is the easy case (calling the run method directly)
    run_job(ActiveRecordInstance, '', 0)
  end

  def run_job(object, string, number)
    # critical code here
  end
end

Administrative commands cannot be effectively namespaced deprecation warning

Warning:

Passing 'script' command to redis as is; administrative commands cannot be effectively namespaced and should be called on the redis connection directly; passthrough has been deprecated and will be removed in redis-namespace 2.0 (at /my-app-name-dir/shared/bundle/ruby/2.5.0/gems/redlock-0.2.2/lib/redlock/client.rb:128:in `load_scripts')

Related issues in redis-namespace repository:

• resque/redis-namespace#111
• resque/redis-namespace#132

Latest redis client not supported

• As subject
• The latest (5.x) version of redis is not supported - https://github.com/redis/redis-rb
• It "should" be enough to update the dependencies.

Feature suggestion: resource locked and time left on lock

Feature suggestion:
Method that tells you if a resource is locked and how much time is left on the lock without having to lock the actual resource.

Wrong exception class in README

The README has an example with rescue Redlock::LockException, but this class doesn't exist, it is should be rescue Redlock::LockError.

TTL question

What happens if TTL < (retry_count * retry_delay)?

Error: "No matching script. Please use EVAL."

It seems that redlock v2.0.0 is possibly broken when used with redis 4.8.1 (the redis gem, not redis-server).

When attempting to lock, I'm getting "Redis::CommandError: NOSCRIPT No matching script. Please use EVAL."

Full exception

Redis::CommandError: NOSCRIPT No matching script. Please use EVAL.
redis (4.8.1) lib/redis/client.rb:162:in `call'
redis (4.8.1) lib/redis.rb:270:in `block in send_command'
redis (4.8.1) lib/redis.rb:269:in `synchronize'
redis (4.8.1) lib/redis.rb:269:in `send_command'
redis (4.8.1) lib/redis/commands.rb:206:in `call'
redlock (2.0.0) lib/redlock/client.rb:171:in `block in lock'
redlock (2.0.0) lib/redlock/client.rb:210:in `recover_from_script_flush'
redlock (2.0.0) lib/redlock/client.rb:170:in `lock'
redlock (2.0.0) lib/redlock/client.rb:260:in `block (2 levels) in lock_instances'
redlock (2.0.0) lib/redlock/client.rb:260:in `select'
redlock (2.0.0) lib/redlock/client.rb:260:in `block in lock_instances'
redlock (2.0.0) lib/redlock/client.rb:313:in `timed'
redlock (2.0.0) lib/redlock/client.rb:259:in `lock_instances'
redlock (2.0.0) lib/redlock/client.rb:234:in `block in try_lock_instances'
redlock (2.0.0) lib/redlock/client.rb:230:in `times'
redlock (2.0.0) lib/redlock/client.rb:230:in `try_lock_instances'

Does redlock need to require 'redis', '>= 5.0.0' ?
(If so, redlock 2.0.0 might need to be yanked to avoid breaking apps currently using redis gem < 5.)

Using redlock 1.3.2 works correctly.

Versions

redis-server 6.2.10
redlock 2.0.0
redis (gem) 4.8.1
rails 7.0.4.2

It Do not lock the resource nor it stores anything to Redis

1. I'm opening 2 rails consoles that are using the same redis instance
2. I run redis-cli and run the monitor command

It doesn't seems to lock, and it doesn't seems to store into redis. I'm I missing something?

rails console #1

rl = Redlock::Client.new([ "redis://127.0.0.1:6397" ])
me = rl.lock("lellisga", 2000000)
#=> {:validity=>1979998, :resource=>"lellisga", :value=>"1b5e0a18-638a-4a45-9be9-487d401acf00"}

rails console #2

rl = Redlock::Client.new([ "redis://127.0.0.1:6397" ])
me = rl.lock("lellisga", 2000000)
#=> {:validity=>1979998, :resource=>"lellisga", :value=>"db238b28-d82a-47e6-b404-1dd72eb714ff"}
# I was expecting this to return `false`

redis-cli monitor

redis 127.0.0.1:6379> MONITOR
OK
# nothing happens

Uses monotonic time

I was reading Antirez 101 and it came to my mind that we're using Time.now and we should be using a monotonic clock and I was looking this implementation that works with cruby, jruby and etc. Thoughts?