himadieievsv / redpulsar

Distributed locks and other utils, can be used with Jedis client and Lettuce client for Redis

RedPulsar

What is RedPulsar?

RedPulsar provides Distributed Locks with Redis and other utilities for cloud computing or different kinds of distributed systems. It is minimalistic, lightweight, and easy to use library written in Kotlin and currently can be used with both Jedis or Lettuce clients.

Features

• Mutex: Distributed lock mechanism on a resource, that uses consensus of the majority of data storage nodes to determine if check obtained successfully.
• Semaphore: Distributed semaphore implementation allowing multiple number of lock on a resource. It also uses consensus of the majority of data storage nodes to determine if check obtained successfully.
• SimplifiedMutex: Simplified distributed lock mechanism on a resource. Unlike Mutex it uses single data storage node.
• ListeningCountDownLatch: Implementation of distributed Count Down Latch, it uses that uses consensus of the majority of data storage instances ensuring count down consistency. ListeningCountDownLatch utilized Redis Pub/Sub mechanism to notify waiting workloads about count reaching zero.

Supporting data storages

Currently, RedPulsar supports Redis as a data storage. It can be used with both Jedis or Lettuce clients.

• See redpulsar-jedis module for details.
• See redpulsar-lettuce module for details. This module also provides convenient lettuce pooled clients.

Java compatibility

Minimal required Java version is 11. RedPulsar project is written in Kotlin, but can be easily used in Java projects too.

Getting started

Gradle dependency:

implementation("com.himadieiev:redpulsar-jedis:1.2.0")
// OR
implementation("com.himadieiev:redpulsar-lettuce:1.2.0")

• See Getting started with redpulsar-jedis for getting started with Jedis client.
• See Getting started with redpulsar-lettuce for getting started with Lettuce client.

Development

To build RedPulsar locally, you need to have JDK 11+ installed. To build or test RedPulsar, run the following command:

git clone [email protected]:himadieievsv/redpulsar.git
cd redpulsar

# Code formatting
./gradlew ktlintFormat

# Run all tests
docker-compose up -d
./gradlew test 

# Run only unit tests
./gradlew test -DexcludeTags="integration"

# Build
./gradlew build -x test

# Publish to local maven repository
./gradlew publishToMavenLocal \
  -Psigning.secretKeyRingFile=... \
  -Psigning.password=... \
  -Psigning.keyId=...

Further development

Extending RedPulsar to use other data stores

Currently, all features are implemented with Redis. However, it is possible to extend RedPulsar to use other distributed data stores like AWS DynamoDB / Casandra / ScyllaDB etc. Even it could be implemented with RDBMS like MySQL or PostgreSQL. RedPulsar project have an abstraction level for data storage called Backend. See package com.himadieiev.redpulsar.core.locks.abstracts.backends for details what particular operation should be implemented. New data storage should use a new module and implement same abstractions as current Redis implementations.

Contributing

Contributions are welcome! Please make sure to create Issue first before working on improvements o new features, feel free to submit a Pull Request from a project fork.

Furthers plans

• Add ListenerLock using Pub/Sub mechanism instead of polling.
• Add FairLock implementation. It supposed to be a lock that grants lock to the longest waiting thread.
• Leader election mechanisms.
• Service discovery service.
• etc.

redpulsar's Issues

ListenerLock implementation

Summary
It nice to have ListenerLock in oppose to polling locks.
Basic requirements:

1. Utilizing Pub/Sub mechanism
2. Still should try to lock with by sending lock call to backend first and then listen for unlock event.

Motivation
Reduce calls to backend attempting to lock resource

It nice to have FairLock

Summary
It nice to have FairLock.
Basic requirements:

1. Polling mechanism to obtain lock
2. First client who tried to lock resource should be able to lock resource after its unlock

Motivation
FairLock is attempt to have fairness in lock obtaining racing, if client first requested lock it should be able obtain lock as soon as lock is available.

issue in multiInstanceExecute

multiInstanceExecute can get result properly with strategy waitForMajority.
There are multiple unit tests failures due to that

Load script prior to commands execution on Redis

Summary
Load LUA script prior to commands execution on Redis.
Initialization can be done when object created once per application.

Motivation
All Redis backends are sending LUA scripts as is. Those calls ending up with slitty bigger size.
Reduce that redundancy with using script hash to call it.

Caveats

• Tracking what Redis instances already has scripts might be challenging if cluster topology changes or individual instance get restarted/replaced.

Provide Leader election algorithm

Summary
It nice to have Leader election algorithm too.
Basic requirements:
TBD

Motivation
Currently it is possible to have only one node in luster to perform workload using locks, but it have many drawbacks for using that approach in cluster with different types of nodes (computing power, network bandwidth, memory capacity, etc ). Ideally nodes should vote on the best candidate.

Workaround
Use locks