Comments (18)
PRs are up:
from vertx-redis-client.
Thanks for reporting this @Dieken
@Ladicek would you have some time to take care of this?
from vertx-redis-client.
So there are 2 ways how the Redis client may be used:
- obtain the
Redis
client and use itssend()
andbatch()
methods; - obtain the
Redis
client, from there obtain aRedisConnection
usingconnect()
, and use itssend()
andbatch()
methods.
The first approach has an obvious disadvantage: for each command, it obtains a new connection. Not a TCP connection, those are pooled, but a connection object that may holds important state. In case of clustered Redis, that state held in the connection object includes the hash slot distribution. So using the first approach, every command indeed requires first sending CLUSTER SLOTS
-- when you use the second approach, that is not the case.
(Looking at the code, I can see that the clustered client handles ASK redirections, but it seems that a MOVED redirection just ends up as an error, and it is a responsibility of the caller to close the connection and reconnect. I might be reading the code wrong, though. I'd have to explore more to be confident.)
I don't know if it is the Quarkus Redis client who doesn't bother maintaining a RedisConnection
and always uses the 1st approach as described above, or whether the Quarkus Redis client exposes both approaches and it is the user who needs to decide. @Dieken could you please share your Quarkus code so I can check more? Thanks!
from vertx-redis-client.
The code is hard to extract, let me describe the call chain:
@Inject ReactiveRedisDataSource
creates ReactiveSortedSetCommandsImpl
and ReactiveKeyCommandsImpl
instances, all methods in these two classes call AbstractRedisCommands.execute()
.
AbstractRedisCommands.execute()
ReactiveRedisDataSourceImpl.execute()
@Override
public Uni<Response> execute(Request request) {
if (connection != null) {
return connection.send(request);
}
return redis.send(request); // <-- reach here
}
ObservableRedis.send()
RedisClusterClient.send()
BaseRedisClient.send()
RedisClusterClient.connect() // reach second way above
So each command from ReactiveSortedSetCommandsImpl
and ReactiveKeyCommandsImpl
triggers a call to RedisClusterClient.connect()
which obtains a pooled RedisConnection
and sends CLUSTER SLOTS
command before the real original command.
I use Quarkus-3.3.0.
from vertx-redis-client.
Yeah, I'm just looking at the Quarkus implementation too. I have never seen that code before, so I might be wrong, but it seems to me that if you use ReactiveRedisDataSource.withConnection(callback)
, the Redis data source first obtains a Vert.x RedisConnection
and uses it to execute all the Redis commands in the callback. Redis commands made simply on the injected ReactiveRedisDataSource
are executed on the Vert.x Redis
object, so for each of them, a RedisConnection
is created. This is unfortunately not documented in https://quarkus.io/guides/redis-reference
So in the short term, you can use either withConnection()
if that works for you, or you can inject the Redis client and use that manually (somewhat lower-level).
Longer-term, I don't really know what's the Vert.x Redis client philosophy. Are the Redis.send()
/ batch()
method a mere convenience mostly for throwaway code and real-world code should obtain a RedisConnection
and use it for all operations (maybe reconnecting in case of transient errors)? If so, the Quarkus Redis extension is at odds with that. Or are the Redis.send()
/ batch()
methods the primary way the Redis client should be used and RedisConnection
should mostly be obtained just for special cases (like Redis transactions)? If so, the RedisConnection
objects holding client-wide state is wrong and the state should be held in Redis
objects instead. Or are these 2 APIs equivalent? That's probably the same situation as the previous one -- client-wide state held in connection objects is still wrong.
Looking at the various Redis*Connection
implementations, it seems that the only one holding client-wide state is RedisClusterConnection
and the only client-wide state it holds is the hash slot assignment. So maybe we should move that to RedisClusterClient
, but I'd first like to hear some expert's opinion on the previous paragraph :-) Maybe @pmlopes or @cescoffier?
from vertx-redis-client.
The high level ReactiveXxxCommands API has no chance to use withConnection() or batch(). I think it’s OK to obtain pooled connection for each command, the problem is Slots data should belong to RedisClusterClient as you described above, and the Slots data should be cached, never send “CLUSTER SLOTS” first for each Redis command.
from vertx-redis-client.
You said you're @Inject
-ing a ReactiveRedisDataSource
, which has the withConnection()
method. So it certainly is available to you. It probably isn't useful to you, but that's hard to know for sure without seeing the code. Hence my other recommendation to descend to the direct Vert.x API (perhaps in the Mutiny variant), which is also available and provides more control.
I overall agree that the hash slot assignment should probably be stored in the RedisClusterClient
, and I'll see what would it take to do it, but I'd like to hear from the experts too.
from vertx-redis-client.
I don't directly use the low level API ReactiveRedisDataSource
, I use the high level API ReactiveXxxCommands
which is created from ReactiveRedisDataSource
, the high level API doesn't have withConnection()
method.
from vertx-redis-client.
ReactiveRedisDataSource
is not a low-level API, it is the Quarkus high-level API. You said that you use @Inject ReactiveRedisDataSource
-- that's what I mean. There's a withConnection()
method there.
When I'm referring to low-level APIs, I'm talking about the Vert.x APIs (Redis
, RedisConnection
).
from vertx-redis-client.
My bad, the Quarkus guide does say ReactiveRedisDataSource
is high level API: https://quarkus.io/guides/redis-reference#apis
from vertx-redis-client.
Checked the ReactiveRedisDataSource.withConnection()
API again, if I create RedisXXXCommands instance in withConnection(), not in constructor of ReactiveEasy resource class as Quarkus Redis guide describes, the uneven load will be proportional to calls of withConnection not number of Redis command, that’s better but uneven load still exists, the Slots data still should be cached.
from vertx-redis-client.
Yeah, I suspected as much. I'll see what I can do.
from vertx-redis-client.
I am wondering if such redis connection should not be pooled as well by the vertx redis client, WDYT @pmlopes ?
from vertx-redis-client.
TCP connections to the Redis server(s) are pooled. The RedisConnection
object is more like a "client session" or something, the name is slightly misleading.
from vertx-redis-client.
I'm trying to reproduce this locally, but I'm hitting a connection pooling issue (basically connection pooling doesn't seem to work at all) on the 4.x branch (master
works fine), need to figure that out first. Just to let everyone know I'm working on this, though it might take a few days :-)
from vertx-redis-client.
OK, so the connection pooling not working at all is #365 (and the fix is #374). This is currently scheduled for 5.0, which in my opinion is wrong -- we totally need to fix this in 4.x, otherwise Redis cluster is basically unusable. I can't see any possible downside to backporting the fix, it's just about adding equals
/hashCode
to 2 classes that are part of the key used to loookup the Endpoint
(= connection pool) from ConnectionManager
.
from vertx-redis-client.
OK, so once I actually make the cluster client behave, this issue is fairly easy to reproduce with this simple Vert.x application:
public class MainVerticle extends AbstractVerticle {
public static void main(String[] args) {
Vertx vertx = Vertx.vertx();
vertx.deployVerticle(new MainVerticle());
}
@Override
public void start(Promise<Void> startPromise) {
Redis redis = Redis.createClient(vertx, new RedisOptions()
.setType(RedisClientType.CLUSTER)
.setUseReplicas(RedisReplicas.SHARE)
.setConnectionString("redis://192.168.1.171"));
RedisAPI client = RedisAPI.api(redis);
call(client, "foo", 0);
call(client, "bar", 0);
call(client, "baz", 0);
call(client, "qux", 0);
call(client, "quux", 0);
call(client, "corge", 0);
call(client, "grault", 0);
call(client, "garply", 0);
call(client, "waldo", 0);
call(client, "fred", 0);
call(client, "plugh", 0);
call(client, "xyzzy", 0);
call(client, "thud", 0);
}
private void call(RedisAPI client, String prefix, int last) {
if (last == 1_000_000) {
return;
}
int x = last + 1;
client.set(List.of(prefix + x, "" + x))
.flatMap(response -> {
return client.get(prefix + x);
}).flatMap(response -> {
System.out.println(prefix + " --> " + response);
return client.set(List.of("__last__" + prefix, "" + x));
}).onSuccess(response -> {
vertx.runOnContext(ignored -> call(client, prefix, x));
}).onFailure(error -> {
System.out.println(error);
});
}
}
I'm using a simple 3-node cluster with 3 masters and no replicas. I run an actual virtual machine with 1 CPU and 1024 MB of RAM for each cluster node, so I can actually measure load average and process CPU consumption.
I made an interesting observation: when I tried to even the load on the cluster nodes by issuing the CLUSTER SLOTS
command to a random node (instead of always the first), I saw slightly increased load on the other nodes, but the first node had even higher load than before. This suggests that the other nodes forward the command to the first one (or something like that), but I'm no Redis cluster expert and didn't really look into the code. Anyway, this doesn't seem to be a good strategy.
Next, I'll look into storing the hash slot assignment in the RedisClusterClient
as mentioned above.
from vertx-redis-client.
OK, storing hash slot assignment in the RedisClusterClient
seems to work well. I'll submit a PR against master
.
I'll also send 2 PRs to 4.x: backport of #374 and a backport of the fix for this issue.
from vertx-redis-client.
Related Issues (20)
- Pick the protocol from the url if provided HOT 1
- Create too many connections to server HOT 4
- Cluster mode (MOVED Exception) HOT 3
- The vertx redis reconnection demo introduced at docs cannot work in the case when redis instances shutdown and get back to ready. HOT 4
- Align Redis deprecations with RedisAPI methods
- hight cpu usage after pressure test HOT 6
- redis client shared in multiple verticles in high concurrency load will make redis call get response belonged to another request with some specific keys HOT 7
- Seeing unhandled exception when client reacts to sentinel failover HOT 2
- Unexpected behavior of UNSUBSCRIBE with cluster mode HOT 5
- Getting io.vertx.core.impl.NoStackTraceThrowable: Cannot connect to any of the provided endpoints Error while trying to connect to redis in replication mode HOT 4
- which situation will make RedisStandaloneConnection message handler a null value HOT 1
- always have the "Redis waiting Queue is full" error
- Missing connection with endpoint fallback in cluster connection HOT 2
- Getting different result when executing `FT.SEARCH` when using `redis/redis-stack:7.0.6-RC8` vs `redis/redis-stack-server:7.2.0-v5`. (Multi is a Map) HOT 6
- How to catch io.vertx.core.impl.NoStackTraceThrowable HOT 4
- [RedisConnectionManager] io.netty.channel.StacklessClosedChannelException throwed HOT 1
- Command.XREADGROUP error MOVED HOT 9
- Expose more properties through CommandReporter
- Support for FalkorDB
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from vertx-redis-client.