Swappable Persistence Backends? about centrifuge

[FZambia]

Yep, badger or bolt with raft leader election is an interesting thing to extend current possibilities.

The hardest part of making engine exported at this time is the fact that publish into PUB/SUB should be atomic with saving message to history cache. So publish operation in a whole can be atomic. This works with Redis (to certain degree though, I suppose atomicity of Redis LUA scripts could be not guaranteed when PUB/SUB operations involved), also in Memory engine. But if we split engine on PUB/SUB part and storage part we lose this atomicity. In this case message can be published to current subscribers but not saved in cache, or saved in cache but not published to subscribers. Currently I don't know what should we do in case of these errors at publish time.

Redis has no sync replication so there is a chance to lose messages even with AOF with fsync on every query when master goes down and last messages not achieved slave yet. Having nature of Redis in mind current Redis engine implementation can only be best effort mechanism to provide message persistence for later recovery process. It will work in practice for most real-world apps but there are no strong guarantees as far as I understand.

[FZambia]

Guys, just want to update status on this. I remember about this issue but the task is not straightforward. I have some ideas but they ruin under various edge cases. At moment I am in progress of validating new idea - maybe it will fit fine but can't be sure.

At least I proved that I can make current engines separate if neseccary. Also played a bit with custom Nats engine which implements only PUB/SUB logic without any persistence layer.

[FZambia]

Hello, my plan is making internal engine parts pluggable in the end. Though I am not sure about correct implementation and currently concentrated on finishing work on Centrifugo v2 built on top of this lib.

I don't fully understand your intents - this library should only persist a short-term cache of messages. What is your use case for persistent store? Are you going to keep unlimited amount of messages in it? Also Redis Pub/Sub is not currently used for persistence - it is used for Centrifuge node scalability.

[stemcc]

Great, I'm glad to hear that's on the roadmap.

My use case for persistence is to have a cache that can survive reboots, while ideally using something a little simpler than Redis but that can still provide simple kv lookups (like Bolt for example). Otherwise I would just load a log file into memory when the app comes back online. I'm using Centrifuge to pass messages and data from a local daemon/server to a local gui client, local daemon to remote client, or remote daemon/server to local daemon/server.

Badger or bolt with raft leader election should be enough to get HA.

I think redis is HA in that you can specify how often it syncs to disk ?

Then use the event sourcing pattern. Write to a boltdb bucket. Have another process push it out to how many durable storage engines you have. This is why event sourcing exist imho. Then the HA of your durable storage engines becomes not so important too. As long as the event store is HA you can always recover. So an event store based on boltdb raft is a good match. The db and cache based on whatever. Tons of options with boltdb or badger.

…

On Mon, 17 Sep 2018, 18:23 Alexander Emelin, ***@***.***> wrote: Yep, badger or bolt with raft leader election is an interesting thing to extend current possibilities. The hardest part of making engine exported at this time is the fact that publish into PUB/SUB should be atomic with saving message to history cache. So publish operation in a whole can be atomic. This works with Redis (to certain degree though, I suppose atomicity of Redis LUA scripts could be not guaranteed when PUB/SUB operations involved), also in Memory engine. But if we split engine on PUB/SUB part and storage part we lose this atomicity. In this case message can be published to current subscribers but not saved in cache, or saved in cache but not published to subscribers. Currently I don't know what should we do in case of these errors at publish time. Redis has no sync replication so there is a chance to lose messages even with AOF with fsync on every query when master goes down and last messages not achieved slave yet. Having nature of Redis in mind current Redis engine implementation can only be best effort mechanism to provide message persistence for later recovery process. It will work in practice for most real-world apps but there are no strong guarantees as far as I understand. — You are receiving this because you commented. Reply to this email directly, view it on GitHub <#12 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/ATuCwgnJ-UGqfyGqVsdSeQxKwCXd78-eks5ub8x3gaJpZM4V5QeV> .

[FZambia]

@gedw99 thanks for suggestions. Centrifuge lib in its current state is very close to event sourcing system. At least this is already kind of. Its hard to say at moment in what direction we should expand it - maybe the understanding will come with usage reports if any:)

I guess it's also because your relying on redis to do the hard stuff which is totally understandable.

You could use emitter golang project instead of redis. Then you can wrap it. It uses mqtt as the messaging layer and supports scaling honizontally

[thienpow]

maybe can use mongodb instead of redis

[FZambia]

@stemcc please take a look at referenced pull request where I am trying to split Engine on three parts - Broker, HistoryManager and PresenceManager. This is a road to pluggable components, hope this will allow to do more with this library. At least look at interesting concept example where Nats used for PUB/SUB and Redis for other stuff.

[FZambia]

Let's reopen until it will be documented

[FZambia]

Just posted a note about pluggable components on Medium, also we have an example of setting custom broker implementation. I suppose this is enough for a moment. If there are any questions about current implementation - feel free to ask.