axonframework / extension-jgroups

Axon Framework extension for JGroups integration to distribute Command messages.

Axon Framework - JGroups Extension

Axon Framework is a framework for building evolutionary, event-driven microservice systems, based on the principles of Domain Driven Design, Command-Query Responsibility Segregation (CQRS) and Event Sourcing.

As such it provides you the necessary building blocks to follow these principles. Building blocks like Aggregate factories and Repositories, Command, Event and Query Buses and an Event Store. The framework provides sensible defaults for all of these components out of the box.

This set up helps you create a well structured application without having to bother with the infrastructure. The main focus can thus become your business functionality.

This repository provides an extension to the Axon Framework: JGroups. It provides functionality to distribute command message between Axon application through means of a JGroups implementation of the CommandRouter and CommandBusConnector. This extension should be regarded as a partial replacement of Axon Server, since it only cover the command routing part.

For more information on anything Axon, please visit our website, http://axoniq.io.

Getting started

The AxonIQ Library contains a section for the guides of all the Axon Framework extensions. The JGroups extension guide can be found here.

Receiving help

Are you having trouble using the extension? We'd like to help you out the best we can! There are a couple of things to consider when you're traversing anything Axon:

• Checking the reference guide should be your first stop, as the majority of possible scenarios you might encounter when using Axon should be covered there.
• If the Reference Guide does not cover a specific topic you would've expected, we'd appreciate if you could post a new thread/topic on our library fourms describing the problem.
• There is a forum to support you in the case the reference guide did not sufficiently answer your question. Axon Framework and Server developers will help out on a best effort basis. Know that any support from contributors on posted question is very much appreciated on the forum.
• Next to the forum we also monitor Stack Overflow for any questions which are tagged with axon.

Feature requests and issue reporting

We use GitHub's issue tracking system for new feature request, extension enhancements and bugs. Prior to filing an issue, please verify that it's not already reported by someone else.

When filing bugs:

• A description of your setup and what's happening helps us figuring out what the issue might be
• Do not forget to provide version you're using
• If possible, share a stack trace, using the Markdown semantic ```

When filing features:

• A description of the envisioned addition or enhancement should be provided
• (Pseudo-)Code snippets showing what it might look like help us understand your suggestion better
• If you have any thoughts on where to plug this into the framework, that would be very helpful too
• Lastly, we value contributions to the framework highly. So please provide a Pull Request as well!

extension-jgroups's Issues

Migrate tests to JUnit 5

Enhancement Description

Currently this project is still running on JUnit4 while JUnit5 is out there.
We should migrate it to the new version.

Current Behaviour

Project is using JUnit4.

Wanted Behaviour

Use JUnit5.

Possible Workarounds

N/A

Enable tracing in JGroupsConnector with SpanFactory

Feature Description

Axon Framework 4.6.0 has seen the introduction of integrated tracing through Axon Framework issue #2294.
This issue did not cover the JGroupsConnector, though.

Current Behaviour

The integrated tracing support is not in place for the JGroupsConnector.

Wanted Behaviour

Introduce the SpanFactory in the JGroupsConnector to trace it's processes.

Set Exception Details in Command Result

In issue #1079, the option was introduced to add application specific exception details back to another node.
The Spring Cloud extension should be adapted to utilize this mechanic too.

NPE when JoinMessages arrive quickly after connecting

When a node connects, there is a brief time frame in which a node is connected to the cluster, but doesn't have an ExecutorService instance, yet. If, in that time frame, a node managed to send a JoinMessage, this results in a NullPointerException.
While these will probably not impact general operation, as the cluster will recover from this issue automatically, it does pollute logs.

Callbacks never called if connection dies while a command is outstanding

If the connection to a JGroups peer dies while a command is executing remotely on that peer, none of the application's callbacks are ever called, including MessageMonitors.

The problem appears to be that when a member vanishes from the view, JGroupsConnector removes it from the consistent hash, but doesn't scan the CommandCallbackRepository looking for callbacks to invoke. (And in fact, the entries in that repository will remain there forever.)

Consistent hash can get out of sync with JGroups

This is pretty obscure, but bit us thanks to an AWS problem that caused one of our EC2 instances to get suspended for a couple minutes then wake up unaware that anything had happened. We saw it on Axon 2.4.5 but it seems like Axon 3 would behave the same. I believe this could also happen due to network outages if the timing is just right.

The sequence:

1. Nodes A and B are in a JGroups cluster with a distributed command bus.
2. A goes to sleep or loses network connectivity.
3. B times out waiting for A to respond to heartbeats, and marks A as dead. JGroupsConnector removes A from the consistent hash.
4. A wakes up or, in the case of a network issue, regains network connectivity before the JGroups timer that checks for missing heartbeats would have fired for the last time.
5. At this point, A thinks it is in a cluster [A,B] and B thinks it is in a cluster [B].
6. JGroups discovery runs, and B discovers that A is alive. The JGroups view on B is updated.
7. B sends a JoinMessage to A, advertising the commands it can handle.
8. A receives the message. Since it had never removed B from its consistent hash, the hash doesn't change as a result of the message.
9. At this point, JGroups on both nodes thinks the cluster is [A,B]. The consistent hash on A has entries for A and B. But since A has had no reason to send a JoinMessage to B, the consistent hash on B only has entries for B.

The inconsistency remains until the application is restarted on one of the hosts.

The fix I came up with is to intercept the incoming JoinMessage and attempt to apply it to the local consistent hash. If the hash is the same before and after applying the data from the message, I send a JoinMessage in response. (Actually I send a subclass of JoinMessage, to avoid the risk of two nodes bouncing redundant messages back and forth forever.)

In the above example, that'd be triggered in step 8 and would cause node A to send a JoinMessage (subclass instance) back to node B, which would cause both nodes to have the same consistent hash.