CallbackHandler async re-subscribe watcher can potentially miss events about helix

[jiajunwang]

The re-subscribe does not rely on the async logic. It is by default done in the ZkClient.
The real problem is the race condition when updating the Helix cache. This prevents the controller from re-calculating the new mapping. Will update with more detail next week.

[jiajunwang]

The root cause of the issue is a race condition in Helix cache refresh logic. Basically, Helix relies on the ZK notification path to set up 2 things:

1. If the current assignment is still valid.
2. If the data cache needs to be refreshed.

The race condition causes these 2 checks to have a different result. The assignment was not invalidated. But the cache was updated. As a result, the rebalancer refused to calculate a new mapping even knows the latest IdealState.

The impact is limited in the CUSTOM rebalance mode only. Because the fist check is done for custom rebalancer only.

[jiajunwang]

@Jackie-Jiang Could you please help to take a look at the proposed fix? Thanks.

[jiajunwang]

Close this ticket since the fix has been merged to master. We will have a release soon next week with one more critical fix.