Track subscriptions by key about cacherpc

[bmuddha]

Here's the brief description of how subscription mechanism works or supposed to work (at least in my understanding)

1. The request is received by cacherpc instance to perform some remote procedure call
2. If the request is getAccountInfo or getProgramAccounts, cacherpc handles it, otherwise the request is routed through to validator
3. For the public key in request system should try to find an active web socket subscription (currently logic for that is broken),
   presence of active subscription would indicate that the entry in cache is being maintained in up to date state, absence of it means that we have never cached data for that key before or that web socket subscription has been terminated and data in cache might be stale.
4. If subscription is present, then we can try to get data from cache and return it immediately
5. If subscription is not present or the data itself is not present in cache then we forward the request to validator
6. Then there's some magic with concurrent async code...
7. After receiving response from validator, put the data into cache, and initiate subscription for that request, to keep it in sync.
8. Finally system returns the response

So currently, instead of checking whether we have an active subscription in step 3, we just compare the number of confirmed subscriptions with the number of initiated subscriptions (but not necesserally confirmed), which leads to the problem described in issue, that even if one the subscriptions, managed by some worker, isn't confirmed, then none of the requests that should be handled by this worker will get answered from cache, until all its subscriptions are confirmed.

So proposed solution is to map public keys with subscriptions (that's a logical thing to do). So here's one of the ways to implement it (admittedly not the elegant one):

• There's already a mapping between public keys and subscriptions, but not a direct one:

  pub struct AccountUpdateManager {
      ...
      sub_to_id: HashMap<(Subscription, Commitment), u64>, // should contain only active confirmed subscriptions
      connection: Connection,
      ...
  }

So the presence of the key (tuple of Subscription and Commitment) in sub_to_id field is an indication of existence of active web socket subscription, but here's the catch:
Subscription is an enum which has two variants (for now), as well as Commitment (3 variants), so in order to verify that the map contains the Pubkey, one has to go through all the combinations of the variants of those two enums (worst case scenario). For now the number of combinations is 2 x 3 = 6, but in case if new variants are added this number will increase in arithmetic fashion, e.g. if the number of supported methods increases from 2 to 10, then the number of combinations will grow to 30 (again that's the worst case).
So here's the summary of this hacky solution: lazily generate a sequence of combinations of Subscription and Commitment variants, and check the presence of the entry for each combination. If the entry is found then active subscription exists. Upsides: easy to implement (almost no changes are necessary), downsides: not really performant.

UPDATE:
Going through the code once more, I've noticed that we do have all the information to compose the Subscription, Commitment tuple, so no combination generation is actually necessary!

[polachok]

Your analysis is correct. What do you propose to solve the problem, I mean, how to pass information about subscription status to rpc?

[bmuddha]

There's a required method on Cacheable trait named is_cacheable, which has the following signature

    fn is_cacheable(&self, state: &State) -> Result<(), UncacheableReason>;

As you can see it has access to self, which is type responsible to hold the information about current request, those types (currently there's 2) have an associated method called commitment (will need to move it to a trait) which can be used to retrieve Commitment for the Pubkey (with which the request is made). Also the the implementors of the is_cacheable can hardcode the variant of Subscription into method itself. Thus when making a call to subscription_active function from the is_cacheable (that seems to be the only call site for now), instead of passing just Pubkey, one must pass Subscription and Commitment as parameters, and the body will look something like this:

pub fn subscription_active(&self, sub: Subscription, commitment: Commitment) -> bool {
    let idx = self.get_idx_by_key(key);
    self.0[idx].0.sub_to_id.contains(&(Subscription, commitment))
}

The problem however is that self.0[idx].0 is not AccountUpdateManager but rather actix::Addr<AccountUpdateManager>, which is just a transmitting end of channel. So the problem boils down to how to access the map inside AccountUpdateManager without having a direct access to AccountUpdateManager.
With the solution proposed above the second member of tuple in vector which is part of

pub struct PubSubManager(Vec<(actix::Addr<AccountUpdateManager>, Arc<AtomicBool>)>);

type, actually becomes useless, and we can substitute it with RwLock<HashMap<K,V>> or Arc<DashMap<K,V>> for better performance, while changing the type of sub_to_id respectively. So we can check subscription status as before, but for a particular Pubkey.

All in all this approach will require some rewrite.

[bmuddha]

Now before serving request from cache we check if we have active ws connection, after which we send a query to subscription manager, which is responsible for handling updates for the cache entry of the request. The query's purpose is figure out if we have maintained active subscription for the given account|program, and if we do, application will serve the request from cache.