Bi-Modal Data Access about brick

[tshedor]

@SubutaDan to answer this question directly, correct, a get won't remove stale data. As APIs can be paginated or filtered, just because the response does not include a model that exists locally, there isn't a guarantee that the model instance was deleted on the server. So get only gets model instances.

For the less direct answer, this problem of data sync issue is a regular question, and one that I've been turning over for a decent solution. Unfortunately, I don't have a straight answer for you, so I'm going to lay out the options and maybe you can help me by weighing in on the pros and cons or the one that makes the most sense, or perhaps they should be mixed.

Leave the decision to the end implementation (current)

Brick would not guarantee remote updates from the server at all, leaving the implementation ambiguous. APIs are rarely consistent and it's impossible to predict how data will be delivered from one implementation to the next. Brick can only provide tooling that allows flexibility to accommodate the varying permutations. Further, via Providers, Brick is designed to be extensible without guarantee of a REST API. As elaborated in solution four, the way a server transmits data updates can vary wildly, and Brick supporting all permutations. A subclass method of a repository that requires this end-user implementation would make the need for data sync extremely apparent, even though it wouldn't necessarily standardize the solution.

Cons: the ease of using Brick is lost in the last mile

Provide an all-encompassing solution

Brick is built to be an all-in-one with a server-side package, like Aqueduct, reducing the setup and increasing code share between client and server.

Cons: new backends/porting backends are not easy; code sharing leads to major security concerns (maybe I want an encrypted password on the server for the User model, but I don't want the client to be aware of that field)

Introduce timestamps on every model and reconcile on hydrate

When a new model is stored in SQLite, it's timestamped, giving the end-user implementation the opportunity to compare and reconcile data when it's hydrated. Or, instead of continuing to pass the buck to the user, reconciliation occurs in the superclass repository during hydration with a user-defined refetch time (e.g. get accepts a { Duration refetchIfOlderThan }).

Cons: Brick should trust the user to manage their own data - inserting a column like "updatedAt" or "createdAt" for every model requires workarounds to remove the system if the user uses these fields in different ways (perhaps reconciliation is not necessary and the extra two columns for every row only increases the size of SQLite); end-users may need to adjust reconciliation logic (e.g. relying on the latest client data instead of the API data, regardless of the delta from the stored timestamp) and this could add unnecessary complexity

Document example implementations

The solutions I've heard of so far:

• Set up a push notification system that notifies the client every time a change occurs in the API
• Short poll the API for new responses every x seconds and hydrate the results in the background
• Subscribe to a websockets channel and stream responses and hydrate the results in the background

Cons: Brick is already extensively documented and discovering this portion during setup/configuration will be easy to lose in the mess; maintaining a separate doc site (example: GatsbyJS) feels a little pre-op and is a lot of overhead

[SubutaDan]

Hi @tshedor,

Thanks for the reply and also for sharing your thoughts about synchronization. I don't have anything resembling a complete answer, but I am happy to provide some suggestions based on what I think I need for my applications. First, though, I will mention a few assumptions that underlie my suggestions. If the assumptions are incorrect then there is a good chance that the suggestions aren't worth considering.

All of my suggestions are about tweaks to Brick as a client library. I am not presenting ideas that require specific behaviour by provider servers. Also, they are all about offlineFirstWithRest.

I hope there are some useful ideas here but I will be neither surprised nor offended if you conclude that there are none.

-Dan

Assumptions

1. When I use "requireRemote: true" for a query then I am forcing Brick to go to REST, retrieve a set of models and update sqlite's contents to include the current state of alll of the models in the result set. The result will be that sqlite's content will be complete and accurate with regard to the result set, but may include models that are not in the result set.

2. Brick's current behaviour is to ensure that any local change (add a model, update a model, delete a model) will be reflected eventually in the REST data source to the extent that the REST API implements those operations.

3. Brick can ensure that the in-memory cache never contradicts the contents of sqlite. Is is therefor safe and reasonable for me to pretend that the in-memory cache doesn't exist for the purposes of this discussion.

4. When Brick returns results of a get to the caller, those results are consistent with what is in sqlite.

Suggestions

1. Brick could have a mode in which deletes are only applied to sqlite and not are not refleted in REST. This mode might apply in any/all cases or perhaps to specific model classes or to specific queries or to specific query executions or to specific providers. I haven't thought through the details. The benefit of this mode is that it would reduce the possibility of Brick causing unintended data loss at the data source. It is a belt-and-braces provision which would make no difference in a perfectly designed app. and so is, perhaps, unnecessary.

2. Brick could support a does-this-model-exist-at-the-provider query. The query would return 'yes', 'no' or, in the case where the provider can't be reached, 'I don't know'.

3. Brick could provide a delete-from-sqlite-only type of query so the app could explicitly remove from sqlite models that don't exist at the provider. These might be limited to one model instance (unique identifier) at a time. Maybe having this means that the case for no. 1 is so weak that no. 1 should be abandoned. I don't know whether/how these deletes would work transitively through associations.

4. Brick could support a style of get that instructs Brick to remove from sqlite any models that are in sqlite but are not found in the query's result set. Perhaps it is redundant for Brick to provide both this capability and also the combination of nos. 2 and 3. Like no. 3, this feature weakens the case for no. 1. As with no. 3, I don't know whether/how these deletes would work transitively through associations.

[tshedor]

@SubutaDan

Your assumptions are all correct, and you've done a great job of explaining them.

To your suggestions

Brick could have a mode in which deletes are only applied to sqlite and not are not reflected in REST. This mode might apply in any/all cases or perhaps to specific model classes or to specific queries or to specific query executions or to specific providers. I haven't thought through the details. The benefit of this mode is that it would reduce the possibility of Brick causing unintended data loss at the data source. It is a belt-and-braces provision which would make no difference in a perfectly designed app. and so is, perhaps, unnecessary.

Brick could provide a delete-from-sqlite-only type of query so the app could explicitly remove from sqlite models that don't exist at the provider. These might be limited to one model instance (unique identifier) at a time. Maybe having this means that the case for no. 1 is so weak that no. 1 should be abandoned. I don't know whether/how these deletes would work transitively through associations.

In this scenario, it sounds like you're only querying one provider, in which case would accessing that provider instead of using a repository method be sufficient? e.g. MyRepository().sqliteProvider.delete<Pizza>(pizza)? Of course, when to sync this data becomes a new problem. Even though it's not recommended, I've used this in a pinch. Unfortunately, this does throw the memory provider out of sync, and so a method that consolidates this may be a better course of action:

MyRepository extends OfflineFirstWithRestRepository { 
  // untested, unrun code; 
  @override
  Future<bool> delete<_Model extends OfflineFirstModel>(_Model instance, {bool onlyLocalProviders = false}) async {
    if (onlyLocalProviders) {
      try {
        await sqliteProvider.delete<_Model>(instance);
        await memoryCacheProvider.delete<_Model>(instance);
        return true;
      } catch (e) {
        return false;
      }
    }
    return await super.delete<_Model>(instance);
  }
}

Brick could support a does-this-model-exist-at-the-provider query. The query would return 'yes', 'no' or, in the case where the provider can't be reached, 'I don't know'.

That is a very interesting idea. SqliteProvider has an exists method that doesn't exist in the RestProvider or MemoryCacheProvider. And maybe that should be an optional method to implement, with a Future<bool> return value; null being the 'I don't know' case. I quite like this being on the Provider super class

Brick could support a style of get that instructs Brick to remove from sqlite any models that are in sqlite but are not found in the query's result set. Perhaps it is redundant for Brick to provide both this capability and also the combination of nos. 2 and 3. Like no. 3, this feature weakens the case for no. 1. As with no. 3, I don't know whether/how these deletes would work transitively through associations.

This is another interesting idea, and one that correlates to the reconciliation problem required to solve #112 . This would be the inverse of OfflineFirstRepository#get:hyrdateUnexisting but for #delete.

I'd still lean, in situations like this, to include this as a mixin in a utils library (like ActiveSupport) instead of being in the core functionality because this is extremely dependent on the API result - the API can't be paginated or filtered when it does this reconciliation removing all models that don't exist in the result set. Something like a #deleteAllExcept instead of a parameter on #delete.

---

Coming away from this discussion, I'd like to move on a few things:

• Add #exists to the Provider super class as an optional method
• An optional set of utils mixins for the OfflineFirstWithRestRepository with a way to refetch data via short polling, force reconciliation on get (if the model does not exist in the REST response but does exist in SQLite and MemoryCache, the response is removed from SQLite), and a #deleteAll method that accepts a query (this would be the deleteAllExcept but the query would define what to delete so it's a little more legible).
• And I'm not as sure about this one, but an OfflineOnly repository that would help with the workflow that your suggestions are steering toward. This is a little bit of Brick magic, but essentially you'd have one repository that hydrates your SQLite database on start, and then a second repository that does get, upsert, delete without reinteracting with REST.

This conversation's starting to expand in range, and may be more than your original issue bargained for. Is this still useful? Is it helpful? Or have I gone off the rails here?

[SubutaDan]

Hi @tshedor ,

Thanks for the thoughtful reply.

Yes, I find the discussion both interesting and useful. Ultimately I would like arrive at an understanding of how my application can re-sync its SQLite data from time to time with the data as resented by the REST provider.

You are correct that I am only querying one provider. I would be happy enough going to the provider directly as opposed to working through the repository.. Were you thinking of an OfflineOnly repository (your third bullet) as an alternative to going direct to the provider?

I think if there was a way to force reconciliation on get (your second bullet point) then I might not need the other changes you are considering. Of course that doesn't mean that they wouldn't be useful.

-Dan

[tshedor]

@SubutaDan

Were you thinking of an OfflineOnly repository (your third bullet) as an alternative to going direct to the provider?

Yes. Primarily because if you ever choose to add or remove a provider, your refactor cost is substantially smaller. Even if you're only accessing the SQLite provider through the repository, accessing the provider through a layer of abstraction will pay dividends as your architecture grows.

I've got a bit of time today. I'll put together a few of these mixins into a PR to declutter these thoughts and illustrate the concepts better.

[tshedor]

Closing this per #121 and #117