Comments (23)
Any plans to sync with Firestore? That would be awesome
from hive.
Are s3, onedrive, google drive, dropbox on the plan?
from hive.
+1000 to this! #49 (comment)
from hive.
Looks good! Thanks.
from hive.
Is this still planned?
from hive.
Nope, the server not necessary to run Hive. The server will not process the data, just store it. Think about syncing to google drive.
from hive.
I happened to stumble on this CRDT implemenation.
docs: https://cluster.ipfs.io/documentation/guides/consensus/
At the bottom its nice to see that the make the distinction between CRDT and RAFT properly.
It means that you can have data on the same types on many devices, and merge them independently.
You dont need to make OT's ( Operational Transforms ) which is very painful and limiting.
This is the Core lib.
https://github.com/ipfs/go-ds-crdt
That lib is used for IPFS Cluster to allow it to synchronise data.
https://github.com/ipfs/ipfs-cluster/blob/master/consensus/crdt/consensus.go
I really hope this is picked up with hive.
I think its an excellent basis for Hive Sync.
from hive.
are there any updates on this? For a uni project, I was looking into a synchronization layer design using CRDTs. Here is my repo in dart https://github.com/Manuelbaun/sync_layer_crdt_playground.
I actually use some form of delta-crdts, sending only the mutation instead of the operation or the full state. It works nicely, but it adds a lot of overhead. For time tracking, I use Hybrid logical clocks. In my use cases ( just prototyping) my design worked fine but has a lot of work todo. For instance, I am not deleting anything and garbage collection will be needed at some point.
any of cause, there are a lot of design issues 😆 and I didn't made it into a library just yet
from hive.
I like the idea. If it has functions that need to be set up on initialization like FCM does, it would be easy for a custom solution to be added. But it still will require a lot of input from the user because of handling updating Hive when the app reopens but the remote db has changed, etc
from hive.
But it still will require a lot of input from the user because of handling updating Hive when the app reopens but the remote db has changed, etc
Yes that's true. Much easier would be an implementation which just creates a backup of Hive.
The goal is to support full sync (including support for remote changes)
from hive.
from hive.
Yes, I'll start experimenting once queries and asset DBs are ready...
If anyone has time and wants to contribute, I'll be there to help.
from hive.
I hope to make it very easy to write a sync solution for every service. This will take time tho.
from hive.
You could provide an interface so every user can implement what he wants.
from hive.
It would be great to know which information users actually need. Do you have something in mind?
from hive.
I'm thinking as I write, so forgive some reasoning holes.
My ideas is having a simple interface with 2 or 4 functions like these:
//to read and write only one element
bool writeToServer(String key, dynamic value, String boxName, Type runtimeType);
T readFromServer(String key, dynamic value, String boxName, Type runtimeType);
// to read and write all values in one time
bool writeToServer(Map<String, dynamic> maps, String boxName, Type runtimeType);
Map<String, dynamic> readFromServer(String boxName);
boxName parameter (and the others) can be useful to the user to know what service call.
I suppose that the first two functions can be used with lazy box.
Now user have to implement this interface and iinjects the implementation within hive, so hive can call this object to sync remotely.
This is just a raw idea.
PS. Sorry for my pseudocode
from hive.
@chemickypes The question of this way is that we have to encrypt and decrypt the data both on client and server.
The best way is reuse the binary format of Hive as I post in this issue.
And also, if we sync to s3, onedrive, google drive, dropbox, we have to do the encryption the decryption on client.
from hive.
@leedstyh
Since the data is stored unencrypted in memory, it will not be necessary to decrypt it before syncing.
I'm not sure about using the binary format. It would be necessary to run Hive on the server too since the binary format can only be used by Hive.
from hive.
@leedstyh I think that we can split this problem into two smaller problems:
- Use the remote server like an extension of locale hive
- Use Hive like a delegate and its goal is get the data from the server when it needs, and the user will not distinguish where the data will come from because it has only one access point.
I don't know what @leisim wants to do with Hive.
from hive.
If the goal is offline editing then we are into the CRDT and vector clocks territory maybe.
Typically the domain model needs to think in terms of Mutations or Ops. This works when no allowing offline editing is allowed because the Server Time is the Global TIME.
When you want to support offline editing you need a way to merge changes.
CRDT, OPS, and Vector clocks is this area. The changes are happening in different time domains now.
Here is something to get the ball rolling maybe..
It is a flutter example that has basic support for offline editing.
https://github.com/memspace/zefyr
It uses Operations and logs them.
But is does not have vector clock support.
The data model is using the quill approach.
https://github.com/memspace/zefyr/blob/master/packages/notus/lib/src/heuristics.dart#L6
https://github.com/pulyaevskiy/quill-delta-dart
- this is where the real OT ( Operational Transform) guts is.
Now rather than use vector clocks, sometimes you can use Context within the data.
I think this is what zefyr uses, but am not sure.
from hive.
Thanks for your valuable input. I'll definitely take a look at these projects and try to implement something similar with hive.
from hive.
Any further news on how the offline to online sync is coming along?
Has anyone been able to work something out?
from hive.
Any further news on how the offline to online sync is coming along? Has anyone been able to work something out?
There's no plans to implement this feature anytime soon. Why?
-
It's best to keep things simple. It's nice to have something that does lots of things for you. But as system complexity increases it's hard to maintain them and also the software itself becomes "bloated" over time. Also having a Swiss army knife in a big projects usually causes issues because the bloated software usually has it's own workflow that doesn't play well with the existing software architecture. In general I'm currently preferring to keep current implementation stable rather than adding new features.
-
Online sync implementation depends on how the server side is implemented and usually differs from project to project. Maybe one project need a static bearer key authentication while other one might require authentication token based on logged in user, or other one might consume XML while some can only consume JSON. Well we might provide library for back-end schema / rules, but it might not play well with the existing back-end in place.
-
You can implement your own sync system based on whatever currently is available. You can either serialize data into json and send it to your server or send contents of database file. (I would prefer 1st option). It really ends up to your own design decision how you want to implement.
I'm open to feedback and suggestions on that.
from hive.
Related Issues (20)
- Is the hive dead?
- Is there a way to use the map type adapter on a nested class using toJson?
- [Web] Issue with "Invalid argument(s): Invalid or corrupted pad block" in Flutter HOT 4
- Unhandled Exception: Out of Memory
- Flutter hive unit test HOT 1
- TypeId is not consistent and how to safely remove adapter
- iOS only: hive with encryption clearing database, app includes db usage in background HOT 2
- Wrong naming of hive constant global variable
- New data written to the disk is not immediately available. HOT 1
- Descending order of data in hive
- Non-primitive Objects example error HOT 1
- Data disappears from encrypted box once it is opened without cypher HOT 1
- Mention clearly in the Readme that Hive lacks support for multiple-isolates/concurrency
- Getting empty Map although it wasn't after the put action HOT 1
- Status of Hive HOT 8
- Error to use a list : HiveError : To use this list, you have to open the box "empresas" first.
- hello i got error while initializing hive ====> UnimplementedError when douing
- Is Hive being abandoned? No more updates yet? HOT 1
- Isar vs Hive (what should we be using now?) HOT 3
- registerAdapter does not work as shown in the example HOT 3
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from hive.