Comments (8)
The indexedDB is shared between the extension tab and the background service. So it should be possible for both to read/write on the same database.
The problem is that IndexedDB reads are asynchronous, whereas sqlite3, being a C library, expects a synchronous file system. With OPFS that's no problem, but with IndexedDB we need to cheat. Essentially, the implementation works by loading the entire database into memory before opening it, and then asynchronously updating IndexedDB when the in-memory copy changes.
The downside of this is that two copies of the database aren't synchronized - they can't be without an asynchronous file system, which sqlite3 doesn't support. So I don't actually think this is possible.
Is the background service a singleton? If so, you could essentially use that service as a shared worker, with the foreground app connecting to it to access the database.
from drift.
The problem is that IndexedDB reads are asynchronous, whereas sqlite3, being a C library, expects a synchronous file system. With OPFS that's no problem, but with IndexedDB we need to cheat. Essentially, the implementation works by loading the entire database into memory before opening it, and then asynchronously updating IndexedDB when the in-memory copy changes.
The downside of this is that two copies of the database aren't synchronized - they can't be without an asynchronous file system, which sqlite3 doesn't support. So I don't actually think this is possible.
I see
Is the background service a singleton? If so, you could essentially use that service as a shared worker, with the foreground app connecting to it to access the database.
Yes, it is a singleton. I tried to do this but adding the following code in the background service
driftWorkerMain(() {
return WebDatabase.withStorage(
DriftWebStorage.indexedDb(
'my_app_db',
migrateFromLocalStorage: false,
inWebWorker: true,
),
);
});
throws StateError('This worker is neither a shared nor a dedicated worker')
from drift.
Update: I managed to get it working with the following web implementation:
import 'dart:html';
import 'package:drift/drift.dart';
import 'package:drift/web.dart';
import 'package:drift/web/worker.dart';
QueryExecutor connect({required bool isWebBackgroundService}) {
if (isWebBackgroundService) {
WorkerGlobalScope.instance.importScripts('sql-wasm.js');
final QueryExecutor queryExecutor = WebDatabase.withStorage(
DriftWebStorage.indexedDb(
'my_app_db',
migrateFromLocalStorage: false,
inWebWorker: true,
),
);
driftWorkerMain(() => queryExecutor);
return queryExecutor;
} else {
return DatabaseConnection.delayed(
connectToDriftWorker(
"background_service.web.js",
mode: DriftWorkerMode.shared,
),
);
}
}
AND with the following change for drift to force accept my background service as a shared worker:
void driftWorkerMain(QueryExecutor Function() openConnection) {
final self = WorkerGlobalScope.instance;
final _RunningDriftWorker worker = _RunningDriftWorker(true, openConnection);
// if (self is SharedWorkerGlobalScope) {
// worker = _RunningDriftWorker(true, openConnection);
// } else if (self is DedicatedWorkerGlobalScope) {
// worker = _RunningDriftWorker(false, openConnection);
// } else {
// throw StateError('This worker is neither a shared nor a dedicated worker');
// }
worker.start();
}
from drift.
Another Update:
While the above solution runs with no visible errors, the 2 databases seem to be out of sync.
When I update the database via the extension window/page, the QueryExecutor
of the background service does not pick up the change.
When I fully restart the background service (and it loads again the DB from indexedDB), it picks up the change.
from drift.
Is the background worker opening an independent database connection or are you connecting it to the drift server already? If there are two separate WebDatabase
instances, they won't share the underlying connection.
There's no API for this at the moment, but since you're making modifications you can use worker._startedServer ??= worker._establishModeAndLaunchServer(DriftWorkerMode.shared)
to obtain a DriftServer
. Then you'll want to do something like this to open a connection to the server in the same context:
DriftServer server = ...;
final channel = StreamChannelController();
server.serve(channel.foreign, serialize: false);
final connection =
await connectToRemoteAndInitialize(channel.local, serialize: false);
connection
implements QueryExecutor
and can be passed to the constructor of your database class. This will make the background service and the foreground contexts share a database connection (including things like stream queries synchronizing between them). You can set serialize
to false because you're opening an in-memory channel to the database server, that makes it a lot cheaper than going over MessagePort
s.
from drift.
I was trying to not rely on the drift server at all for the DB connection of the background service. So, what I have is:
- Background service creates a query executor and wraps it with a database connection. Then it starts a drift server based on this
DatabaseConnection
. It then returns the same database connection to be used by the DriftDatabase in the service. - The foreground window uses
connectToDriftWorker
to connect.
I think this approach should normally work (for the most part) to share the same database between the two.
Update since my last comment:
I think the issue is actually a bigger one, making this more like a completely new feature as opposed to something to be slightly retrofitted.
The APIs used to communicate between the chrome background service and the extension window is actually not exactly the same as the one used by workers.
To create a MessagePort, I need:
- The background service to call and listen at
chrome.runtime.onConnect.addListener((port) => _newConnection(port))
- The foreground service to get port by using
port = chrome.runtime.connect({name: "drift db"})
This needs quite a bit more dart-js interop done ofc. I already did a rough proof of concept for this. Now, The foreground and background are sharing the same database (after updating the DB in foreground, the background db also picks up the change)
The issues:
a) I see a couple of errors in console as it starts, something about message types received by background not being expected type
b) Using watch
on the background service doesn't automatically get changes pushed to stream when foreground app changes data. I expect this is more about a misuse of database connection in service as opposed to an issue caused by drift implementation.
I think this can work, but also seems different enough to warrant a new mode
altogether. Is this something you'd consider including in the main drift package? Or too niche to be worthwhile adding/supporting over time?
from drift.
I added this draft pull request with my changes to have a better idea of what I did.
P.s. That is insanely rough proof of concept code to test if it would work. There's no "good coding practices" being followed.
from drift.
a) I see a couple of errors in console as it starts, something about message types received by background not being expected type
Did your debugging code to print the type reveal the problem? If you're using the new interop APIs you might have to call dartify
and jsify
before passing Dart objects to a chrome port.
b) Using
watch
on the background service doesn't automatically get changes pushed to stream when foreground app changes data
Huh, it should. It might be that the foreground thinks its the only connection user if the mode is not shared
and thus doesn't send any update notifications.
Is this something you'd consider including in the main drift package? Or too niche to be worthwhile adding/supporting over time?
It's very cool work, but I think it's too niche to be included in package:drift
. Given that you're always using a shared connection setup though, it hopefully shouldn't be too hard to copy existing parts from drift sources and release this as an independent package.
I think drift should expose the underlying APIs to share drift databases across any kind of communication channel (it does with package:drift/remote.dart
). Isolates and drift's support for web workers are thin wrappers around the remote API. External use cases can use that API too.
from drift.
Related Issues (20)
- Migration issues with Firefox HOT 1
- Documentation issue: Setup HOT 2
- File table and dart:io File clash HOT 4
- CRASH: Stack dump aborted because GetAndValidateThreadStackBounds failed. HOT 4
- NullAwareTypeConverter not usable from drift files when the column is non-nullable HOT 3
- Error while trying to run dart run moor_generator migrate HOT 4
- SqliteException(1): while preparing statement, near "RETURNING": syntax error, SQL logic error dart test HOT 2
- [Bug] 1 positional argument expected by 'AppDatabase.new', but 0 found. Try adding the missing argument. HOT 2
- Type clash after 2.16.0 upgrade (breaking change in drift_dev) HOT 2
- Dialect aware Postgres types not working for .drift files HOT 7
- A Optional Generated Wrapper for simple Statements HOT 27
- because framework depends on both package_info_plus ^5.0.1 and drift_dev ^2.16.0, version solving failed. HOT 1
- Postgres giving the error Unsupported type: Instance of 'UuidValue' when using where clause. HOT 5
- Generate companion data from json
- Code generation failure HOT 1
- How to listen data in stream, while write it at the same time
- "Null check operator used on a null value" during update.replace call HOT 5
- WSL Ubuntu22 - app that uses drift renders as fully white screen HOT 2
- Mentioning that `drift_db_viewer v2.0.0` is breaking due to changes in `Flutter v3.22+` Material implementation HOT 4
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from drift.