Comments (6)
It might also be interesting to consider an approach that solely uses async iterators. Along the lines of:
const { on } = require('events')
async function* live (db) {
// Current entries
for await (const [key, value] of db.iterator()) {
yield [key, value]
}
// Live entries (this is missing logic for batch and del)
for await (const [key, value] of on(db, 'put')) {
yield [key, value]
}
}
from level.
A couple of things I want to do that will benefit live streams/iterators:
- Remove
'put'
and'del'
events in favor of'batch'
event (just one event to listen to). This will also prepare us for hooks, especially a prewrite hook that would re-routedb.put()
to use a batch under the hood (and thus no longer emit a'put'
event). - In the
'batch'
event, use "normalized" operation objects, meaning they should include encodings and other (default) options. And encodings should be objects here, rather than their string names. - The operation objects should include both the original input data (as today) and encoded data. Aka public data (what the public API received, e.g.
db.batch()
) and private data (what the private API received, e.g.db._batch()
).
We can then support aligning the encoding of live data with the encoding of the iterator. E.g. if you do live(db, { valueEncoding: 'json' })
but then a db.put(key, value, { valueEncoding: 'buffer' })
, the resulting 'batch'
event includes the necessary information to transcode from buffer to json. Roughly like so:
const valueEncoding = db.valueEncoding('json')
const iterator = db.iterator({ valueEncoding })
db.on('batch', (operations) => {
for (const op of operations) {
if (op.valueEncoding.commonName === valueEncoding.commonName) {
// Public data matches desired encoding
const value = op.value
} else if (op.valueEncoding.format === valueEncoding.commonName) {
// Private data matches desired encoding
const value = op.encodedValue
} else {
// Decode private data (one of view, buffer, utf8) to match desired encoding
const transcoder = valueEncoding.createTranscoder(op.valueEncoding.format)
const value = transcoder.decode(op.encodedValue)
}
}
})
That last step there (createTranscoder()
) will just need a small utility method in level-transcoder
:
Click to expand
function createTranscoder (format) {
if (format === 'view') {
return this.createViewTranscoder()
} else if (format === 'buffer') {
return this.createBufferTranscoder()
} else if (format === 'utf8') {
return this.createUTF8Transcoder()
} else {
throw new Error('nope')
}
}
from level.
For a quick solution, you could fork level-live
and replace its db.createReadStream(opts).on('data', ..)
line with const { EntryStream } = require('level-read-stream')
and new EntryStream(db, opts).on('data', ..)
. Alternatively, replace that part with an iterator, or remove it altogether if you're only interested in the live part (which should still work the same because level-live
uses db events for that).
from level.
It might also be interesting to consider an approach that solely uses async iterators.
This looks really clean. I take it async function*
is the pattern for returning back an async iterator. Yeah, sorry, I have hardly used them before.
from level.
Yeah, that function is an async generator. PS. My example looks too good to be true; it needs some form of buffering to capture events emitted while it's busy iterating db.iterator()
.
from level.
Having that private data also means ltgt logic (like level-live
has) can work regardless of the user's choice of encoding, because it can compare buffers and strings rather than public data with arbitrary types.
from level.
Related Issues (20)
- Question: Fetch all values HOT 5
- Insert Lock, Read Lock, Dead Lock Concatency Control HOT 2
- Find specific key createReadStream HOT 6
- docker node alpine image exited with code 139 (arm32v7) HOT 4
- Can this be used to polyfill IndexedDB on Node? HOT 3
- [Question] how can "level-js" make data persistence? HOT 2
- db.isOperational is not a function HOT 2
- Trying to store different types of data but getting stored as string HOT 1
- Error while creating a web bundle with esbuild HOT 1
- Recommended way of retrieving a list of all keys? HOT 1
- Exception handling during storing a value HOT 4
- Types do not export `destroy` property HOT 1
- Random "Error: Database is not open" errors HOT 14
- Minor Compactions is triggered frequently
- Every time my node script restarts, it creates a new .LDB file in the database directory HOT 1
- Checking if a key exists HOT 1
- Get read access to locked database HOT 1
- how to get all sublevel db HOT 2
- No native build was found error on MacOS with M1 processor HOT 3
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from level.