Coder Social home page Coder Social logo

level / subleveldown Goto Github PK

View Code? Open in Web Editor NEW
119.0 8.0 12.0 128 KB

Split a levelup database into sublevels with their own keyspace, encoding and events.

License: MIT License

JavaScript 100.00%
level sublevel abstract-leveldown levelup nodejs namespaces browser

subleveldown's Introduction

subleveldown

Split a levelup database into sublevels with their own keyspace, encoding and events.

level badge npm Node version Test Coverage Standard Common Changelog Donate

Table of Contents

Click to expand

Usage

If you are upgrading: please see UPGRADING.md.

const sub = require('subleveldown')
const level = require('level')

const db = level('db')
const example = sub(db, 'example')
const nested = sub(example, 'nested')

The example and nested db's are just regular levelup instances:

example.put('hello', 'world', function () {
  nested.put('hi', 'welt', function () {
    // Prints { key: 'hi', value: 'welt' }
    nested.createReadStream().on('data', console.log)
  })
})

Or with promises and iterators:

await example.put('hello', 'world')
await nested.put('hi', 'welt')

for await (const [key, value] of nested.iterator()) {
  // Prints ['hi', 'welt']
  console.log([key, value])
}

Sublevels see their own keys as well as keys of any nested sublevels:

// Prints:
// { key: '!nested!hi', value: 'welt' }
// { key: 'hello', value: 'world' }
example.createReadStream().on('data', console.log)

They also support db.clear() which is very useful to empty a bucket of stuff:

example.clear(function (err) {})

// Or delete a range within `example`
example.clear({ gt: 'hello' }, function (err) {})

// With promises
await example.clear()

Background

subleveldown separates a levelup database into sections - or sublevels from here on out. Think SQL tables, but evented, ranged and realtime!

Each sublevel is a levelup of its own. This means it has the exact same interface as its parent database, but its own keyspace and events. In addition, sublevels are individually wrapped with encoding-down, giving us per-sublevel encodings. For example, it's possible to have one sublevel with Buffer keys and another with 'utf8' encoded keys. The same goes for values. Like so:

sub(db, 'one', { valueEncoding: 'json' })
sub(db, 'two', { keyEncoding: 'binary' })

There is one limitation, however: keys must encode to either strings or Buffers. This is not likely to affect you, unless you use custom encodings or the id encoding (which bypasses encodings and thus makes it your responsibility to ensure keys are either strings or Buffers). If in that case you do pass in a key that is not a string or Buffer, it will be irreversibly converted to a string.

Authored by @mafintosh and inspired by level-sublevel by @dominictarr, subleveldown has become an official part of Level. As level-sublevel is no longer under active development, we recommend switching to subleveldown to get the latest and greatest of the Level ecosystem. These two modules largely offer the same functionality, except for hooks and per-batch prefixes.

API

subdb = sub(db[, prefix][, options])

Returns a levelup instance that uses subleveldown to prefix the keys of the underlying store of db. The required db parameter must be a levelup instance. Any layers that this instance may have (like encoding-down or subleveldown itself) are peeled off to get to the innermost abstract-leveldown compliant store (like leveldown). This ensures there is no double encoding step.

The prefix must be a string. If omitted, the effective prefix is two separators, e.g. '!!'. If db is already a subleveldown-powered instance, the effective prefix is a combined prefix, e.g. '!one!!two!'.

The optional options parameter has the following subleveldown specific properties:

  • separator (string, default: '!') Character for separating sublevel prefixes from user keys and each other. Must sort before characters used in prefixes. An error will be thrown if that's not the case.
  • open (function) Optional open hook called when the underlying levelup instance has been opened. The hook receives a callback which must be called to finish opening.

Any other options are passed along to the underlying levelup and encoding-down constructors. See their documentation for further details.

Install

With npm do:

npm i subleveldown -S

Contributing

Level/subleveldown is an OPEN Open Source Project. This means that:

Individuals making significant and valuable contributions are given commit-access to the project to contribute as they see fit. This project is more like an open wiki than a standard guarded open source project.

See the Contribution Guide for more details.

Donate

Support us with a monthly donation on Open Collective and help us continue our work.

License

MIT

subleveldown's People

Contributors

andymatuschak avatar dependabot[bot] avatar greenkeeper[bot] avatar juanpicado avatar mafintosh avatar meirionhughes avatar nocory avatar ralphtheninja avatar vweevers avatar

Stargazers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

Watchers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

subleveldown's Issues

properly support non-string keys

right now we do an implicit toString() on all keys to add the prefix. would be cool to just Buffer.concat them if the key encoding !== string

Question: how to do batch() across multiple sublevels?

Hi,

My apologies if this has been answered before.

If I want to batch operations across multiple sublevels, do I have to construct the operations using sublevel.db.prefix property?

root_levelup.batch([
  {type: 'put', 'key': sub1.db.prefix + 'key', value: 'val'},
  {type: 'put', 'key': sub2.db.prefix + 'key', value: 'val'}
]);

Or is there already some utils function to do this? πŸ˜…

Remove approximateSize() and getProperty()

subleveldown/leveldown.js

Lines 122 to 128 in 249abb9

SubDown.prototype.approximateSize = function (start, end, cb) {
this.leveldown.approximateSize.apply(this.leveldown, arguments)
}
SubDown.prototype.getProperty = function () {
return this.leveldown.getProperty.apply(this.leveldown, arguments)
}

They are leveldown specific and no longer part of the latest abstract-leveldown.

Write more tests

  • main function, with different combinations of prefix and/or opts
  • SubDown constructor with different prefix, separator etc (needs docs as well)
  • different subdbs with different encodings
  • wrap a closed levelup and re-open the levelup

opts always set in ._iterator()

SubDown.prototype._iterator = function (opts) {                      
  if (!opts) opts = {}                                               
  var xopts = extend(wrap(fixRange(opts), this._wrap), opts)         
  return new SubIterator(this.leveldown.iterator(xopts), this.prefix)
}                                                                    

Require deferredOpen support

To simplify the open/close handling on sublevels (and fix issues like #60) I propose the following.

  1. Only accept a db that supports deferredOpen (currently that's only levelup, in the future also abstract-leveldown). In subleveldown we then never open or close the parent db. More precisely: we never initiate a state change. The parent db must open itself (or once closed by the user, be explicitly reopened by the user) because sublevels shouldn't touch (the state of) the rest of the db.
  2. We can then have subdown._open() wait for an open event of the parent db. It would be nice to not have an _open() method at all, i.e. sublevels not having to care about whether the db is open, because deferredOpen support means you can always do operations on it, but this currently can't work because we operate on an unwrapped db. When abstract-leveldown gets deferredOpen support, things will get better.

test/common.js vs testCommon.js in abstract-leveldown

  • remove makeExistingDbTest, not used
  • cleanup(cb) calling cb sync
  • setUp(), tearDown() and collectEntries() are identical so should be able to reuse them

require testCommon.js from abstract-leveldown and override location, lastLocation and cleanup which are the only methods that are different

does not respect valueEncoding: 'json' when using .createReadStream

When passing an valueEncoding of json to a sublevel instance, it does not appear to respect the encoding when using createReadStream. It does however work with the .get method. Here's my example code:

const level = require('level')
const sub = require('subleveldown')

const db = level('./stream.db')
const jsonDB = sub(db, 'json', {valueEncoding:'json'})

jsonDB.put('foo', {'bar': 'baz'}, (err) => {
  jsonDB.get('foo', (err, val) => {
    console.log('GET',err || val)
    jsonDB.createReadStream().on('data', data => console.log(data))
  })
})

with an output of:

GET { bar: 'baz' }

events.js:183
      throw er; // Unhandled 'error' event
      ^
EncodingError: Unexpected token o in JSON at position 1
    at /home/austin/projects/proj-hyperlog/hyperlog-rep/node_modules/level-iterator-stream/index.js:36:28
    at /home/austin/projects/proj-hyperlog/hyperlog-rep/node_modules/subleveldown/leveldown.js:23:8
    at /home/austin/projects/proj-hyperlog/hyperlog-rep/node_modules/encoding-down/node_modules/abstract-leveldown/abstract-iterator.js:24:16
    at /home/austin/projects/proj-hyperlog/hyperlog-rep/node_modules/encoding-down/index.js:114:5
    at /home/austin/projects/proj-hyperlog/hyperlog-rep/node_modules/leveldown/node_modules/abstract-leveldown/abstract-iterator.js:24:16
    at /home/austin/projects/proj-hyperlog/hyperlog-rep/node_modules/leveldown/iterator.js:43:7
    at _combinedTickCallback (internal/process/next_tick.js:131:7)
    at process._tickCallback (internal/process/next_tick.js:180:9)

This is what I was expecting to see:

GET { bar: 'baz' }
{ key: 'foo', value: { bar: 'baz' } }

Any help appreciated. Thanks

Applies prefix twice on nested sublevel

Because the prefix is applied on open:

if (subdb && subdb.prefix) {
self.prefix = subdb.prefix + self.prefix
self.leveldown = reachdown(subdb.db, matchdown, false)
} else {

So if you open, close and reopen, self.prefix = subdb.prefix + self.prefix happens twice.

There are more issues with the open logic (#60, #77), so let's refactor it a bit.

Drop support of memdb

In favor of the newer level-mem. I already planned to drop memdb; I'm writing this because @bcomnes asked for some context in bcomnes/level-hookdown#29 (comment).

Warning: this is a long story. You can skip it if you accept "memdb is old" as a good enough reason πŸ˜‰

When we moved subleveldown to the Level org and updated it to work with latest Level modules, we did not account for memdb (as level-mem was already the preferred module). It's pure luck that wrapping memdb with subleveldown still works today. I recently added an integration test (#75) to avoid breaking it (after seeing that memdb is still depended upon). However, we're now at a point where continued support is just madness.

A typical db passed to subleveldown consist of several "layers". E.g. leveldown wrapped in encoding-down wrapped in deferred-leveldown wrapped in levelup. Behind the scenes, subleveldown unwraps that db to get to leveldown, then wraps it in a layer that prefixes keys, then (re)wraps that with encoding-down and levelup. For more background, see Level/community#82.

The problem with memdb is that it depends on an old version of deferred-leveldown (< 2.0.0) which when closed, doesn't expose its underlying db. So subleveldown can only unwrap after opening.

This limitation means we can't add a manifest (Level/community#83), which allows to inspect what features a db supports via db.supports. For example, db.supports.permanence indicates whether the underlying storage is persistent. Would be true for leveldown, false for memdown. For subleveldown to determine that, it must inspect the manifest of the unwrapped db, and this must happen synchronously, possibly before the db is open. Then you would be able to do:

var sub = require('subleveldown')(db)

if (!sub.supports.permanence) {
  throw new Error('Persistent storage is required')
}

If we drop support of memdb, we can unwrap synchronously and expose a manifest.

Manifests also make the long-term merging of levelup with abstract-leveldown (Level/community#58) easier. They allow us to determine which features a db already has and skip adding those. Because at some point, db's will be a mix of level(up) and abstract-leveldown features. Eventually, the unwrapping in subleveldown will also become simpler (thanks to needing less layers). Full circle :)

Update README

  • badges
  • homogenize headers
  • ~~link to UPGRADING.md~~~

Release v3.0.0

This feels pretty ready now. We have plenty of more tests as well.

Simplify _close()

From

SubDown.prototype._close = function () {               
  this.leveldown.close.apply(this.leveldown, arguments)
}                                                      

to

SubDown.prototype._close = function (cb) {             
  this.leveldown.close(cb)                             
}                                                      

An in-range update of levelup is breaking the build 🚨

The dependency levelup was updated from 4.3.0 to 4.3.1.

🚨 View failing branch.

This version is covered by your current version range and after updating it in your project the build failed.

levelup is a direct dependency of this project, and it is very likely causing it to break. If other packages depend on yours, this update is probably also breaking those in turn.

Status Details
  • ❌ continuous-integration/travis-ci/push: The Travis CI build failed (Details).

Release Notes for v4.3.1

Fixed

Commits

The new version differs by 3 commits.

See the full diff

FAQ and help

There is a collection of frequently asked questions. If those don’t help, you can always ask the humans behind Greenkeeper.


Your Greenkeeper Bot 🌴

Issues with `level-party`

Seems like a subtle incompatibility with what I'm doing here. As a note level-party-hyper is just a direct fork that uses level-hyper rather than level-prebuilt so I can compile it on CENTOS (unfortunate need).

Remove unnecessary condition

var subdb = reachdown(self.db, 'subleveldown')
if (subdb && subdb.prefix) {
self.prefix = subdb.prefix + self.ownPrefix
self.leveldown = subdb.leveldown
} else {
self.leveldown = reachdown(self.db, matchdown, false)
}

The && subdb.prefix here should not be needed, because at this point we've already determined subdb to be a subleveldown instance and such an instance always has a prefix (even when the user passes in an empty prefix).

If it doesn't have a prefix, that's either a bug or incorrect usage (someone modifying the prefix). Either way it should be surfaced as an error, which is preferable to hitting the else branch on L94, which can result in an incorrect self.leveldown.

Omitting `prefix`?

Omitting prefix is equivalent to passing in the empty string for the prefix, which seems like it defeats the point of using subleveldown. Is there a use case for omitting prefix, or could we make it a required argument (for which the user can still pass in an empty string)?

Streams are ignoring fillCache option

fillCache: true has no effect on the cache or memory usage, when reading data via createReadStream in a subleveldown database.

This can apparently be fixed by adding xopts.fillCache = opts.fillCache to the extend function in leveldown.js.

function extend (xopts, opts) {

After adding that line, extra memory was utilized according to the cacheSize specified in the options for the main level instance.
I didn't submit a pull request, since I am not familiar with the rest of the workings of that file, but that line seemed to do the trick here.

Segfault with subleveldown

Disclaimer: this is almost definitely my fault. I'm completely unfamiliar with leveldown and I'm probably doing everything wrong.

With that out of the way, I have a reproducible segfault while using subleveldown that's highly correlated with open/close events.

git clone https://github.com/flumedb/flumeview-level
cd flumeview-level
git checkout -b sub
git pull origin sub
npm ci
node test/read.js

Here's the output:

$ node test/read.js
  flumeview-level re-init +0ms
  flumeview-level re-start +2ms
  flumeview-level opened +9ms
  flumeview-level create() +0ms
FATAL ERROR: v8::Object::Cast Could not convert to object
 1: 0x55a75a67d6d1 node::Abort() [node]
 2: 0x55a75a67d71f  [node]
 3: 0x55a75a82fe9b v8::Utils::ReportApiFailure(char const*, char const*) [node]
 4: 0x7f52204e2b1c v8::Object::Cast(v8::Value*) [/home/christianbundy/src/flumeview-level/node_modules/leveldown/build/Debug/leveldown.node]
 5: 0x7f52204e490c v8::Local<v8::Object> v8::Local<v8::Object>::Cast<v8::Value>(v8::Local<v8::Value>) [/home/christianbundy/src/flumeview-level/node_modules/leveldown/build/Debug/leveldown.node]
 6: 0x7f52204e4036 v8::Local<v8::Object> v8::Local<v8::Value>::As<v8::Object>() const [/home/christianbundy/src/flumeview-level/node_modules/leveldown/build/Debug/leveldown.node]
 7: 0x7f52204e92d2 leveldown::Database::Get(Nan::FunctionCallbackInfo<v8::Value> const&) [/home/christianbundy/src/flumeview-level/node_modules/leveldown/build/Debug/leveldown.node]
 8: 0x7f52204e0ee6  [/home/christianbundy/src/flumeview-level/node_modules/leveldown/build/Debug/leveldown.node]
 9: 0x55a75a8b2ab0  [node]
10: 0x55a75a8b40d7  [node]
11: 0x2eccedadbe1d 
[1]    21605 abort (core dumped)  node test/read.js

And of course:

$ coredumpctl info 21605
           PID: 21605 (node)
           UID: 1000 (christianbundy)
           GID: 1000 (christianbundy)
        Signal: 6 (ABRT)
     Timestamp: Fri 2019-02-01 14:36:20 PST (55s ago)
  Command Line: node test/read.js
    Executable: /usr/bin/node
 Control Group: /user.slice/user-1000.slice/[email protected]/gnome-terminal-server.service
          Unit: [email protected]
     User Unit: gnome-terminal-server.service
         Slice: user-1000.slice
     Owner UID: 1000 (christianbundy)
       Boot ID: a26e2f3a62384290a1884245840c594f
    Machine ID: 4e83385b599d4d2ea6f5173888d92afc
      Hostname: samus
       Storage: /var/lib/systemd/coredump/core.node.1000.a26e2f3a62384290a1884245840c594f.21605.1549060580000000.lz4
       Message: Process 21605 (node) of user 1000 dumped core.
                
                Stack trace of thread 21605:
                #0  0x00007f5225387d7f raise (libc.so.6)
                #1  0x00007f5225372672 abort (libc.so.6)
                #2  0x000055a75a67d6e4 _ZN4node5AbortEv (node)
                #3  0x000055a75a67d71f n/a (node)
                #4  0x000055a75a82fe9b _ZN2v85Utils16ReportApiFailureEPKcS2_ (node)
                #5  0x00007f52204e2b1c n/a (/home/christianbundy/src/flumeview-level/node_modules/leveldown/build/Debug/leveldown.node)
                #6  0x00007f52204e490c n/a (/home/christianbundy/src/flumeview-level/node_modules/leveldown/build/Debug/leveldown.node)
                #7  0x00007f52204e4036 n/a (/home/christianbundy/src/flumeview-level/node_modules/leveldown/build/Debug/leveldown.node)
                #8  0x00007f52204e92d2 n/a (/home/christianbundy/src/flumeview-level/node_modules/leveldown/build/Debug/leveldown.node)
                #9  0x00007f52204e0ee6 n/a (/home/christianbundy/src/flumeview-level/node_modules/leveldown/build/Debug/leveldown.node)
                #10 0x000055a75a8b2ab0 n/a (node)
                #11 0x000055a75a8b40d7 n/a (node)
                #12 0x00002eccedadbe1d n/a (n/a)
                #13 0x00002ecceda918d5 n/a (n/a)
                #14 0x00002ecceda918d5 n/a (n/a)
                #15 0x00002ecceda918d5 n/a (n/a)
                #16 0x00002ecceda918d5 n/a (n/a)
                #17 0x00002ecceda918d5 n/a (n/a)
                #18 0x00002ecceda918d5 n/a (n/a)
                #19 0x00002ecceda918d5 n/a (n/a)
                #20 0x00002ecceda918d5 n/a (n/a)
                #21 0x00002ecceda918d5 n/a (n/a)
                #22 0x00002ecceda918d5 n/a (n/a)
                #23 0x00002ecceda918d5 n/a (n/a)
                #24 0x00002ecceda8ee75 n/a (n/a)
                #25 0x00002ecceda892c1 n/a (n/a)
                #26 0x000055a75ab7eb06 n/a (node)
                #27 0x000055a75ab7f03d n/a (node)
                #28 0x000055a75ab7f0f2 _ZN2v88internal9Execution4CallEPNS0_7IsolateENS0_6HandleINS0_6ObjectEEES6_iPS6_ (node)
                #29 0x000055a75a844785 _ZN2v88Function4CallENS_5LocalINS_7ContextEEENS1_INS_5ValueEEEiPS5_ (node)
                #30 0x000055a75a67e92c _ZN4node20InternalMakeCallbackEPNS_11EnvironmentEN2v85LocalINS2_6ObjectEEENS3_INS2_8FunctionEEEiPNS3_INS2_5ValueEEENS_13async_contextE (node)
                #31 0x000055a75a67ea2d _ZN4node12MakeCallbackEPN2v87IsolateENS0_5LocalINS0_6ObjectEEENS3_INS0_8FunctionEEEiPNS3_INS0_5ValueEEENS_13async_contextE (node)
                #32 0x000055a75a6639e9 _ZN4node11Environment14CheckImmediateEP10uv_check_s (node)
                #33 0x00007f52260d7241 uv__run_check (libuv.so.1)
                #34 0x00007f52260d1ab9 uv_run (libuv.so.1)
                #35 0x000055a75a688986 _ZN4node5StartEPN2v87IsolateEPNS_11IsolateDataERKSt6vectorINSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEESaISB_EESF_ (node)
                #36 0x000055a75a686c50 _ZN4node5StartEiPPc (node)
                #37 0x00007f5225374223 __libc_start_main (libc.so.6)
                #38 0x000055a75a63f7fe _start (node)
                
                Stack trace of thread 21606:
                #0  0x00007f522544bc7e epoll_pwait (libc.so.6)
                #1  0x00007f52260e02aa uv__io_poll (libuv.so.1)
                #2  0x00007f52260d1ab0 uv_run (libuv.so.1)
                #3  0x000055a75a7091b4 _ZZN4node20BackgroundTaskRunner20DelayedTaskScheduler5StartEvENUlPvE_4_FUNES2_ (node)
                #4  0x00007f522551ba9d start_thread (libpthread.so.0)
                #5  0x00007f522544bb23 __clone (libc.so.6)
                
                Stack trace of thread 21612:
                #0  0x00007f5225521afc pthread_cond_wait@@GLIBC_2.3.2 (libpthread.so.0)
                #1  0x00007f52260dd80a uv_cond_wait (libuv.so.1)
                #2  0x00007f52260ccac3 n/a (libuv.so.1)
                #3  0x00007f522551ba9d start_thread (libpthread.so.0)
                #4  0x00007f522544bb23 __clone (libc.so.6)
                
                Stack trace of thread 21615:
                #0  0x00007f5225521afc pthread_cond_wait@@GLIBC_2.3.2 (libpthread.so.0)
                #1  0x00007f52260dd80a uv_cond_wait (libuv.so.1)
                #2  0x00007f52260ccac3 n/a (libuv.so.1)
                #3  0x00007f522551ba9d start_thread (libpthread.so.0)
                #4  0x00007f522544bb23 __clone (libc.so.6)
                
                Stack trace of thread 21610:
                #0  0x00007f5225521afc pthread_cond_wait@@GLIBC_2.3.2 (libpthread.so.0)
                #1  0x00007f52260dd80a uv_cond_wait (libuv.so.1)
                #2  0x000055a75a706e1c n/a (node)
                #3  0x00007f522551ba9d start_thread (libpthread.so.0)
                #4  0x00007f522544bb23 __clone (libc.so.6)
                
                Stack trace of thread 21607:
                #0  0x00007f5225521afc pthread_cond_wait@@GLIBC_2.3.2 (libpthread.so.0)
                #1  0x00007f52260dd80a uv_cond_wait (libuv.so.1)
                #2  0x000055a75a706e1c n/a (node)
                #3  0x00007f522551ba9d start_thread (libpthread.so.0)
                #4  0x00007f522544bb23 __clone (libc.so.6)
                
                Stack trace of thread 21608:
                #0  0x00007f5225521afc pthread_cond_wait@@GLIBC_2.3.2 (libpthread.so.0)
                #1  0x00007f52260dd80a uv_cond_wait (libuv.so.1)
                #2  0x000055a75a706e1c n/a (node)
                #3  0x00007f522551ba9d start_thread (libpthread.so.0)
                #4  0x00007f522544bb23 __clone (libc.so.6)
                
                Stack trace of thread 21609:
                #0  0x00007f5225521afc pthread_cond_wait@@GLIBC_2.3.2 (libpthread.so.0)
                #1  0x00007f52260dd80a uv_cond_wait (libuv.so.1)
                #2  0x000055a75a706e1c n/a (node)
                #3  0x00007f522551ba9d start_thread (libpthread.so.0)
                #4  0x00007f522544bb23 __clone (libc.so.6)
                
                Stack trace of thread 21611:
                #0  0x00007f5225524436 do_futex_wait.constprop.1 (libpthread.so.0)
                #1  0x00007f5225524538 __new_sem_wait_slow.constprop.0 (libpthread.so.0)
                #2  0x00007f52260dd894 uv_sem_wait (libuv.so.1)
                #3  0x000055a75a766e61 n/a (node)
                #4  0x00007f522551ba9d start_thread (libpthread.so.0)
                #5  0x00007f522544bb23 __clone (libc.so.6)
                
                Stack trace of thread 21613:
                #0  0x00007f5225521afc pthread_cond_wait@@GLIBC_2.3.2 (libpthread.so.0)
                #1  0x00007f52260dd80a uv_cond_wait (libuv.so.1)
                #2  0x00007f52260ccac3 n/a (libuv.so.1)
                #3  0x00007f522551ba9d start_thread (libpthread.so.0)
                #4  0x00007f522544bb23 __clone (libc.so.6)
                
                Stack trace of thread 21614:
                #0  0x00007f5225521afc pthread_cond_wait@@GLIBC_2.3.2 (libpthread.so.0)
                #1  0x00007f52260dd80a uv_cond_wait (libuv.so.1)
                #2  0x00007f52260ccac3 n/a (libuv.so.1)
                #3  0x00007f522551ba9d start_thread (libpthread.so.0)
                #4  0x00007f522544bb23 __clone (libc.so.6)

Please let me know if there's anything else I can do to help debug this. Thanks a lot! I've been really enjoying my experience with leveldown so far.

keyEncoding: utf8 seems to be ignored

var level = require('level-prebuilt')
var subdown = require('subleveldown')

var enc = {keyEncoding: 'utf8', valueEncoding: 'json'}
var db = level('testdata', enc, function() {
  var subdb = subdown(db, 'subdb', enc)
  subdb.put('foo', {hello: 'world'}, function(){
    db.createReadStream().on('data', function(r) {
      console.log('parent', r)
    })
    subdb.createReadStream().on('data', function(r) {
      console.log('sub', r)
    })
  })
})
node tester.js 
parent { key: '!subdb!foo', value: { hello: 'world' } }
sub { key: <Buffer 66 6f 6f>, value: { hello: 'world' } }

Adding _seek to SubIterator

So I've been trying to add support for _seek on SubIterator. In principle the change is simply the added lines:

SubIterator.prototype._seek = function (key) {
  this.iterator.seek(concat(this.prefix, key))
}

but this never gets called.

If I do:

    var db = levelup(memdown())
    var sub = subdb(db, 'sub')

    db.once('open', function () {
     var it = sub.iterator({ keyAsBuffer: false, valueAsBuffer: false })
   
     console.log(it)

I get...

Iterator {
...
  it:
   SubIterator {
     iterator:
      MemIterator {
       ...

It seems clear Iterator is not calling this.it.seek, but I have no idea where this Iterator prototype is. Any ideas?

Current functionality and the future (hoping on support for all use cases)

Just wanted to jot down how it actually works but with words. Mostly for my own processing but also to explain to others that might be interested.

What I like the most with this module, is that it allows you to create sub levels (duh) but also with their specific encodings and it does this quite cleverly by peeling off the levelup layer, adding one down layer (SubDb) on top of the current down (leveldown, memdown etc wrapped by levelup) and then finishes off by adding back a levelup layer.

This trickery happens in ._open() (with some added comments):

SubDown.prototype._open = function (opts, cb) {
  var self = this

  if (this.db.isOpen()) {
    if (this.db.db.type === 'subdown' && this.db.db.prefix) {
      // This happens when we do a nested sub level
      // this.db is a levelup and this.db.db is the SubDown and
      // this.db.db.leveldown is the original down (see else case below)
      this.prefix = this.db.db.prefix + this.prefix
      this.leveldown = this.db.db.leveldown
    } else {
      // this.db is a levelup and this.db.db is the *down it's wrapping
      this.leveldown = this.db.db
    }
    return done()
  }

  this.db.on('open', this.open.bind(this, opts, done))

  function done (err) {
    if (err || !self._beforeOpen) return cb(err)
    self._beforeOpen(cb)
  }
}

The reason this works is because older versions of levelup takes care of the encodings and just applies them to a *down (SubDown in this case).

From index.js:

module.exports = function (db, prefix, opts) {
  if (typeof prefix === 'object' && !opts) return module.exports(db, null, prefix)
  if (!opts) opts = {}

  opts.db = function () {
    return subdown(db, prefix, opts)
  }

  return levelup(opts)
}

The problem we face now is that levelup was rewritten and encodings moved out into encoding-down so if we want to support encodings in the same way we can no longer rely on levelup alone.

So what if we in index.js finish off with:

module.exports = function (db, prefix, opts) {
  // ..
  return levelup(encoding(subdown(db, prefix, opts), opts), opts)
}

That should take care of the current functionality of levelup. But it's not enough. What if we want to create a sub level out of a level? This will not work (as @emilbayes pointed out here #7 (comment)) since it would mean double decodings (and encodings I guess).

So what I propose is that we continue with this trickery a bit and tweak ._open() to peel off two layers if there's an encoding-down:

SubDown.prototype._open = function (opts, cb) {
  var self = this

  if (this.db.isOpen()) {
    if (this.db.db.type === 'subdown' && this.db.db.prefix) {
      this.prefix = this.db.db.prefix + this.prefix
      this.leveldown = this.db.db.leveldown
    } else if (encDown.isEncodingDown(this.db.db)) { // <-- HERE!
      this.leveldown = this.db.db.db
    } else {
      this.leveldown = this.db.db
    }
    return done()
  }

  this.db.on('open', this.open.bind(this, opts, done))

  function done (err) {
    if (err || !self._beforeOpen) return cb(err)
    self._beforeOpen(cb)
  }
}

It's a bit hacky, but I think it should work well. We need to implement a proper static encDown.isEncodingDown() function though which should support cross realms (using Symbol.for() etc).

Simplify SubIterator#_next()

SubIterator.prototype._next = function (cb) {          
  var self = this                                      
  this.iterator.next(cb && function (err, key, value) {
    if (err) return cb(err)                            
    if (key) key = key.slice(self.prefix.length)       
    cb.apply(null, arguments)                          
  })                                                   
}                                                      

cb is always set, so remove check

SubIterator.prototype._next = function (cb) {          
  var self = this                                      
  this.iterator.next(function (err, key, value) {
    if (err) return cb(err)                            
    if (key) key = key.slice(self.prefix.length)       
    cb.apply(null, arguments)                          
  })                                                   
}                                                      

bytewise key-encoding on sub-level causes not found on other level

I'm getting odd behavior when using bytewise encoding on a sublevel. The intention is have bytewise encode a number to lex-sortable key, and then have that key appended on the end of the sub-level. It works if I write one value, but the moment I write a second, I can't read keys from other nested subs. Any idea where I'm going wrong? Docs says a sub-level must encode to a buffer and bytewise should be doing just that.

var sub = require('subleveldown')
var memdown = require('memdown')
var levelup = require('levelup');
var encoding = require('encoding-down');
var bytewise = require('bytewise');
var msgpack = require('msgpack-lite');
var {streamToRx} = require('rxjs-stream');

var db = levelup(encoding(memdown()));

var test1= sub(db, "logs", {valueEncoding: "json" });
var test2 = sub(db, "data");
var nested1 = sub(test2, '1234', { keyEncoding: bytewise, valueEncoding: msgpack })

async function main(){
  await test1.put("1234", "FOO");

  console.log("Got: " + await test1.get("1234"));
  console.log("put one..")
  await nested1.put(10,  10);

  console.log("Got: " + await test1.get("1234"));

  await dumpKeys(db);

  await nested1.put(20,  20);
  console.log("put another..")

  await dumpKeys(db);

  console.log(await test1.get("1234"));

  await dumpKeys(db);
}

async function dumpKeys(db){
  console.log("DUMP:")
  await streamToRx(db.createKeyStream()).forEach(x=>console.log(" " + x.toString()));
}

main().catch(console.log);

Console output:

Got: FOO
put one..
Got: FOO
DUMP:
 !logs!1234
 !data!!1234!B@$
put another..
DUMP:
 !logs!1234
 !data!!1234!B@$
 !data!!1234!B@4
NotFoundError: Key not found in database [1234]
    at D:\Code\geo\node_modules\levelup\lib\levelup.js:160:15
    at D:\Code\geo\node_modules\encoding-down\index.js:50:21
    at Immediate.callNext (D:\Code\geo\node_modules\memdown\memdown.js:162:7)
    at runCallback (timers.js:694:18)
    at tryOnImmediate (timers.js:665:5)
    at processImmediate (timers.js:647:5)

deps:

    "bytewise": "^1.1.0",
    "encoding-down": "^6.0.2",
    "leveldown": "^5.1.0",
    "levelup": "^4.0.2",
    "msgpack-lite": "^0.1.26",
    "rxjs": "^6.5.1",
    "rxjs-stream": "^3.0.2",
    "subleveldown": "^4.0.0",

Possible issue with buffer keys

Hi everyone, I've been experimenting with Buffer keys and I found that when I run the next code, the createReadStream gives me different results every time.

It should return result 10 every time but instead it gives randomly 10 or 6 or 7 (it depends of the crypto random buffer key generated).

const crypto = require('crypto')
const memdown = require('memdown')
const sub = require('subleveldown')

const db = sub(memdown(), 'test', { keyEncoding: 'binary', valueEncoding: 'json' })

;(async () => {
  for (let i = 0; i < 10; i++) {
    await db.put(crypto.randomBytes(32), { test: 'test ' + i })
  }

  let i = 0
  for await (const item of db.createReadStream()) {
    console.log(item.key.toString('hex'), item)
    i++
  }
  console.log('result ' + i)
})()

It works using level directly:

level(encode(memdown(), { keyEncoding: 'binary', valueEncoding: 'json' }))

An in-range update of abstract-leveldown is breaking the build 🚨

The dependency abstract-leveldown was updated from 6.1.1 to 6.2.0.

🚨 View failing branch.

This version is covered by your current version range and after updating it in your project the build failed.

abstract-leveldown is a direct dependency of this project, and it is very likely causing it to break. If other packages depend on yours, this update is probably also breaking those in turn.

Status Details
  • ❌ continuous-integration/travis-ci/push: The Travis CI build failed (Details).

Release Notes for v6.2.0

Changed

  • Upgrade hallmark devDependency from ^1.0.0 to ^2.0.0 (#349) (@vweevers)
  • Upgrade standard devDependency from ^13.0.1 to ^14.0.0 (#348) (@vweevers)

Added

Commits

The new version differs by 8 commits.

  • 77301e5 6.2.0
  • 3ac3991 Prepare 6.2.0
  • 539abd0 Add manifest (Level/community#83) (#351)
  • d7ba459 Document mandatory methods (#350)
  • a86d320 Upgrade hallmark devDependency from ^1.0.0 to ^2.0.0 (#349)
  • 8ab7493 Remove empty Unreleased section from changelog
  • 1516511 Fix links and order of changelog entries
  • 2a4f1a8 Upgrade standard devDependency from ^13.0.1 to ^14.0.0 (#348)

See the full diff

FAQ and help

There is a collection of frequently asked questions. If those don’t help, you can always ask the humans behind Greenkeeper.


Your Greenkeeper Bot 🌴

Take advantage of manifests and the squash down

Dumping some thoughts. A couple of efforts are underway and relevant to subleveldown:

  1. Add manifests (Level/community#83)
  2. Add type (Level/community#82)
  3. The squash down (let's give it a name) (Level/community#58)

Let's assume that somewhere in the future, abstract-leveldown has reached feature parity with levelup. Then subleveldown won't need to unwrap levelup and deferred-leveldown, nor will it need to rewrap. Issues like #60 will be gone.

Whether it will need to unwrap/rewrap encoding-down is an open question. In any case we'll have more options to feature-detect: subleveldown might detect db.type === 'encoding-down' or db.supports.encodings.

In the latter case maybe subleveldown will be able to "punch through" the encodings of its input db by using { keyEncoding: 'binary' } internally, and still support encodings externally (assuming that'll be an abstract-leveldown feature). That would be quite nice.

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    πŸ–– Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. πŸ“ŠπŸ“ˆπŸŽ‰

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❀️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.