level / memdown Goto Github PK

Over the weekend I threw together a varient of memdown based on a redblack tree. Initial tests seem to show it is both faster and support snapshots. It might make sense to think about moving memdown over to a similar type of persistent backend.

Sometimes the tests fail with the following error:

# test iterator with end being a midway key (50.5)
events.js:160
      throw er; // Unhandled 'error' event
      ^

Error: connection refused: localtunnel.me:40685 (check your firewall settings)
    at Socket.<anonymous> (/home/travis/build/Level/memdown/node_modules/localtunnel/client.js:84:32)
    at emitOne (events.js:96:13)
    at Socket.emit (events.js:188:7)
    at emitErrorNT (net.js:1272:8)
    at _combinedTickCallback (internal/process/next_tick.js:74:11)
    at process._tickCallback (internal/process/next_tick.js:98:9)

This is super annoying.

They were written for "old memdown", before we had functional-red-black-tree.

Wrote this optimization for local-storage down, but it could just as easily apply to memdown: No9/localstorage-down#23.

Hello,
I'm using memdown for my testsuite and if often starts writing folders to the filesystem.
How can that be?

I want to use levelgraph, which requires levelup, which in-turn requires either leveldown/memdown/level.js... Unfortunately the platform is mobile without an IndexedDB (Fuse to be precise ) so I can't use level.js. So I was thinking there was a way to work with memdown and write the database to the filesystem, then load it anytime I need it

Is it possible to disable these noisy coveralls comments?

The status check is enough I think.

• add node version badge
• remove greenkeeper badge
• use SVG badge for Travis
• add level badge
• normalize markdown headers (## instead of --)
• common spelling: Licence => License
• database => store

• zuul info is no longer accurate
• fix LevelUP -> levelup etc

@vweevers Maybe we could add some entries for the next major and start from there?

Now homepage is https://github.com/Level/memdown
but in package.json still https://github.com/rvagg/node-memdown

When I cd memdown && git co v2 && npm link abstract-leveldown && npm t, TypeScript doesn't compile, with many errors like:

TSError: ⨯ Unable to compile TypeScript
..\abstract-leveldown\abstract-leveldown.js (34,19): Property '_open' does not exist on type '{ location: string; status: string; open: (options: any, callback: any) => void; close: (callback...'. Did you mean 'open'? (2551)
..\abstract-leveldown\abstract-leveldown.js (92,19): Property '_get' does not exist on type '{ location: string; status: string; open: (options: any, callback: any) => void; close: (callback...'. Did you mean 'get'? (2551)

Doesn't happen with a regular npm i abstract-leveldown or npm i level/abstract-leveldown (which is the same version as I have on disk and want to link).

@MeirionHughes any idea?

if keys are buffers, they are coerced to utf8 strings.
utf8 strings dont support arbitrary binary data (e.g. hashes).
invalid utf8 byte sequences are replaced with the diamond-question mark character, potentially causing unintended key collisions.
I also imagine the subsequent throw-catch ruins performance.

I am not sure if this is an issue with level-ttl or memdown, but I can't get this to work:

const level  = require('levelup');
const ttl = require('level-ttl');
const memdown = require('memdown');

var db = level({db: memdown});
db.put('foo', 'bar');
db.get('foo', (err, data) => console.log(data));
//prints bar

const aday = 24 * 60 * 60 * 1000;
var ttldb = ttl(db, { checkFrequency: aday });
ttldb.put('foo', 'bar', { ttl: aday })
ttldb.get('foo', (err, data) => console.log(data));
//prints undefined

It works if I use leveldown.

it would be nice if get could return the result directly.

see https://travis-ci.org/Level/memdown/jobs/62325417

🚨 You need to enable Continuous Integration on all branches of this repository. 🚨

To enable Greenkeeper, you need to make sure that a commit status is reported on all branches. This is required by Greenkeeper because we are using your CI build statuses to figure out when to notify you about breaking changes.

Since we did not receive a CI status on the greenkeeper/initial branch, we assume that you still need to configure it.

If you have already set up a CI for this repository, you might need to check your configuration. Make sure it will run on all new branches. If you don’t want it to run on every branch, you can whitelist branches starting with greenkeeper/.

We recommend using Travis CI, but Greenkeeper will work with every other CI service as well.

Once you have installed CI on this repository, you’ll need to re-trigger Greenkeeper’s initial Pull Request. To do this, please delete the greenkeeper/initial branch in this repository, and then remove and re-add this repository to the Greenkeeper integration’s white list on Github. You'll find this list on your repo or organiszation’s settings page, under Installed GitHub Apps.

Remove this line (as well as the unit test):

And describe the new behavior in a changelog. Pending Level/abstract-leveldown#142.

• Tilde for dependencies
• Caret for devDependencies
• Update ts-node

Looks like a change in abstract-leveldown broke us:

⨯ test put()/get()/del() with `null` value
  not ok 236 should be equal
    ---
      operator: equal
      expected: ''
      actual:   'null'
      at: Immediate.callNext [as _onImmediate] (/Users/nolan/workspace/memdown/memdown.js:177:5)
    ...
⨯ test put()/get()/del() with `undefined` value
  not ok 244 should be equal
    ---
      operator: equal
      expected: ''
      actual:   'undefined'
      at: Immediate.callNext [as _onImmediate] (/Users/nolan/workspace/memdown/memdown.js:177:5)
    ...

I'm having a heck of a time testing this because it doesn't reproduce in the browser, and entirely different tests are broken in the browser for abstract-leveldown.

Just ran into this after getting an updated memdown as a dependency of dynalite. After a quick glance very strangely it seems like ./immedate.js is missing from the npm package of the 1.2.3 release (though present in this repo):

$ curl -s https://registry.npmjs.org/memdown/-/memdown-1.2.3.tgz | tar zxv
package/package.json
package/README.md
package/LICENSE
package/memdown.js
$ cd package
$ npm install
$ node package/memdown.js
module.js:327
    throw err;
    ^

Error: Cannot find module './immediate'
    at Function.Module._resolveFilename (module.js:325:15)
    at Function.Module._load (module.js:276:25)
    at Module.require (module.js:353:17)
    at require (internal/module.js:12:17)
    at Object.<anonymous> (/home/mapbox/tmp/package/memdown.js:10:20)
    at Module._compile (module.js:409:26)
    at Object.Module._extensions..js (module.js:416:10)
    at Module.load (module.js:343:32)
    at Function.Module._load (module.js:300:12)
    at Function.Module.runMain (module.js:441:10)

Not sure how but it seems like this file was omitted unintentionally from the npm package in the latest release?

cc @nolanlawson

This was added in 001c86e, excluded from browser builds in 80822a7 and necessary because abstract-leveldown/testCommon.js requires rimraf.

I think we should solve this in abstract-leveldown so that memdown (and others) don't need these hacks. I'll open an issue. /cc @ralphtheninja

Because stream reads can happen asynchronously, modifications to _keys in between _next calls can result in items being emitted twice (or skipped entirely).

Eg: stream is iterating, _pos is 1 so _next returns the value from that index, but then a key is inserted at index 0 due to a _put, so now, even though _pos has been incremented to 2, _next will return the same value as last time because it's been shifted up the array.

The are two solutions that I can think for this off the top of my head:

1. Take a snapshot/clone of _keys in the MemIterator constructor
2. Increment/decrement _pos whenever an entry is added/removed (iterators could listen for these events)

I'd be leaning towards 1 as it's simpler, but it could take up a chunk of memory.

Following leveldown.

See this comment for background.

I feel like there are others who have much more vested interest in the success and stability of this lib than I do at the moment. For me it hasn't graduated beyond being an interesting exercise in extending the leveldown pattern and as a useful backend for tests. I'd like to npm owner add at least one person to it. I'd prefer not to open that right up to everyone on the collaborators list as I believe that even OPEN Open Source needs a tight release mechanism.

I'd like to hear thoughts from others, particularly those who are using this in some real way. I'm nominating @nolanlawson.

See #50

Continuing from #84 (comment). @ralphtheninja please elaborate :)

I haven't investigated it yet but here is the automated issue:
ethereumjs/ethereumjs-monorepo#111

    */node_modules/levelup/lib/levelup.js:103
      db.open(this.options, function (err) {
         ^
    TypeError: Cannot call method 'open' of undefined
        at LevelUP.open (*/node_modules/levelup/lib/levelup.js:103:6)
        at new LevelUP (*/node_modules/levelup/lib/levelup.js:73:8)
        at LevelUP (*/node_modules/levelup/lib/levelup.js:44:12)
        at worker (*:239:33)
        at Object.<anonymous> (*:690:2)
        at Module._compile (module.js:456:26)
        at Object.Module._extensions..js (module.js:474:10)
        at Module.load (module.js:356:32)
        at Function.Module._load (module.js:312:12)
        at Function.Module.runMain (module.js:497:10)

Not sure if it's a recession, but it fails to open.

    require('levelup')("", {
        db: function(location) {
            return new require('memdown')(location)
        }
    });

the global db store is a breaking change I had some tests that where using throwaway db names but where not destroying, suddenly breaking because each instance isn't fresh

Prerequisite: Level/abstract-leveldown#119 has to be released

What we have here is more like the upgrade guide of abstract-leveldown. WDYT @ralphtheninja?

See https://github.com/Level/abstract-leveldown/blob/master/UPGRADING.md#testbuffer-parameter-removed.

get always seems to return a string, even if you used a number or object as the value in a put,

I can't find any documentation that shows any example, or has any explanation other than with a string.

According to LevelDB, I should be able to store any type of key and value I want as they are stored as

LevelDB supports arbitrary byte arrays as both keys and values..

and

put() is the primary method for inserting data into the store. Both the key and value can be arbitrary data objects.

Is there something I am not understanding or..?

This feature doesn't fit Node.js patterns and even in browsers nowadays, global state is a no-no. Better put, leaking into global state. There are times when it is appropriate (e.g. React with a global store) but IMO this is an application-level choice, not to be made by a module.

Without a global store, the "location" parameter becomes irrelevant. Usage then is just memdown(). If you do need global state, you can attach the database to a global yourself, e.g. window.db = db.

WDYT @ralphtheninja @juliangruber? We can do this after releasing levelup@2.

Also while we're at it, it's about time we added a test to the abstract-leveldown tests. We can borrow the one from localstorage-down.

Iterator:

Get:

The iterator clones Buffers (if the key or value is already a Buffer) and does not stringify non-buffers. I think the get behavior is better, for performance and safety.

A possible improvement for both is to also support typed arrays (don't stringify those).

const encode = require('encoding-down')

const db = levelup(encode(memdown(), { valueEncoding: 'json' }))
const obj = { thing: 'original' }

db.put('key', obj, (err) => {
  obj.thing = 'modified'

  db.get('key', { asBuffer: false }, (err, value) => {
    console.log(value === obj) // false
    console.log(obj.thing) // 'original'
  })
})

Last line should be console.log(value.thing) // 'original'

Is the DB size limited by v8 memory limits or is it limited by RAM on host machine?