level / party

Open a leveldb handle multiple times.

level-party

Open a leveldb handle multiple times, transparently upgrading to use multileveldown when more than 1 process try to use the same leveldb data directory at once and re-electing a new master when the primary unix socket goes down.

Example

Normally with level, when you try to open a database handle from more than one process you will get a locking error:

events.js:72
        throw er; // Unhandled 'error' event
              ^
OpenError: IO error: lock /home/substack/projects/level-party/example/data/LOCK: Resource temporarily unavailable
    at /home/substack/projects/level-party/node_modules/level/node_modules/level-packager/node_modules/levelup/lib/levelup.js:114:34

With level-party, the database open will automatically drop down to using multilevel over a unix socket using metadata placed into the level data directory transparently.

This means that if you have 2 programs, 1 that gets:

const level = require('level-party')
const db = level(__dirname + '/data', { valueEncoding: 'json' })

setInterval(function () {
  db.get('a', function (err, value) {
    console.log('a=', value)
  })
}, 250)

And 1 that puts:

const level = require('level-party')
const db = level(__dirname + '/data', { valueEncoding: 'json' })

const n = Math.floor(Math.random() * 100000)

setInterval(function () {
  db.put('a', n + 1)
}, 1000)

and you start them up in any order, everything will just work! No more IO error: lock exceptions.

$ node put.js & sleep 0.2; node put.js & sleep 0.2; node put.js & sleep 0.2; node put.js & sleep 0.2
[1] 3498
[2] 3502
[3] 3509
[4] 3513
$ node get.js
a= 35340
a= 31575
a= 37639
a= 58874
a= 35341
a= 31576
$ node get.js
a= 35344
a= 31579
a= 37643
a= 58878
a= 35345
^C

Hooray!

Seamless failover

level-party does seamless failover. This means that if you create a read-stream and the leader goes down while you are reading that stream level-party will resume your stream on the new leader.

This disables leveldb snapshotting so if your app relies on this you should disable this by setting opts.retry = false:

const db = level('./data', { retry: false }) // will not retry streams / gets / puts if the leader goes down

Windows support

level-party works on Windows as well using named pipes.

API

db = level(...)

The arguments are exactly the same as level. You will sometimes get a real leveldb handle and sometimes get a multileveldown handle back in the response.

Install

With npm do:

npm install level-party

Contributing

Level/party is an OPEN Open Source Project. This means that:

Individuals making significant and valuable contributions are given commit-access to the project to contribute as they see fit. This project is more like an open wiki than a standard guarded open source project.

See the Contribution Guide for more details.

Donate

Support us with a monthly donation on Open Collective and help us continue our work.

License

MIT

party's Issues

Using accountdown with level-party causes hangup

Not sure if this is a problem with accountdown or level-party. Running the following code:

 var party = require('level-party');
 var util = require('util');
 var accountdown = require('accountdown');
 var users = accountdown(party('./users'), {
   login: { basic: require('accountdown-basic') }
 });

var opts = {
  login: { basic: {username: "username", password: "password" } },
  value: { }
};

users.create("username", opts, function(err) {
  console.log('test');
});

Prints nothing. The callback to user.create never executes. This works fine with a plain levelup instance. I am inclined to think this is a bug with level-party if it is intended to be a drop in replacement for levelup.

cc @substack @mafintosh

Can't use level party with aws efs

Hi, Thanks for the amazing library.

I'm using LevelDB as embedded storage to do stateful stream processing.
I mount an AWS EFS storage into my docker containers which are running using AWS ECS Fargate.

The issue is the EFS uses nfsv4 which doesn't support persisting/sharing the virtual file (Unix socket).

I ended up using a giant container that spins a dozen of processes but this is a so limiting solution. I wonder if there are any other approaches that I can try out.

Thanks in advance.

Wrapping level-party breaks it

Hello guys and gals!

I'm having a bit of an issue using level-party. See, I have a wrapper that promisifies stuff, called native-level-promise and it calls level-party instead of calling level.

I would have expected that I can call my native-level-party from multiple files/processes and it would work. Unfortunately, I can't - I get exactly the same error using level-party as I got using level. aka "IO Error: Lockfile".

Any way around this? Any ideas? https://github.com/eslachance/native-level-promise

Add changelog

Does the leader need to go through socket?

Couldn't the leader use the db directly?

lev and level-party

Howdy,

lev -r --port ~/.party.lev/level-party.sock

I've been using lev to connect to a partied level instance while level-party is connected to it by connecting to the socket, this worked in v1.0.1 but not is 2.x.

I've no idea where the issue is, lev just uses net.connect to connect to the socket, but do you know any work around that would make this work? Sorry its a bit vague, I'm also making an assumption you're at least a little familiar with lev.

I should probably clarify that net.connect does actually connect to the socket, but any operation to query the db just hangs.

I guess a different approach would be if anyone knows a CLI that would let me poke around with the db with a level-party instance is connected?

Issue when two processes are accessing same level db instance (same folder name during level party setup)

I have two node js processes running on two different ports (websoket servers), I try to use same level db instance from both. When I was not using level-party, the second process (node server) would not even start. But after using level-party, both processes start properly and work for some cases. But in some cases I see the following error and the corresponding process terminates.
This happens consistently, The one that fails is the server which started later. Possibly it could be due to the levelgraphdb im using, as normal queries work fine, but only on graph search this fails.
Also if one process is stopped, the same queries work fine.

Here is the error.

events.js:85
throw er; // Unhandled 'error' event
^
EncodingError: Unexpected token o
at /node_modules/level-party/node_modules/multileveldown/node_modules/levelup/lib/read-stream.js:60:28
at Iterator.next (/node_modules/level-party/node_modules/multileveldown/leveldown.js:265:12)
at Array. (/node_modules/level-party/node_modules/multileveldown/leveldown.js:96:26)
at eval [as _decode](eval at %28/node_modules/level-party/node_modules/multileveldown/node_modules/pbs/node_modules/generate-function/index.js:55:21%29, :20:22)
at Decoder._pushMessage (/node_modules/level-party/node_modules/multileveldown/node_modules/pbs/decoder.js:130:10)
at Decoder._parseMissing (/node_modules/level-party/node_modules/multileveldown/node_modules/pbs/decoder.js:102:19)
at Decoder._parse (/node_modules/level-party/node_modules/multileveldown/node_modules/pbs/decoder.js:143:23)
at Decoder._write (/node_modules/level-party/node_modules/multileveldown/node_modules/pbs/decoder.js:193:10)
at doWrite (/node_modules/level-party/node_modules/multileveldown/node_modules/pbs/node_modules/readable-stream/lib/_stream_writable.js:279:12)
at writeOrBuffer (/node_modules/level-party/node_modules/multileveldown/node_modules/pbs/node_modules/readable-stream/lib/_stream_writable.js:266:5)

No level.destroy() ?

Hi guys,

Sorry to be a bother. I have a little native-level-promise module that works great with level itself, but when I try to use level-party as a drop-in replacement, it complains about level.destroy() not being a function. You can see the code here:
https://github.com/eslachance/native-level-promise/blob/master/index.js

Error on line 19, when using level-party instead of just level.

I am unfortunately not talented enough to really understand your code, and would greatly appreciate the assist on this one.

Gratzie,
Evie

Add support for db.sublevel(prefix)?

Would be useful. Not sure how hard it is...

When client becomes the "leader" no operations execute against the database

I'm not sure where to start debugging this except by creating a long running process that does this:

• 1. connect to database
• 2. get item
• 3. write item to console
• 4. sleep
• 5. goto 2

When running the process along, it will read one item then quit after it sleeps (when it attempts to read an item again)

If I add a second setTimeout and just loops and does nothing to keep the process alone, it's the only one that does anything (after the first read).

When I start 2 processes with the same script, the second one to start will read the item in the loop as expected. If I kill the first process (the "leader"), the second process becomes the "leader" and stops reading from the database.

I'm not sure if this is because the "leader" is a client and server and there's an error that's not propagating or if there's some other cause to this.

Update

I did some more tests and it seems to only happen when using accountdown and using the .list() method (which returns a stream).

Here's the script that I was using. I have the timeouts set high in this case so I can switch to another console and start it again. After the "leader" dies, and another "leader" is elected, it stops reading, then dies.

var mkdirp = require('mkdirp');
var level = require('level-party');
var sublevel = require('level-sublevel');
var accountdown = require('accountdown');

mkdirp.sync('./tmp/data');
var db = level('./tmp/data')
db.on('leader', console.log.bind(console, "I'm the leader"));
// db = sublevel(db);
var accounts = accountdown(db, {
  login: { basic: require('accountdown-basic') }
});

var start = new Date();

function writeMsg () {
  var opts = {
    login: {basic: {username: 'doowb', password: 'password'}},
    value: {name: 'Brian Woodward'}
  };
  accounts.create('doowb', opts, function (err) {
    setTimeout(writeMsg, 10000);
  });
  // var diff = (new Date()) - start;
  // db.put('time', diff.toString(), function (err) {
  //   if (err) console.error('Error writing diff', err);
  //   setTimeout(writeMsg, 1000);
  // });
}

function readMsg () {
  accounts.list()
    .on('data', console.log)
    .on('end', function () {
      setTimeout(readMsg, 2000);
    });
  // db.get('time', function (err, diff) {
  //   if (err && err.notFound) console.error('Message not found', err);
  //   if (err && !err.notFound) console.error('Error', err);
  //   console.log('Diff:', diff);
  //   setTimeout(readMsg, 200);
  // });
}

writeMsg();
setTimeout(readMsg, 5000);

Broken on node v6.0?

Your latest version prints this error and a very long stack trace

(node) v8::ObjectTemplate::Set() with non-primitive values is deprecated

Think the makers of leveldown already fixed this but somehow, due to your package.json config, their update isn't fetched.

Install master-specific modules.

I often use the level-ttl module, which I must attach only once in the 'master' process. What is the right approach? I think this use case is not handled at the moment.

I can't just add a 'promoted' event, as I need to intercept the creation of the new database. So, if it's ok for you I'll add a wrapMaster option with a function to install the master-only modules. So we can replace this line
https://github.com/substack/level-party/blob/master/index.js#L54
with:

proxy.swap(opts.wrapMaster && opts.wrapMaster(db) || db);

Shall I submit a PR?

Node 10

Regular leveldown doesn't support node 10 for older versions, including the ancient version used in this module. Any plans to upgrade?

Can level-party be used with RocksDB?

I'm getting a bit lost with all those layers: is there a way to pass a rocksdb version of abstract-leveldown? I have the impression that it's not possible, but maybe I'm missing something?

level-party leader process holding memory

env

• OS: ubuntu 20.04
• node: v16.13.1
• level-party: v5.1.1
• leveldown options: none

problem

Our level-party leader process holds up to 3GB of RAM, and refuses to release it once the activity peak passes, before being eventually killed by Linux OOM Killer (See inventaire/inventaire#542)

It's actually probably not a level-party-specific issue, as trying to run with only level produces the same behavior, but I didn't know where to post: would that be a leveldown issue? Is that an expected amount of RAM for leveldb with the default options? Any clue on what we can try to address this?

Thanks for any help, I'm running short of ideas ^^'

Doesn't always connect

I have a simple command line script that just opens a connection and gets a value. Every other time I run the script nothing happens and the script ends. There aren't any errors or anything.

If I have another process accessing the database at the same time, the command line script just hangs.

I'm using level-party with levelup-defaults and level-sublevel, so I'm not sure if that combination is causing the problem. My script works fine when I use level directly (I just can't access it from multiple processes)

I'm on node v0.12.0.

Error: Connection to leader lost

Error: Connection to leader lost
at Multilevel._clearRequests (/var/www/videomail/staging/source/node_modules/level-party/node_modules/multileveldown/leveldown.js:122:38)
at /var/www/videomail/staging/source/node_modules/level-party/node_modules/multileveldown/leveldown.js:104:10
at f (/var/www/videomail/staging/source/node_modules/once/once.js:17:25)
at onclose (/var/www/videomail/staging/source/node_modules/level-party/node_modules/multileveldown/node_modules/end-of-stream/index.js:43:45)
at emitNone (events.js:72:20)
at emit (events.js:166:7)
at Encoder.destroy (/var/www/videomail/staging/source/node_modules/level-party/node_modules/multileveldown/node_modules/pbs/encoder.js:24:10)
at Duplexify._destroy (/var/www/videomail/staging/source/node_modules/level-party/node_modules/multileveldown/node_modules/duplexify/index.js:187:66)
at /var/www/videomail/staging/source/node_modules/level-party /node_modules/multileveldown/node_modules/duplexify/index.js:174:10
at doNTCallback0 (node.js:408:9)
at process._tickCallback (node.js:337:13)

Not sure what I can do here. Probably a bug in multileveldown?

Errors not propagating?

It seems when using level-party errors do not propagate.

Sorry for not providing more helpful information...

Level-party upgrade for Node 11+ ?

This module is too useful to be left unattended. It needs to be updated for Node 11+.

Anyone knows any future plan for this module?

I have not found any module as easy and useful as this for multiple processes/clusters LevelDB connections.

Cannot successfully access same database with two different system users

If, when logged in as user A, I initialize database, and then when I try to connect to same database using user B, no data is propagated, and no error (that would indicate that something went wrong) is exposed.

How to reproduce:

Try to run below code on database initiated by other user. No logs will be reported, as if operation hanged. Same code will run as expected when run by user which initialized first access.

'use strict';

var level = require('level-party');
var db = level(pathToLevelDbFolder);

var stream = db.createReadStream();
stream.on('data', function () {});
stream.once('error', function (err) {
    console.log("Error", err.stack);
});
stream.once('end', function () {
    console.log("End");
});

Expected errors different between level-up and level-party

Hi,

Where the level-up get() method returns a 'NotFoundError', level-party get() returns an OperationalError "Key not found in database ... "

Chained batch feature broken

batch() method with no parameter returns undefined instead of a batch object for chaining.

Why level-prebuilt?

If leveldown (>= 1.2.2) now supports prebuilds, we probably shouldn't be using the fork level-prebuilt anymore (it causes node-gyp.js troubles under iojs anyway)

Always use worker for server?

If worker is available. Would it make sense to always run the server inside a Worker and even have the "leader" client act like a "follower" in relation to the worker?