level / multileveldown

multilevel implemented using leveldowns with reconnect support.

multileveldown

multilevel implemented using leveldowns with seamless retry support

Usage

Similar to multilevel you can use this to share a levelup database across multiple processes over a stream. In addition multileveldown supports seamless retry so you can reconnect to a server without your read streams / puts failing etc.

First create a server:

const multileveldown = require('multileveldown')
const level = require('level')
const net = require('net')

const db = level('db')

const server = net.createServer(function (sock) {
  sock.on('error', function () {
    sock.destroy()
  })

  sock.pipe(multileveldown.server(db)).pipe(sock)
})

server.listen(9000)

Then create some clients:

const multileveldown = require('multileveldown')
const net = require('net')

const db = multileveldown.client()

const sock = net.connect(9000)
sock.pipe(db.connect()).pipe(sock)

db.put('hello', 'world', function () {
  db.get('hello', console.log)
})

Reconnect

To setup reconnect in your client simply set retry: true and reconnect to your server when the connection fails:

const multileveldown = require('multileveldown')
const net = require('net')

const db = multileveldown.client({
  retry: true
})

const connect = function () {
  const sock = net.connect(9000)
  const remote = db.connect()

  sock.on('error', function () {
    sock.destroy()
  })

  sock.on('close', function () {
    remote.destroy()
    setTimeout(connect, 1000) // reconnect after 1s
  })

  sock.pipe(remote).pipe(sock)
}

connect()

multileveldown will now make sure to retry your pending operations when you reconnect. If you create a read stream and your connection fails half way through reading that stream multileveldown makes sure to only retry the part of the stream you are missing. Please note that this might not guarantee leveldb snapshotting if you rely on that.

API

multileveldown.server(db, [options])

Returns a new duplex server stream that you should connect with a client. Options include:

{
  readonly: true, // make the database be accessible as read only
  preput: function (key, val, cb) {}, // called before puts
  predel: function (key, cb) {}, // called before dels
  prebatch: function (batch, cb) {} // called before batches
}

clientDb = multileveldown.client([options])

Creates a new client levelup that you can connect to a server. Options are forwarded to the levelup constructor.

clientDb.connect()

Returns a new duplex client stream that you should connect with a server stream

clientDb.createRpcStream()

Just an alias to .connect for multilevel API compatibility.

Install

With npm do:

npm i multileveldown

Contributing

Level/multileveldown is an OPEN Open Source Project. This means that:

Individuals making significant and valuable contributions are given commit-access to the project to contribute as they see fit. This project is more like an open wiki than a standard guarded open source project.

See the Contribution Guide for more details.

Donate

Support us with a monthly donation on Open Collective and help us continue our work.

License

MIT

multileveldown's Issues

Allow client to name the db?

Not exactly sure how/if this would work, but... it would be cool if the client could choose a db name/location.

May not be a common case, but we've found it useful to have an app create new databases over time.

Immediately unref the `ref` provided to `createRpcStream()`

Copied from Level/party#28 (comment): we don't initially unref. You must make at least one db operation for the logic to work. So this hangs:

const name = Date.now().toString()

const db1 = party(name)
const db2 = party(name)

await db1.open()
await db2.open()

console.log('opened')

// Comment this out to not make the event loop hang
// await db1.put('a', 'b')
// console.log(await db2.get('a'))

Failed to run server when use level@8

Unhandled exception during server initialization for leveldb@8.

Code:

const multileveldown = require('multileveldown')
const { Level } = require('level')
const net = require('net')

function server() {
const db = new Level('db')

const server = net.createServer(function (sock) {
  sock.on('error', function () {
    sock.destroy()
  })

  sock.pipe(multileveldown.server(db)).pipe(sock)
})

console.log('Server');
server.listen(9000);
console.log('Server started');
}

function client() {
const db = multileveldown.client()

const sock = net.connect(9000)
console.log('Client');
sock.pipe(db.connect()).pipe(sock)
console.log('Client started');

db.put('hello', 'world', function () {
  db.get('hello', console.log)
})
}
server();
client();

Expected

# node ./server.js 
Server
Server started
Client
Client started
null world

That is what you get

# node ./server.js 
Server
Server started
Client
Client started
/home/mvtm/projects/bugs-level/node_modules/multileveldown/server.js:35
  if (db.isOpen()) ready()
         ^

TypeError: db.isOpen is not a function
    at Object.module.exports [as server] (/home/mvtm/projects/bugs-level/node_modules/multileveldown/server.js:35:10)
    at Server.<anonymous> (/home/mvtm/projects/bugs-level/server.js:13:28)
    at Server.emit (node:events:526:28)
    at TCP.onconnection (node:net:1573:8)

Issues with electron+noderify

After my code is packaged up with noderify and placed into electron, __dirname becomes /, then /node_modules/multileveldown/schema.proto is the path. Any ideas how this might be worked around? /cc @dominictarr

Missing backpressure

Not sure if this is a problem in practice. But right now there is no back-pressure between client and server.

Don't swallow decode errors

https://github.com/Level/multileveldown/blob/master/leveldown.js#L60-L63

Will swallow errors and not notify the user.

Read-only connection

An ideas on how to create a read-only connection (only allow get + readStream)?

live stream support