holepunchto / hyperbee Goto Github PK

View Code? Open in Web Editor NEW

246.0 246.0 29.0 353 KB

An append-only B-tree running on a Hypercore

Home Page: https://docs.holepunch.to

License: MIT License

JavaScript 100.00%

hyperbee's People

Contributors

Stargazers

Watchers

hyperbee's Issues

issue with hyperbee.put

hello , below code give me the error as no update available from peers

const { corestore, replicate } = new Client()
const store = corestore()

// Get the stats core.
const statsCore = store.get({ key: STATS_CORE_KEY, valueEncoding: 'json' })

const db = new Hyperbee(statsCore, { keyEncoding: 'utf-8', valueEncoding: 'utf-8' })

await db.ready()

await db.put('a', 'b')
await db.put('c', 'd')

History stream errors for encrypted hypercores

This one is puzzling, to say the least. at the end of this issue, you can find an isolated test to replicate it.

NOTE: the same errors occur when using this PR of Hypercore.

none of the following issues occur if you comment line 9 to disable encryption.

What I am trying to achieve

Listen to a Hyperbee's feed, and log the changed key whenever the feed emits append event:

clone.on('append', async () => {
   // Check that the clone is setup correctly
    console.log({
      name,
      valid:
        cloneDB.feed.peers.length > 0 &&
        (encryptionKey ? !!cloneDB.feed.encryption : true),
    });

    const stream = cloneDB.createHistoryStream({
      reverse: true,
      limit: 1,
    });

    stream?.on('data', (data) => console.log('Data at: ' + name, data));
    stream?.on('error', (err) =>
      console.log('Error at: ' + name + ':\n', err),
    );
});

What happens

There are two cases:
1- Replicating an encrypted Hyperbee and creating a HistroyStream whenever the core emits append.
Two errors are thrown then it works fine :

{ name: 'ONE', valid: true }
{ name: 'ONE', valid: true }
Error at: ONE:
 TypeError: Cannot read properties of undefined (reading 'keys')
    at BlockEntry.getTreeNode (/Users/ward/code/play/hyperbee-tests/node_modules/hyperbee/index.js:253:37)
    at HyperBee.getRoot (/Users/ward/code/play/hyperbee-tests/node_modules/hyperbee/index.js:342:50)
    at processTicksAndRejections (node:internal/process/task_queues:96:5)
    at async HistoryIterator.open (/Users/ward/code/play/hyperbee-tests/node_modules/hyperbee/iterators/history.js:16:5)
Error at: ONE:
 Error: Groups are not supported
    at exports.skip (/Users/ward/code/play/hyperbee-tests/node_modules/protocol-buffers-encodings/index.js:29:13)
    at Object.decode (/Users/ward/code/play/hyperbee-tests/node_modules/hyperbee/lib/messages.js:469:18)
    at Object.decode (/Users/ward/code/play/hyperbee-tests/node_modules/compact-encoding/index.js:416:21)
    at Object.decode (/Users/ward/code/play/hyperbee-tests/node_modules/compact-encoding/index.js:432:14)
    at Hypercore._decode (/Users/ward/code/play/hyperbee-tests/node_modules/hypercore/index.js:789:23)
    at Hypercore._decryptAndDecode (/Users/ward/code/play/hyperbee-tests/node_modules/hypercore/index.js:630:24)
    at processTicksAndRejections (node:internal/process/task_queues:96:5)
    at async HyperBee.getBlock (/Users/ward/code/play/hyperbee-tests/node_modules/hyperbee/index.js:354:21)
    at async Batch.getBlock (/Users/ward/code/play/hyperbee-tests/node_modules/hyperbee/index.js:525:9)
    at async HyperBee.getRoot (/Users/ward/code/play/hyperbee-tests/node_modules/hyperbee/index.js:342:13)
{ name: 'ONE', valid: true }
Data at: ONE { type: 'put', seq: 2, key: 'foo', value: 'update number: 1' }
{ name: 'ONE', valid: true }
Data at: ONE { type: 'put', seq: 3, key: 'foo', value: 'update number: 2' }

2- It gets worse if I try to do the same for two Hyperbees from the same Hypercore (literally the same object or two sessions of the same core), all logs are errors, here is an example:

{ name: 'ONE', valid: true }
{ name: 'TWO', valid: true }
{ name: 'ONE', valid: true }
{ name: 'TWO', valid: true }
Error at: TWO:
 RangeError: Could not decode varint
    at Object.read [as decode] (/Users/ward/code/play/hyperbee-tests/node_modules/varint/decode.js:17:13)
    at exports.skip (/Users/ward/code/play/hyperbee-tests/node_modules/protocol-buffers-encodings/index.js:24:24)
    at Object.decode (/Users/ward/code/play/hyperbee-tests/node_modules/hyperbee/lib/messages.js:232:18)
    at new Pointers (/Users/ward/code/play/hyperbee-tests/node_modules/hyperbee/index.js:36:29)
    at inflate (/Users/ward/code/play/hyperbee-tests/node_modules/hyperbee/index.js:67:10)
    at BlockEntry.getTreeNode (/Users/ward/code/play/hyperbee-tests/node_modules/hyperbee/index.js:249:20)
    at HyperBee.getRoot (/Users/ward/code/play/hyperbee-tests/node_modules/hyperbee/index.js:342:50)
    at processTicksAndRejections (node:internal/process/task_queues:96:5)
    at async HistoryIterator.open (/Users/ward/code/play/hyperbee-tests/node_modules/hyperbee/iterators/history.js:16:5)
Error at: ONE:
 Error: Groups are not supported
    at exports.skip (/Users/ward/code/play/hyperbee-tests/node_modules/protocol-buffers-encodings/index.js:29:13)
    at Object.decode (/Users/ward/code/play/hyperbee-tests/node_modules/hyperbee/lib/messages.js:469:18)
    at Object.decode (/Users/ward/code/play/hyperbee-tests/node_modules/compact-encoding/index.js:416:21)
    at Object.decode (/Users/ward/code/play/hyperbee-tests/node_modules/compact-encoding/index.js:432:14)
    at Hypercore._decode (/Users/ward/code/play/hyperbee-tests/node_modules/hypercore/index.js:789:23)
    at Hypercore._decryptAndDecode (/Users/ward/code/play/hyperbee-tests/node_modules/hypercore/index.js:630:24)
    at processTicksAndRejections (node:internal/process/task_queues:96:5)
    at async HyperBee.getBlock (/Users/ward/code/play/hyperbee-tests/node_modules/hyperbee/index.js:354:21)
    at async Batch.getBlock (/Users/ward/code/play/hyperbee-tests/node_modules/hyperbee/index.js:525:9)
    at async HyperBee.getRoot (/Users/ward/code/play/hyperbee-tests/node_modules/hyperbee/index.js:342:13)
Error at: ONE:
 RangeError: Could not decode varint
    at Object.read [as decode] (/Users/ward/code/play/hyperbee-tests/node_modules/varint/decode.js:17:13)
    at exports.skip (/Users/ward/code/play/hyperbee-tests/node_modules/protocol-buffers-encodings/index.js:24:24)
    at Object.decode (/Users/ward/code/play/hyperbee-tests/node_modules/hyperbee/lib/messages.js:232:18)
    at new Pointers (/Users/ward/code/play/hyperbee-tests/node_modules/hyperbee/index.js:36:29)
    at inflate (/Users/ward/code/play/hyperbee-tests/node_modules/hyperbee/index.js:67:10)
    at BlockEntry.getTreeNode (/Users/ward/code/play/hyperbee-tests/node_modules/hyperbee/index.js:249:20)
    at HyperBee.getRoot (/Users/ward/code/play/hyperbee-tests/node_modules/hyperbee/index.js:342:50)
    at processTicksAndRejections (node:internal/process/task_queues:96:5)
    at async HistoryIterator.open (/Users/ward/code/play/hyperbee-tests/node_modules/hyperbee/iterators/history.js:16:5)
Error at: TWO:
 Error: Groups are not supported
    at exports.skip (/Users/ward/code/play/hyperbee-tests/node_modules/protocol-buffers-encodings/index.js:29:13)
    at Object.decode (/Users/ward/code/play/hyperbee-tests/node_modules/hyperbee/lib/messages.js:469:18)
    at Object.decode (/Users/ward/code/play/hyperbee-tests/node_modules/compact-encoding/index.js:416:21)
    at Object.decode (/Users/ward/code/play/hyperbee-tests/node_modules/compact-encoding/index.js:432:14)
    at Hypercore._decode (/Users/ward/code/play/hyperbee-tests/node_modules/hypercore/index.js:789:23)
    at Hypercore._decryptAndDecode (/Users/ward/code/play/hyperbee-tests/node_modules/hypercore/index.js:630:24)
    at processTicksAndRejections (node:internal/process/task_queues:96:5)
    at async HyperBee.getBlock (/Users/ward/code/play/hyperbee-tests/node_modules/hyperbee/index.js:354:21)
    at async Batch.getBlock (/Users/ward/code/play/hyperbee-tests/node_modules/hyperbee/index.js:525:9)
    at async HyperBee.getRoot (/Users/ward/code/play/hyperbee-tests/node_modules/hyperbee/index.js:342:13)
{ name: 'ONE', valid: true }
{ name: 'TWO', valid: true }
Error at: TWO:
 TypeError: Cannot read properties of undefined (reading 'keys')
    at BlockEntry.getTreeNode (/Users/ward/code/play/hyperbee-tests/node_modules/hyperbee/index.js:253:37)
    at HyperBee.getRoot (/Users/ward/code/play/hyperbee-tests/node_modules/hyperbee/index.js:342:50)
    at processTicksAndRejections (node:internal/process/task_queues:96:5)
    at async HistoryIterator.open (/Users/ward/code/play/hyperbee-tests/node_modules/hyperbee/iterators/history.js:16:5)
Error at: ONE:
 Error: Groups are not supported
    at exports.skip (/Users/ward/code/play/hyperbee-tests/node_modules/protocol-buffers-encodings/index.js:29:13)
    at Object.decode (/Users/ward/code/play/hyperbee-tests/node_modules/hyperbee/lib/messages.js:469:18)
    at Object.decode (/Users/ward/code/play/hyperbee-tests/node_modules/compact-encoding/index.js:416:21)
    at Object.decode (/Users/ward/code/play/hyperbee-tests/node_modules/compact-encoding/index.js:432:14)
    at Hypercore._decode (/Users/ward/code/play/hyperbee-tests/node_modules/hypercore/index.js:789:23)
    at Hypercore._decryptAndDecode (/Users/ward/code/play/hyperbee-tests/node_modules/hypercore/index.js:630:24)
    at processTicksAndRejections (node:internal/process/task_queues:96:5)
    at async HyperBee.getBlock (/Users/ward/code/play/hyperbee-tests/node_modules/hyperbee/index.js:354:21)
    at async Batch.getBlock (/Users/ward/code/play/hyperbee-tests/node_modules/hyperbee/index.js:525:9)
    at async HyperBee.getRoot (/Users/ward/code/play/hyperbee-tests/node_modules/hyperbee/index.js:342:13)
{ name: 'ONE', valid: true }
{ name: 'TWO', valid: true }
Error at: TWO:
 TypeError: Cannot read properties of undefined (reading 'keys')
    at BlockEntry.getTreeNode (/Users/ward/code/play/hyperbee-tests/node_modules/hyperbee/index.js:253:37)
    at HyperBee.getRoot (/Users/ward/code/play/hyperbee-tests/node_modules/hyperbee/index.js:342:50)
    at processTicksAndRejections (node:internal/process/task_queues:96:5)
    at async HistoryIterator.open (/Users/ward/code/play/hyperbee-tests/node_modules/hyperbee/iterators/history.js:16:5)
Error at: ONE:
 Error: Decoded message is not valid
    at Object.decode (/Users/ward/code/play/hyperbee-tests/node_modules/hyperbee/lib/messages.js:446:39)
    at Object.decode (/Users/ward/code/play/hyperbee-tests/node_modules/compact-encoding/index.js:416:21)
    at Object.decode (/Users/ward/code/play/hyperbee-tests/node_modules/compact-encoding/index.js:432:14)
    at Hypercore._decode (/Users/ward/code/play/hyperbee-tests/node_modules/hypercore/index.js:789:23)
    at Hypercore._decryptAndDecode (/Users/ward/code/play/hyperbee-tests/node_modules/hypercore/index.js:630:24)
    at processTicksAndRejections (node:internal/process/task_queues:96:5)
    at async HyperBee.getBlock (/Users/ward/code/play/hyperbee-tests/node_modules/hyperbee/index.js:354:21)
    at async Batch.getBlock (/Users/ward/code/play/hyperbee-tests/node_modules/hyperbee/index.js:525:9)
    at async HyperBee.getRoot (/Users/ward/code/play/hyperbee-tests/node_modules/hyperbee/index.js:342:13)
    at async HistoryIterator.open (/Users/ward/code/play/hyperbee-tests/node_modules/hyperbee/iterators/history.js:16:5)
{ name: 'ONE', valid: true }
{ name: 'TWO', valid: true }
Error at: TWO:
 Error: Unknown wire type: 7
    at exports.skip (/Users/ward/code/play/hyperbee-tests/node_modules/protocol-buffers-encodings/index.js:35:9)
    at Object.decode (/Users/ward/code/play/hyperbee-tests/node_modules/hyperbee/lib/messages.js:232:18)
    at new Pointers (/Users/ward/code/play/hyperbee-tests/node_modules/hyperbee/index.js:36:29)
    at inflate (/Users/ward/code/play/hyperbee-tests/node_modules/hyperbee/index.js:67:10)
    at BlockEntry.getTreeNode (/Users/ward/code/play/hyperbee-tests/node_modules/hyperbee/index.js:249:20)
    at HyperBee.getRoot (/Users/ward/code/play/hyperbee-tests/node_modules/hyperbee/index.js:342:50)
    at processTicksAndRejections (node:internal/process/task_queues:96:5)
    at async HistoryIterator.open (/Users/ward/code/play/hyperbee-tests/node_modules/hyperbee/iterators/history.js:16:5)
Error at: ONE:
 Error: Unknown wire type: 6
    at exports.skip (/Users/ward/code/play/hyperbee-tests/node_modules/protocol-buffers-encodings/index.js:35:9)
    at Object.decode (/Users/ward/code/play/hyperbee-tests/node_modules/hyperbee/lib/messages.js:469:18)
    at Object.decode (/Users/ward/code/play/hyperbee-tests/node_modules/compact-encoding/index.js:416:21)
    at Object.decode (/Users/ward/code/play/hyperbee-tests/node_modules/compact-encoding/index.js:432:14)
    at Hypercore._decode (/Users/ward/code/play/hyperbee-tests/node_modules/hypercore/index.js:789:23)
    at Hypercore._decryptAndDecode (/Users/ward/code/play/hyperbee-tests/node_modules/hypercore/index.js:630:24)
    at processTicksAndRejections (node:internal/process/task_queues:96:5)
    at async HyperBee.getBlock (/Users/ward/code/play/hyperbee-tests/node_modules/hyperbee/index.js:354:21)
    at async Batch.getBlock (/Users/ward/code/play/hyperbee-tests/node_modules/hyperbee/index.js:525:9)
    at async HyperBee.getRoot (/Users/ward/code/play/hyperbee-tests/node_modules/hyperbee/index.js:342:13)

I am happy to offer any more context if needed.

Here is the code to replicate this bug:

import Corestore from 'corestore';
import Hyperbee from 'hyperbee';
import RAM from 'random-access-memory';

let key;
let originStore;

let encryptionKey;
// Commenting next line will resolve all issues
encryptionKey = Buffer.alloc(32, 'hello world');

{
  const store = new Corestore(RAM);

  const origin = store.get({
    name: 'origin',
    encryptionKey,
  });
  await origin.ready();

  const originDB = new Hyperbee(origin, {
    keyEncoding: 'utf8',
    valueEncoding: 'utf8',
  });

  let count = 0;
  setInterval(() => originDB.put('foo', 'update number: ' + count++), 1000);

  key = origin.key;
  originStore = store;
}

{
  const store = new Corestore(RAM);

  const s = store.replicate(true);
  s.pipe(originStore.replicate(false)).pipe(s);

  await test('ONE');
  // Try commenting next line
  await test('TWO');

  async function test(name) {
    const clone = store.get({
      key: key,
      encryptionKey,
    });
    await clone.update();

    const cloneDB = new Hyperbee(clone, {
      keyEncoding: 'utf8',
      valueEncoding: 'utf8',
    });

    clone.on('append', async () => {
      // Check that the clone is setup correctly
      console.log({
        name,
        valid:
          cloneDB.feed.peers.length > 0 &&
          (encryptionKey ? !!cloneDB.feed.encryption : true),
      });

      const stream = cloneDB.createHistoryStream({
        reverse: true,
        limit: 1,
      });

      stream?.on('data', (data) => console.log('Data at: ' + name, data));
      stream?.on('error', (err) =>
        console.log('Error at: ' + name + ':\n', err),
      );
    });
  }
}

package.json

{
  "type": "module",
  "dependencies": {
    "corestore": "^6.0.1-alpha.17",
    "hyperbee": "^1.8.3",
    "random-access-memory": "^4.1.0"
  }
}

Are the sparse cores working?

I am replicating the example from the Hyperbee documentation.
I'm referring to this one:
https://docs.holepunch.to/quick-start#hyperbee-sharing-append-only-databases

The example encourages me to make a couple of queries to see how, in the reading node, only a smaller amount of data has been transferred. It refers to the "sparse" type of query where only the necessary blocks to answer the query will be downloaded.

The problem I'm facing is that it only takes 1 or 2 queries for the entire dictionary (approximately 31 mb) to be transferred.

I then thought the problem was line number 11 (in the reader):
swarm.on('connection', conn => store.replicate(conn))

Fearing I was forcing synchronization, I removed it, but I'm again facing the same behavior.

Am I doing something wrong?
Is this how it works?

Thank you and sorry for the inconvenience

Node 18
Apple M1

Real deletes?

An append-only b-tree, but the API has a method for deletes. How do you achieve deletes when the underlying datastructure is append-only?

Add option to change the value encoding in sub()

Is there any issue with sub() taking a valueEncoding opt which overrides the parent's?

update() should accept options

db.update() does not accept any options and passes legacy options ({ ifAvailable: true, hash: false }) to Hypercore.
Since Hyperdrive changed update() to be local by default, this behaviour change needs documented in Hyperbee.

Hyperbee put with the same key

I am using hyperbee and have a question.

bee.put('key', 'value')

If I put in hyperbee with the same key twice, what will happen?
The first one is replaced with the second one, or two will be added with the same key?

Error in `createDiffStream` when using sub()

I have some code which calls createDiffStream() on a sub():

    const co = this.db.bee.checkout(other).sub(this._schemaDomain).sub(this._schemaName)
    return new Promise((resolve, reject) => {
      pump(
        co.createDiffStream(this.bee.version),
        concat(resolve),
        err => {
          pend()
          if (err) reject(err)
        }
      )
    })

This will sometimes produce an error:

TypeError: Cannot read property 'key' of undefined
    at TreeIterator.nextKey (/Users/paulfrazee/work/ctzn/node_modules/hyperbee/iterators/diff.js:110:21)
    at runMicrotasks (<anonymous>)
    at processTicksAndRejections (internal/process/task_queues.js:93:5)
    at async DiffIterator._next (/Users/paulfrazee/work/ctzn/node_modules/hyperbee/iterators/diff.js:168:43)
    at async DiffIterator.next (/Users/paulfrazee/work/ctzn/node_modules/hyperbee/iterators/diff.js:154:17)

I can currently work around it by running createDiffStream() on the root and filtering the results:

    const co = this.db.bee.checkout(other)
    const diffs = await new Promise((resolve, reject) => {
      pump(
        co.createDiffStream(this.bee.version),
        concat(resolve),
        err => {
          pend()
          if (err) reject(err)
        }
      )
    })
    const prefix = `${this._schemaDomain}\x00${this._schemaName}\x00`
    return diffs.filter(diff => {
      const key = (diff.right||diff.left).key
      if (key.startsWith(prefix)) {
        if (diff.left) diff.left.key = diff.left.key.slice(prefix.length)
        if (diff.right) diff.right.key = diff.right.key.slice(prefix.length)
        return true
      }
      return false
    })

SyntaxError: await is only valid in async functions and the top level bodies of modules

I tried to install hyperbee and run the usage example (as it is). It returns me this error message:

$ node hyperbee-usage.js 
/home/user/hyperbee-usage.js:8
await db.put('key', 'value')
^^^^^

SyntaxError: await is only valid in async functions and the top level bodies of modules
    at wrapSafe (node:internal/modules/cjs/loader:1024:16)
    at Module._compile (node:internal/modules/cjs/loader:1072:27)
    at Object.Module._extensions..js (node:internal/modules/cjs/loader:1137:10)
    at Module.load (node:internal/modules/cjs/loader:973:32)
    at Function.Module._load (node:internal/modules/cjs/loader:813:14)
    at Function.executeUserEntryPoint [as runMain] (node:internal/modules/run_main:76:12)
    at node:internal/main/run_main_module:17:47

Write/read access management, multi-master replicas

Is there a way where I can set rules for read/write accesses for particular ip? If not ip, maybe some hyper hash or something.

I look though the docs and can’t find that.

The most basic example:

I wanna make a few different machines with allow-write access
I wanna make everyone else allow-read access
(Think or this idea like decentralized database, with multiple master-replicas that are automatically synchronized with each other)

Maybe I look at the wrong tool, and for that there’s not hyperbee, but hyperdrive, or hyperswarm, or hyperspace, or hypercore, or hyper-something-else. Please guide me 😰

Currently I assume there only one approach with probably more typical database approach: master+slaves replicas. No multi-master approach. Is that so?

rw lock for batches

currently concurrent read and write ops on a single batch is not safe, it should be

`Session is closed` timing issue in createDiffStream

I'm trying to convert the range watcher into an rxjs observable, based on this test, and I'm running into a HypercoreError: SESSION_CLOSED: Session is closed error when running the following code:

import Hyperbee from "hyperbee";
import Hypercore from "hypercore";
import RAM from "random-access-memory";
import { from, concatMap, catchError, throwError } from "rxjs";
const core = new Hypercore(RAM);
const db = new Hyperbee(core, {
  keyEncoding: "utf-8",
  valueEncoding: "utf-8",
});

const watcher = db.watch();
await watcher.ready();

const withObservable = async () => {
  const watcher$ = from(watcher);

  watcher$
    .pipe(
      concatMap(([current, previous]) =>
        from(current.createDiffStream(previous)).pipe(
          catchError((err) => {
            console.error("Error in inner observable:", err);
            return throwError(() => err);
          }),
        ),
      ),
      catchError((err) => {
        console.error("Error in main observable:", err);
        return throwError(() => err);
      }),
    )
    .subscribe({
      next: (result) => console.log("result", result),
      error: (err) => console.error("Error in subscription:", err),
      complete: () => console.log("complete"),
    });

  await db.put("a", "1");
  // await new Promise((r) => setTimeout(r));
  await db.put("b", "2");
};

await withObservable();

The full error is:

Error in inner observable: HypercoreError: SESSION_CLOSED: Session is closed
    at Hypercore.get (/Users/jun/dev/iios/libraries/api/node_modules/hypercore/index.js:741:38)
    at process.processTicksAndRejections (node:internal/process/task_queues:95:5)
    at async Batch.getBlock (/Users/jun/dev/iios/node_modules/hyperbee/index.js:637:19) {
  code: 'SESSION_CLOSED'
}
Error in main observable: HypercoreError: SESSION_CLOSED: Session is closed
    at Hypercore.get (/Users/jun/dev/iios/libraries/api/node_modules/hypercore/index.js:741:38)
    at process.processTicksAndRejections (node:internal/process/task_queues:95:5)
    at async Batch.getBlock (/Users/jun/dev/iios/node_modules/hyperbee/index.js:637:19) {
  code: 'SESSION_CLOSED'
}
Error in subscription: HypercoreError: SESSION_CLOSED: Session is closed
    at Hypercore.get (/Users/jun/dev/iios/libraries/api/node_modules/hypercore/index.js:741:38)
    at process.processTicksAndRejections (node:internal/process/task_queues:95:5)
    at async Batch.getBlock (/Users/jun/dev/iios/node_modules/hyperbee/index.js:637:19) {
  code: 'SESSION_CLOSED'
}

If I add the setTimeout in between the db.put()s then there's no error, and I do get the expected results:

result { left: { seq: 1, key: 'a', value: '1' }, right: null }
result { left: { seq: 2, key: 'b', value: '2' }, right: null }

Closing a hyperbee?

Is there a way for hyperbee to be closed?

`createHistoryStream` on `sub` returns the history with empty keys

Hey guys,

I was trying the sub namespacing of hyperbee and I notice that running a createHistoryStream over sub databases gets the history of the entire database and also for some reason with empty keys.

const ram = require('random-access-memory')
const hypercore = require('hypercore')
const Hyperbee = require('hyperbee')

function config() {
  const feed = hypercore(ram)

  const db = new Hyperbee(feed, {
    keyEncoding: 'utf-8',
    valueEncoding: 'utf-8'
  })

  return db
}

;(async () => {
  const db = config()

  const sub = db.sub('prefix-a')

  await db.put('key0', 'value0')

  for await (const item of sub.createHistoryStream()) {
    console.log('sub', item) // sub { type: 'put', seq: 1, key: '', value: 'value0' }
  }
})()

Change notification?

Is there a way to get notifications about changes, like db.watch in hyperdb?

Share batch between subs

It'd be nice to use a single batch between several subs.

e.g. sub.batch({batch: dbBatch}) to specify a parent batch to append onto with the sub batch. Specifying an existing batch to add onto with a sub vs using batch.sub(prefix) is useful because people often deal with db instances instead of memorizing prefixes.

This is useful when you're inserting or deleting from several subs at once, for example here in hyperbeedeebee where I'm inserting keys into several subs used for indexing documents.

Documentation re: read / write permissions

Hypercores are not inherently mult-writer. As I understand it, multi-writer functionality must be implemented on top of hypercore. See multifeed

But Hyperbee has no documentation about read / write capabilities. Is it possible to read and write to shared addresses?

Connect Two Node Server

How yo connect two server?

batch.clear() feature

hyperbee has ]batch.flush() which is similar to batch.write() on levelup

batch.clear() example https://github.com/Level/levelup#dbbatch-chained-form

it does not have concept of batch.clear() is there a reason for this?

there is batch.blocks.clear() but if I use it I get an error on batch.flush()

return `Promise<{seq}>` from `put` calls

This would be useful for knowing the seq of a newly inserted item

Is it possible to have an on('update') event on subs?

Is it possible to watch the Hypercore on('append') event and analyze the latest seq to figure out which keys were updated?

If it is not convenient to add this feature in Hyperbee, what can I do to implement that on a higher level of abstraction?

Possible to create history stream on sub database?

My project has two sub databases in a single hyperbee called payload DB and a meta DB that each use the same key but have different values
Given a sub database called metaDB expect only keys from that sub database to stream.
Currently this returns "data" twice once for each sub DB and no clear way to deconflict which "data" other than inspecting the values

  watch(cb) {
    const rs = this.metaDB.createHistoryStream({ live: true })
    rs.on("data", (data) => {
      console.log("watch", data);
      cb(null, data);
    });
  }

Here is how they are initialized

db = new Hyperbee(core, {
  keyEncoding: "utf-8",
  valueEncoding: "binary",
});

const metaDB = db.sub("meta", {
  valueEncoding: "utf-8",
});

const dataDB = db.sub("data", {
  valueEncoding: "binary",
});

returns data like

watch {
type: 'put',
seq: 105,
key: '1ea746dc-d050-41fd-bbba-5d33a50afdc3',
value: '{"type":"text/html","etag":"a622c40426aedb88f24559877cb8c6df66ed527c","template":"article"}'
}
WATCH [ '2d723249-5d0e-46ba-b1c4-df0014c107ab' ]
SSE:RELOAD 2d723249-5d0e-46ba-b1c4-df0014c107ab
watch {
type: 'put',
seq: 106,
key: '1ea746dc-d050-41fd-bbba-5d33a50afdc3',
value: '<h2>hello world</h2>'
}
WATCH [ '2d723249-5d0e-46ba-b1c4-df0014c107ab' ]
SSE:RELOAD 2d723249-5d0e-46ba-b1c4-df0014c107ab

My use case is live reloading large dependency trees in the browser when a file changes so I only need the meta data to build that graph not the values of the "payload"
Maybe there is another pattern that would better support? thinking about 2 hyperbees now.

Values stored as keys when using sub of sub

I'm getting values stored as keys when using sub, minimal example to reproduce:

let Hypercore = require('hypercore');
let Hyperbee = require('hyperbee');
let myCore = new Hypercore('/myCore2', {valueEncoding: 'utf-8'})
let myBee
let a
myCore.ready(async () => {
	console.log('core ready')
	myBee = new Hyperbee(myCore,{keyEncoding: 'utf-8', valueEncoding: 'binary'})
	await myBee.ready()
	console.log('bee ready')
	let mySubBee = myBee.sub(myCore.key.toString('hex'))
	let mySubSubBee = myBee.sub('fe766f9a420f08c28f4c067fd7a5a01f86aa6e198e44dd68af1470c1db611c09')
	await mySubSubBee.put('e3e03a4a511d49a87468f13d1aa297081812d51e6138cfc70071d237a561a2c9','testvalue')
	await mySubSubBee.put('ca7bd4311e9d21e4c8d04162970998d715b8b92c92ee4ff434357f312647b401','anothertestvalue')
	for await (let val of mySubSubBee.createReadStream()){
		console.log(val)
	}		
});

Gives:

{
  seq: 2,
  key: 'ca7bd4311e9d21e4c8d04162970998d715b8b92c92ee4ff434357f312647b401\x1A\x10anothertestvalue',
  value: null
}
{
  seq: 1,
  key: 'e3e03a4a511d49a87468f13d1aa297081812d51e6138cfc70071d237a561a2c9\x1A\ttestvalue',
  value: null
}

I expected the value fields to be populated with testvalue and anothertestvalue

"dnshyper" - domain name service for hyperprotocol (concept)

Hi all!

1. feature-name

"dnshyper" - domain name service for hyperprotocol (concept)

2. feature-description

2.1 before:

hyper://94f0cab7f60fcc2a711df11c85db5e0594d11e8a3efd04a06f46a3c34d03c418/posts/agregore-unity

2.2 after:

hyper://websitedomain.com/posts/agregore-unity

3. dnshyper - domain name service for hyperprotocol

94f0cab7f60fcc2a711df11c85db5e0594d11e8a3efd04a06f46a3c34d03c418 -> websitedomain.com

4. Why dnshyper - domain name service for hyperprotocol?

The Domain Name System (DNS) is the phonebook of the Internet. Humans access information online through domain names, like nytimes.com or espn.com. Web browsers interact through Internet Protocol (IP) addresses. DNS translates domain names to IP addresses so browsers can load Internet resources.
Each device connected to the Internet has a unique IP address which other machines use to find the device. DNS servers eliminate the need for humans to memorize IP addresses such as 192.168.1.1 (in IPv4), or more complex newer alphanumeric IP addresses such as 2400:cb00:2048:1::c629:d7a2 (in IPv6).
dnshyper - domain name service for hyperprotocol has the same goal as traditional dns, but the concept is for p2p. But... the idea is that it is the people who manage the domains of the site instead of the company.

5. Notes

I'm not promoting or wanting to promote any product, company, person, technology, service, business. My purpose in referencing the links is for bibliography only.
What do you all think about the idea?

6. References

https://www.cloudflare.com/learning/dns/what-is-dns/

Bug: Encoding issue when loading hyperbee on browser

Summary

I'm running into an encoding issue with Hyperbee on the browser, using hypercore-next. The content arrives in UInt8-arrays instead of the expected Buffer format.

See https://gist.github.com/HDegroote/cf6d3c09a1f07e8dd7b56a8f00168bde for code to replicate. Do note that you need to bundle the code to run it in the browser. I can share bundled code or a webpack config which worked for me if relevant.

Description

I'm using dht-relay to download a hyperbee over hyperswarm. It works as expected on Node, but on the browser the key and value are in Uint8Array format at this point (on Node these are Buffers): https://github.com/hypercore-protocol/hyperbee/blob/4a2f2ab6e17d7c42031576ad3a73209bc667c4bf/index.js#L245

So when the hyperbee-specific decoding is applied, it either crashes (on JSON), or returns something unexpected (e.g. for utf-8).
When running the example in the gist, the output is as expected on Node:

Created swarm--joining on key 9dd044511ea8227e4b992cc0e074b77be3fdc9895fcd6ef20a3f26630cf41548
joined swarm -- requesting block (can take up to ~30s)

//Debugging output: printing the values just before executing 
//https://github.com/hypercore-protocol/hyperbee/blob/4a2f2ab6e17d7c42031576ad3a73209bc667c4bf/index.js#L245
seq:  1
value:  <Buffer 22 76 61 6c 75 65 20 30 22>
key:  <Buffer 62 6c 6f 63 6b 20 23 30>

// Result
block 0:  { seq: 1, key: 'block #0', value: 'value 0' }

When executing on the browser, the code fails:

Created swarm--joining on key 9dd044511ea8227e4b992cc0e074b77be3fdc9895fcd6ef20a3f26630cf41548
joined swarm -- requesting block (can take up to ~30s)

//Debugging output: printing the values just before executing 
//https://github.com/hypercore-protocol/hyperbee/blob/4a2f2ab6e17d7c42031576ad3a73209bc667c4bf/index.js#L245
seq:  1
key: Uint8Array(8) [ 98, 108, 111, 99, 107, 32, 35, 48 ]
value: Uint8Array(9) [ 34, 118, 97, 108, 117, 101, 32, 48, 34 ]

// Crashes when attempting to JSON decode the value:
Uncaught (in promise) SyntaxError: JSON.parse: unexpected non-whitespace character after JSON data at line 1 column 3 of the JSON data

The full trace for the hyperbee part is:

Uncaught (in promise) SyntaxError: JSON.parse: unexpected non-whitespace character after JSON data at line 1 column 3 of the JSON data
    decodeJSON index.js:52 // JSON lib
    final index.js:248 // This and following are hyperbee line nr's
    get index.js:569
    get index.js:428

Relevant Versions

├─ [email protected]
│ ├─ hypercore-crypto@^3.2.1
├─ [email protected]
├─ [email protected]
│ ├─ @hyperswarm/dht@^6.0.1
├─ @hyperswarm/[email protected]
│ ├─ @hyperswarm/dht@^6.0.1
│ ├─ @hyperswarm/secret-stream@^6.0.0

Update docs request

Hello,

It would make sense, and considerably lower the entry bar, to have readme.md being a complete example.

Right now the 2nd line errors with feed is not defined, and reading https://github.com/hypercore-protocol/hypercore it's not obvious where feed comes from either.

To finish up the readme, it would also be nice to just add a quick line to print all the keys or something, as not everyone knows how to handle db.createReadStream nor it's clear from docs