level / level-js Goto Github PK

Continuing from #68.

TBD if this should go in v3.

Hey, I'm trying to get levelgraph browser tests working and changing this to return 0 fixes a whole tonne of failures.

Level.prototype._approximateSize = function() {
  return 0 //throw new Error('Not implemented')
}

@rvagg suggested it was okay when you asked about it on #stackvm

Would you take a PR for this?

Cheers.

How does the multilevel syncing work?

Because with modern browserify, Buffer is an Uint8Array.

https://github.com/Level/level.js/blob/bb19ca3c797349c5cd27fece4db0bc0ad6e83788/index.js#L67-L73

idb-wrapper now detects if shimIndexedDB is set.

So using level-js with IndexedDBShim works better in environments only supporting webSQL.

iOS 8 homescreen - I'm looking at you!

does this really need levelup?

anyone can do this, patch welcome :)

Current: Chrome latest and FF latest. Add:

• Edge
• Safari
• IE11
• More Chrome or FF versions? Nah, they're evergreen.
• ..?

@jez0990 and me are trying to make LevelGraph run in the browser on top of Level.js.

However we have a couple of failing tests on del support: stuff do not get deleted.

Any ideas in how to debug them?

we'll need that for replication.

We get random errors (screenshot from Sauce Labs):

see the TODO by onItem: https://github.com/maxogden/level.js/blob/master/iterator.js#L61

this test (simplified for this issue):

test('iterator create snapshot correctly', function (t) {
    db.put('foobatch1', 'bar1', function(err){
      t.error(err)
      var iterator = db.iterator()
      db.del('foobatch1', function () {
        iterator.next(function (err, key, value) {
          t.notOk(err, 'no error')
          t.ok(key, 'got a key')
          t.equal(key.toString(), 'foobatch1', 'correct key')
          t.equal(value.toString(), 'bar1', 'correct value')
          iterator.end(t.end.bind(t))
        })
      })
    })
  })

is failing:

# iterator create snapshot correctly
default_stream.js:27 ok 419 no error
default_stream.js:27 not ok 420 got a key
default_stream.js:27   ---
default_stream.js:27     operator: ok
default_stream.js:27     expected: true
default_stream.js:27     actual:   undefined
default_stream.js:27     at: Test.assert (http://localhost:9966/test.js:12977:17)
default_stream.js:27   ...
iterator-test.js:467 Uncaught TypeError: Cannot read property 'toString' of undefined

@maxogden do you know what's going on here?

I'm running into the exception that toArrayBuffer is not a method of value on line #68. I wonder if this code in #L67-L68 is correct and what the purpose of it is? I really have no clue, for now I just disabled #L67-L69 and "It works for me".

I get the following stack trace when sending level-js through browserify and factor-bundle:

"browserify": "^6.1.0"
"levelup": "^0.19.0"
"factor-bundle": "^2.3.0"

Rewrite value instanceof ArrayBuffer etc.

Since JSON is the native encoding to IndexedDB, it shouldn't be necessary to stringify data before storing it. Is it AbstractLevelDOWN that does the conversion? Does it make sense to replace 'json' encoding with raw:true when the database is opened?

I'm using the following to make sure I can cleanly work with JSON in both node and browser environments, but it feels like a hack:

package.json:

  ...
  "browser": {
    "leveldown": "level-js",
  }
  ...

index.js:

var levelup = require('levelup');
var leveldown = require('leveldown');
var leveljs = require('level-js');

var options = {
  db: leveldown
};
if (leveljs !== leveldown) {
  options.valueEncoding = 'json';
}
var db = levelup(name, options);

for some reason the tests seem to have stopped working in Canary: https://ci.testling.com/maxogden/level.js

also they don't work in FF right now

if anyone feels like debugging, feel free! I may not get to it for a while

As it's been removed from abstract-leveldown.

https://github.com/Level/level.js/blob/3a9b36c66df096a94258df5e9778ec50c20c9470/index.js#L128-L134

Same as memdown, see Level/memdown#61.

Not sure if this is something missing from the dependencies in this or maybe just my browserify inexperience, but it seems to want leveldown even if it isn't really using it.

Installing leveldown made browserify happy. If it's not just user error on my part it might help to put it in the package.json as a dep.

https://hacks.mozilla.org/2014/02/localforage-offline-storage-improved/

perhaps interesting, perhaps not.

Why is everything returned as an ArrayBuffer by default?
I expected the json that I put in to come out as json, and passing {asBuffer: false} to db.get gave me what I expected, but why not do that by default?

I propose to move to zuul to run tests and to SauceLabs for running it in the CI. SauceLabs offers free accounts for OSS.

I am using those for LevelGraph and it works really well, see:

https://github.com/mcollina/levelgraph/blob/master/.zuul.yml
https://github.com/mcollina/levelgraph/blob/master/.travis.yml
https://github.com/mcollina/levelgraph/blob/master/package.json#L8-L9

(I am using Mocha for LG - my bad - but tape is also supported)

C:_data\leveljs>npm install level-js
npm http GET https://registry.npmjs.org/level-js
npm http 304 https://registry.npmjs.org/level-js
npm http GET https://registry.npmjs.org/abstract-leveldown
npm http GET https://registry.npmjs.org/isbuffer/0.0.0
npm ERR! not found: git
npm ERR!
npm ERR! Failed using git.
npm ERR! This is most likely not a problem with npm itself.
npm ERR! Please check if you have git installed and in your PATH.

npm ERR! System Windows_NT 6.1.7601
npm ERR! command "C:\Program Files\nodejs\node.exe" "C:\Program Files\nodejs\node_modules\npm\bin\npm-cli.js" "install" "level-js"
npm ERR! cwd C:_data\leveljs
npm ERR! node -v v0.10.7
npm ERR! npm -v 1.2.21
npm ERR! code ENOGIT
npm http 304 https://registry.npmjs.org/isbuffer/0.0.0
npm http 304 https://registry.npmjs.org/abstract-leveldown
npm ERR!
npm ERR! Additional logging details can be found in:
npm ERR! C:_data\leveljs\npm-debug.log
npm ERR! not ok code 0

This is the culprit: https://github.com/maxogden/level.js/blob/master/index.js#L66-L72

Copying operations is probably a better solution.

I wonder if level.js works with https://github.com/axemclion/IndexedDBShim, haven't tested it yet. someone should test it and report back!

Without updating anything yet.

It doesn't pass the iterator snapshot test currently, because level.js creates the IDB cursor lazily (on next). It should create it in the Iterator constructor instead.

Added in #79.

Background: https://github.com/Level/abstract-leveldown/blob/master/UPGRADING.md#_isbuffer-method-removed

Must update browserify before this.

... on cdnjs.cloudflare.com / unpkg.org
Currently https://unpkg.org/files/[email protected]/ shows no file

Because closing happens in a browser background thread.

https://github.com/Level/level.js/blob/8c9f713e8db7e0383a733504c5996e507ae7fff5/index.js#L118-L121

If I try to put array buffer into a db that has valueEncoding set to 'binary' I get an error about

message: "value cannot be an empty ArrayBuffer"
name: "WriteError"

as far as I can tell this is due to it being improperly turned into a Uint8 at some point, I managed to trace it to here where the array Buffer failed the Buffer.isBuffer test, but then when new Buffer(ArrayBuffer) is called it uses ArrayBuffer.prototype.length instead of ArrayBuffer.prototype.byteLength, so I'll likely have to open an issue there as well

npm install gives this warning:

npm WARN deprecated [email protected]: Please update to the latest object-keys

npm ls object-keys shows it as an indirect dependency of level-js:

│     └─┬ [email protected]
│       └─┬ [email protected]
│         └── [email protected]

Reading npm info shows that [email protected] is the current version, and depends on an outdated version of xtend.

• badges

https://github.com/Level/level.js/blob/e819338e87aa305e57ec9c2fe58168ab5dca6da3/index.js#L47-L53

Guys,

I've been trying to port my application from Node to the browser using level-browserify. My code depends heavily on binary keys but unfortunately all write operations fail with the error "DataError: Data provided to an operation does not meet requirements." (Fx 45.0a2). When looking up the specs binary keys are not part of IndexedDB 1.0 but they are going to be part of IndexedDB Second Edition.

I've patched IDBWrapper to map Uint8Array keys to plain arrays in the meantime, but I think the level-js package is a better place to do this, especially considering issue #46. I was wondering if a patch that handles binary keys would be welcome. Otherwise I would suggest at least a patch that throws if keyEncoding is binary. Or maybe if idb-blob-store is the future I better open this issue there?

Seems that Safari is not supported here.

I am trying to run the tests and I got these kind of errors:

TypeError: 'undefined' is not an object (evaluating 'this.db.iterate')

TypeError: 'undefined' is not an object (evaluating 'this.idb.put')

right now the tests don't pass in latest chrome. Not sure if it was something in chrome, idb-wrapper, or abstract-leveldown that changed but if someone has some time to dig in and try and make the tests pass it would be much appreciated

From #60 (comment):

Iterators should by default keep reading from the underlying IDB cursor, to fulfill the snapshot guarantee

• Have the IDB cursor fill a cache (similar to how leveldown works with highWaterMark)
• Have Iterator.prototype._next() read from the cache.

This cleanly separates the two mechanisms (IDB pushes, Iterator pulls).

I feel it's time to start focusing on this repository now. I'd really like to get this done and then ship level as one package to rule them all 😄

But first, lets breath some air into it.

However, I'm unsure of where we stand, like what should be done to make this work well.

Some links:

• #46
• ipfs/js-ipfs-repo#127

Lets use this issue to summarize what's needed and which direction we want to take it.

@vweevers @timkuijsten I feel you guys have have been digging around the most in this code and most likely have the most valuable input.

Some questions:

• what is the big problem with the current implementation?
• what are we fixing, except for using latest abstract-leveldown?

From #60 (comment):

level.js has custom logic for serializing buffers, typed arrays etc. Similar to what we did in memdown, we should strip most of the logic, prefer Buffer over typed arrays, and use _serialize* as the only code path for serialization.

• Have the abstract-leveldown test suite assume binary keys are supported (remove process.browser checks) (Level/abstract-leveldown#121) Remove process.browser checks and move binary key tests to isolated test file(s) that can be skipped by implementations.
• Or until then, add a custom test to level-js
• Assume use of modern browserify (where Buffer is a subclass of Uint8Array)
• Feature-detect binary key support
• Update to latest abstract-leveldown and implement _serializeKey as (simplified):

var supportsBinaryKeys = true;

Level.prototype._serializeKey = function (key) {
  return !supportsBinaryKeys && this._isBuffer(key)
    ? String(key)
    : key
}

This means the test suite can pass in browsers that support binary keys as well as those that don't (IE, Edge, Chrome 49).

@ralphtheninja when the v3 milestone is done, let's do an RC here too?

idbwrapper has outlived its usefulness to us, and has performance issues

fastest throughput I can get with idbwrapper is 60KB/s. Without it I can get 1MB/s (using substacks idb-blob-store module which uses the IDB API directly)

const leveldown = require('level-js')
const levelup = require('levelup');
const db = leveldown('my-database-name')
levelup(db);

Causes the assertion error:

".status required, old abstract-leveldown"

The status property is documented in the README of:
https://github.com/Level/abstract-leveldown
AbstractLevelDOWN#status

(As a workaround I'm setting db.status = 'unknown' before doing levelup(db))