yunong / node-fash Goto Github PK

View Code? Open in Web Editor NEW

22.0 22.0 12.0 665 KB

Node Consistent Hashing Library

JavaScript 99.84% Shell 0.16%

node-fash's People

Contributors

Stargazers

Watchers

Forkers

mcavage bawerd fkuo zachrip tritondatacenter jdrew1303 bahamas10

node-fash's Issues

Deserializing a 10 million node hash takes way too long.

Add redis backend

Want diff tool to compare topologies

After a reshard it would be useful to be able to easily generate the diff of pnode <-> vnode mappings between two topologies.

Please add a bunyan serializer, and get rid of all the `log.<level>()` invocations in fash

With large ring sizes, setting my log level to trace is effectively unusable, as fash just spews whenever i touch it. Please add a bunyan serializer so I can dump its state when I want to, and get rid of the log invocations in mainline paths.

enable read-only status on vnodes

creating a large leveldb ring > 10000000 vnodes causes node to run out of memory.

Even if max-old-space-size is set to 81920, node still crashes with OOM errors at around 15-20G of heap space.

empty pnode argument to fash create does not throw

this should throw but doesn't

[root@15e718de-9f88-4c1d-80e9-2d9610a206fb (emy-10:manta0) ~]# /opt/smartdc/manta-deployment/node_modules/.bin/fash create -v 100 -l foo -b leveldb -p
[root@15e718de-9f88-4c1d-80e9-2d9610a206fb (emy-10:manta0) ~]#

make leveldb configurable

cli should return error codes on error

add dtrace probes

fash create doesn't fail when vnode count isn't passed in

[root@c53b592e-5818-4cea-96c7-d7bfa64684b8 (emy-10:manta0) /opt/smartdc/manta-deployment]# /opt/smartdc/manta-deployment/node_modules/.bin/fash create -v -p '2.moray.coal.joyent.us
3.moray.coal.joyent.us
4.moray.coal.joyent.us' -b leveldb -l /var/tmp/manta-ring
[root@c53b592e-5818-4cea-96c7-d7bfa64684b8 (emy-10:manta0) /opt/smartdc/manta-deployment]# ./node_modules/.bin/fash get_pnodes -l /var/tmp/manta-ring/ -b leveldb
{ jse_shortmsg: 'unable to load ring from db',
  jse_summary: 'unable to load ring from db: : Invalid argument: /var/tmp/manta-ring/: does not exist (create_if_missing is false)',
  jse_cause:
   { jse_shortmsg: '',
     jse_summary: ': Invalid argument: /var/tmp/manta-ring/: does not exist (create_if_missing is false)',
     jse_cause: [Error: Invalid argument: /var/tmp/manta-ring/: does not exist (create_if_missing is false)],
     message: ': Invalid argument: /var/tmp/manta-ring/: does not exist (create_if_missing is false)' },
  message: 'unable to load ring from db: : Invalid argument: /var/tmp/manta-ring/: does not exist (create_if_missing is false)' }

Add getPnodes() method

Add getVnodeData() Function for Fast Look up Vnodes with Data.

space should not be parsed as a pnode

/opt/smartdc/manta-deployment/node_modules/.bin/fash create -v 100 -p '2.moray.coal.joyent.us 3.moray.coal.joyent.us 4.moray.coal.joyent.us ' -b leveldb -l /var/tmp/manta-ring

results in a ring like this:

[root@c53b592e-5818-4cea-96c7-d7bfa64684b8 (emy-10:manta0) ~]# /opt/smartdc/manta-deployment/node_modules/.bin/fash print_hash -l /var/tmp/manta-ring/ -b leveldb
{"vnodes":100,"pnodeToVnodeMap":{"":{"0":1,"4":1,"8":1,"12":1,"16":1,"20":1,"24":1,"28":1,"32":1,"36":1,"40":1,"44":1,"48":1,"52":1,"56":1,"60":1,"64":1,"68":1,"72":1,"76":1,"80":1,"84":1,"88":1,"92":1,"96":1},"2.moray.coal.joyent.us":{"1":1,"5":1,"9":1,"13":1,"17":1,"21":1,"25":1,"29":1,"33":1,"37":1,"41":1,"45":1,"49":1,"53":1,"57":1,"61":1,"65":1,"69":1,"73":1,"77":1,"81":1,"85":1,"89":1,"93":1,"97":1},"3.moray.coal.joyent.us":{"2":1,"6":1,"10":1,"14":1,"18":1,"22":1,"26":1,"30":1,"34":1,"38":1,"42":1,"46":1,"50":1,"54":1,"58":1,"62":1,"66":1,"70":1,"74":1,"78":1,"82":1,"86":1,"90":1,"94":1,"98":1},"4.moray.coal.joyent.us":{"3":1,"7":1,"11":1,"15":1,"19":1,"23":1,"27":1,"31":1,"35":1,"39":1,"43":1,"47":1,"51":1,"55":1,"59":1,"63":1,"67":1,"71":1,"75":1,"79":1,"83":1,"87":1,"91":1,"95":1,"99":1}},"algorithm":{"NAME":"sha256","MAX":"FFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFF","VNODE_HASH_INTERVAL":"028f5c28f5c28f5c28f5c28f5c28f5c28f5c28f5c28f5c28f5c28f5c28f5c28f"},"version":"2.1.0"}

note the first empty pnode.

add deserialize_ring command to CLI

Enable deserializing of a ring topology.

Add leveldb backend

invoking addData with data=null removes vnode.

doesn't work on node 0.10

Your bignum dep is too old:

npm install
npm http GET https://registry.npmjs.org/fash/0.1.1
npm http 304 https://registry.npmjs.org/fash/0.1.1
npm http GET https://registry.npmjs.org/bignum/0.4.1
npm http 304 https://registry.npmjs.org/bignum/0.4.1

> [email protected] install /home/mcavage/work/node-libmanta/node_modules/fash/node_modules/bignum
> node-waf configure build

sh: line 1: node-waf: not found
npm ERR! [email protected] install: `node-waf configure build`
npm ERR! `sh "-c" "node-waf configure build"` failed with 127
npm ERR!
npm ERR! Failed at the [email protected] install script.
npm ERR! This is most likely a problem with the bignum package,
npm ERR! not with npm itself.
npm ERR! Tell the author that this fails on your system:
npm ERR!     node-waf configure build
npm ERR! You can get their info via:
npm ERR!     npm owner ls bignum
npm ERR! There is likely additional logging output above.

npm ERR! System SunOS 5.11
npm ERR! command "/home/mcavage/.nvm/v0.10.3/bin/node" "/home/mcavage/.nvm/v0.10.3/bin/npm" "install"
npm ERR! cwd /home/mcavage/work/node-libmanta
npm ERR! node -v v0.10.3
npm ERR! npm -v 1.2.17
npm ERR! code ELIFECYCLE
npm ERR!
npm ERR! Additional logging details can be found in:
npm ERR!     /home/mcavage/work/node-libmanta/npm-debug.log
npm ERR! not ok code 0
make: *** [all] Error 1