mostlyserious / riak-js

Riak client for Javascript

riak-js

Node.js client for Riak.

Installation

npm install riak-js

Features and documentation

Documentation

Follow updates on Twitter: @riakjs

If you need a tiny little lib, you can use riak-dc

Changelog

1.1.0

• Support Riak 2.0 (see #225)

1.0.3

• Fix a bug with buckets stream
• Add possibility to overwrite timeout value in Poolee for any requests

1.0.2

• Fix javascript error when we use data with an error
• options.keys / options.buckets are now taken into account with client.keys / client.buckets

1.0.1

• Remove useless console.log
• Attach response headers to error, enabling for example tombstone detection on 404.
• Allow user to override the riak port with HTTP_PORT / PB_PORT environement variables
• Fix a streaming error with node.js 0.10.x

1.0.0

• Using riakpbc 2.1 : dropping support for node 0.8 / dropping support for non-streams on protobuf for keys and buckets retrievals.
• Fix Travis CI
• Cleaner dependencies

0.10.2

• Fix HTTP status codes intepreted as errors when using the connection pool (github.com/chirag04 and github.com/domasx2)

0.10.1

• Support for HTTP connection pooling (github.com/andrewjstone)
• Fix unwanted errors in ping and save operation callbacks (github.com/matehat)

0.10.0

• Preliminary support for Protocol Buffers via a single connection.
• Fetch siblings by default, restoring previous riak-js behavior
• Populate meta.links when saving an object based on the response's HTTP headers (github.com/accelerated)
• Fix content length override. Allows handing in streaming data from e.g. connect.js. (github.com/englercj)
• Only set the language attribute on map and reduce phases. Avoid setting it on phases where it's not according to spec. (github.com/accelerated)

0.9.3

• Switch testing framework to mocha/should.js (github.com/kenperkins)
• Remove requirement of a Unix system from package.json (github.com/yawnt)
• Fix a global leak of HttpSearchMeta (github.com/brianedgerton)

0.9.2

• Fix Search options so it handles hostname and port (github.com/kenperkins)
• Fix unwanted vector clock overwriting during link walking (github.com/TauZero)
• Handle connection errors gracefully so that they don't kill the process

0.9.1

• Request instrumentation for metrics, logging, etc.
• Fix a leak in Httpmeta (github.com/rkusa), (#134)
• Handle content length properly for Unicode data (#132)
• Fix MapReduce options so it handles hostname and port (github.com/spencergibb)
• Add support for link-walking (github.com/TauZero)
• Handle JSONified error messages (github.com/spencergibb)
• Fix parsing of links on get requests (github.com/TauZero)

0.9.0

• Complete rewrite in JavaScript (two reasons: better debugging in production deployments, and more people can contribute)
• Support Riak 1.0 features
• First class support for streams (Luwak, key listing, Map/Reduce, logging)
• Faster testing with node-seq + assert
• Extra HTTP headers can now be passed in via headers (for instance custom usermeta, X-Riak-Meta-*) - these will override whatever has been set elsewhere
• addSearch is now search
• The project is open for other implementations
• Removed Protocol Buffers support – contributions welcome (extend Client in a file called protobuf-client.js along with a protobuf-meta.js)
• A client-side impl is also possible (look at https://github.com/ded/reqwest)
• Support for Riak Search, including adding and removing documents directly. Search is namespaced on the client separately: riak.search.find('users', 'email:[email protected]')
• Moved MapReduce to a separate namespace on the client to separate concerns in a nicer way: riak.mapreduce.add('users').run()

TODO

• Unify meta in all responses (check getAll, chunked map/reduce) - meta should allow for fine-grained mappings
• SSL support
• Use path internally but don't expose it as API (see path branch), to prepare for new URIs in Riak 1.0
• More rigorous test suite (port from master)

LATER

• Integrate test backend

0.4.1

• Add search params [gpascale]
• SSL client [isaacs]
• Last-modified date patch [siculars]
• Add meta.noError404 so that you can prevent 404s from showing up as errors [syrio]
• Deprecation warnings

0.4.0

• db.update: convenience method for updating documents
• Changed debug behaviour: always log to stdout. Not providing a callback will output to console.log regardless of the setting

0.4.0rc3

• Allow headers to override, useful for reverse proxies sitting between node and Riak
• URI-encoding bucket/key is now an option (encodeUri)
• Add documentation for map/reduce arguments
• Remove agent from meta, which should fix critical bug (#64, #67)
• Allow logging to alternate streams. And default to stderr, not stdout (thanks Mark!)
• Add docs for host/port (thanks Ken!)
• Add the new presort property for Riak Search (thanks Greg!)

0.4.0rc2

• Major docs update => http://riak-js.org
• keys=stream emits keys events
• walk now accepts noJSON
• Support Map/Reduce timeout option
• Protobuf interface refactor (still experimental support in 0.4)
• Minor bug fixes

0.4.0rc1

• node 0.4.x compatibility
• npm 1.0 compatibility
• Minor bug fixes

0.3.6

• Fix index.js and package.json for npm compat
• Fix recursing usermeta
• Fix for luwak location parsing
• Updates related to Riak Search operations

0.3.0beta6

• Test backend implementation
• Connect session store
• Protobuf parsing is now more robust
• Replace client every time ECONNREFUSED occurs
• Initial Riak Search support
• Minor bugfixing and internals refactoring

0.3.0beta5

• Add support for bucket listing in HTTP (new Riak 0.14 feature)
• Updated to CoffeeScript 1.0.0
• Fixed a memory leak caused by adding too many Event Listeners
• If there's no vclock present in a Meta, do not send any clientId

License

Simplified BSD License

Copyright 2011 Francisco Treacy. All rights reserved.

Redistribution and use in source and binary forms, with or without modification, are permitted provided that the following conditions are met:

1. Redistributions of source code must retain the above copyright notice, this list of conditions and the following disclaimer.

2. Redistributions in binary form must reproduce the above copyright notice, this list of conditions and the following disclaimer in the documentation and/or other materials provided with the distribution.

THIS SOFTWARE IS PROVIDED BY FRANCISCO TREACY ''AS IS'' AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL FRANCISCO TREACY OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.

The views and conclusions contained in the software and documentation are those of the authors and should not be interpreted as representing official policies, either expressed or implied, of Francisco Treacy.

riak-js's Issues

Problem with Riak server is being down

I dunno if it's critical, since riak is no point of failure DB, but, in case of when all instances are dead, I couldn't get any callback on command at all. I use this code:

var db = Riak.getClient(riakConfig);
db.save(type, key, data, options)(function(data, meta){
sys.puts(1);
});

and it never hits sys.puts(1); I see following in console output:

[riak-js] PUT /riak/xxx/ppp
{"message":"ECONNREFUSED, Connection refused","stack":"Error: ECONNREFUSED, Connection refused\n at IOWatcher.callback (net:854:22)\n at node.js:772:9","errno":111,"syscall":"connect"}
Error: ECONNREFUSED, Connection refused

That's it. Maybe I should catch errors like that somewhere else?

Store a new object and assign random key

According to Riaks documentation it is possible to get a random key if using POST against a bucket without specifying the key.

I simulate this in riak-js like this:

client = require('riak-js').getClient();
client.save('test', 0, 'this is a test');

which performs a POST, but how can I get the "Location" in order to get the random generated key?

url-unescape for links

removeLink({bucket: 'users', key: '[email protected]'}) does not work, cause the links are stored in a URL-escaped HTTP header

e.g. removeLink({bucket: 'users', key: 'foo%40bar.com'}) works fine

commit javascripts?

Including generated javascript adds a lot of noise when viewing/diffing commits.

Would it be smart to remove the lib folder altogether and add a compilation step when people install via npm? For those who clone the repo or download tarballs a simple coffee -wc --no-wrap -o lib src/*.coffee would do the trick.

npm 1.0 compatibility

Since npm 1.0 ignores files that are excluded by a .gitignore if no .npmignore is present, your lib dir is getting stripped out of the package.

There are a few ways to fix this:

1. Add a blank .npmignore file. If npm sees a .npmignore, it won't use the .gitignore. That's probably the simplest way to fix it.
2. Add "files" : ["lib"] to the package.json. (This is like explicitly un-ignoring lib.)
3. Add "scripts": { "preinstall" : "cake install" } and make coffee-script a regular dependency instead of a devDependency.

Also, I notice that you have some vows tests. It'd be good to add a "test" script in there as well, so that you'll be set to take advantage of the auto-testing coming in the next few months.

Add development dependencies to package.json (+ patch)

NPM has support for listing a package's development dependencies, so they can be taken care of with npm install --dev.

diff --git a/package.json b/package.json
index c6dda55..a484413 100644
--- a/package.json
+++ b/package.json
@@ -41,5 +41,9 @@
    },
    "os": [ "linux", "darwin" ],
    "dependencies": {},
+   "devDependencies": {
+       "vows": "*",
+       "coffee-script": "*"
+   },
    "engines": { "node": ">=0.1.97" }
-}
\ No newline at end of file
+}

The actual versions may need some editing for correctness.

Error: unrecognized option: --no-wrap running 'cake dev'

Coffee-script: 1.0.0
Node.js: 0.3.1
npm: 0.2.13
riak-js: 0.3.0beta4

$ cake dev

node.js:50
    throw e; // process.nextTick error, or 'error' event on first tick
    ^
Error: unrecognized option: --no-wrap
    at OptionParser.parse (/usr/local/lib/node/.npm/coffee-script/1.0.0/package/lib/optparse.js:34:17)
    at /usr/local/lib/node/.npm/coffee-script/1.0.0/package/lib/command.js:234:29
    at Object.run (/usr/local/lib/node/.npm/coffee-script/1.0.0/package/lib/command.js:25:5)
    at Object. (/usr/local/lib/node/.npm/coffee-script/1.0.0/package/bin/coffee:7:27)
    at Module._compile (node.js:329:23)
    at Object..js (node.js:337:12)
    at Module.load (node.js:260:25)
    at loadModule (node.js:232:12)
    at require (node.js:272:14)
    at Object. (/usr/local/bin/coffee:11:18)

benchmarking

Setup test environment to check speed and correctness.

calling save in get callback crash sometimes.

I am running an example code which is identical to the one on the webpage:
db.get('flights', 'KLM-5034', function(err, flight, meta) {
if (err) throw err
flight.status = 'delayed'
meta.links.push({ bucket: 'airlines', key: 'IBE', tag: 'operated_by' })
db.save('flights', 'KLM-5034', flight, meta)
})

But I got almost all the time the following error ( very seldom it works without error ):

/node_modules/riak-js/lib/utils.js:84
target[k] = d.value; ^
undefined

Any clues what is happening here?

thanks!.

merge keys branch

Changing timeout in mapred

How can I change the default timeout (60000) in maprep? According mapper's source, run() seems to be accepting [options,callback] but it doesn't seem to work at least with {timeout:999999} option. I tried to add timeout in bunch of different methods related to mapred that according source should accept options but can't get it to work :/

Allow extending Meta with user-defined behaviour

Hey,

My goal was to override encode and decode in order to allow special processing for specific buckets.
A real world example would be something like:

decode = (data) ->
  json = JSON.parse data
  json.id = @key
  json
encode = (data) ->
  delete data.id
  JSON.stringify data

Unfortunately I currently can not provide custom Meta objects currently as they are passed to the Meta constructor anyway. What I do at the moment looks like:

Meta = require 'riak-js/lib/meta'
Meta::encode = (data) ->
  super unless @bucket == 'foo'
  # do something

A nice way to do that would be:

class FooMeta extends Meta
  decode = (data) ->
    # ...

meta = new FooMeta()
db.save 'foo', {hello: 'world'}, meta, (err, res) ->
  # hi

What do you think? :)

improve tests

At the moment there is some testing but I feel it should be easier. Evaluate using vows or expresso (but needs to play nicely with coffeescript).

We should have amazing coverage just as Ripple.

On with the "airplane" theme, we could generate fake passengers with http://github.com/Marak/faker.js

Also make use of Sean's riak_kv_test_backend?
http://github.com/seancribbs/ripple/blob/ed9a65ee4c830236ec6c61808a2c1cba54dc7945/riak-client/erl_src/riak_kv_test_backend.erl

require fails with new npm

Using the older versions of npm to install the riak-js module works fine, but using the latest version doesn't seem to install the lib directory:

mjr@test1-01:$ npm install riak-js
[email protected] ./node_modules/riak-js
mjr@test1-01:$ node

require("riak-js");
Error: Cannot find module './lib/index'
at Function._resolveFilename (module.js:320:11)
at Function._load (module.js:266:25)
at require (module.js:348:19)
at Object. (/home/mjr/node_modules/riak-js/index.js:1:80)
at Module._compile (module.js:404:26)
at Object..js (module.js:410:10)
at Module.load (module.js:336:31)
at Function._load (module.js:297:12)
at require (module.js:348:19)
at [object Context]:1:1
mjr@test1-01:~$ ls node_modules/riak-js
Cakefile docs erl_src index.js LICENSE package.json README.md spec src

check if the connection to Riak is alive

coffee script

I've ported the Meta object to coffee script, using the master branch of coffee script. Jeremy just merged in a big patch, so the syntax is a bit different if you already know coffeescript. Also, once I get further in, I'll be using the coffeescript watch command to generate the JS files in /lib from /src.

I wanted to move the link encoding/decoding code to Meta and add some tests for that. The PBC client won't use it, but it won't hurt to have it there.

Any thoughts on the coffee script branch so far? http://github.com/technoweenie/riak-js/compare/coffee

agent.maxSockets in http node 0.4.X

Guys, do you think that this agent.maxSockets can make a queue instead of doing requests parallel in HTTP client of riak-js? Maybe make some setting to set it higher, because default value is just 5.

space in id

it seems that spaces in the id are not escaped:

db.save('airlines', 'ARG 1', { name: 'Aerolíneas Argentinas 1' , fleet: 1, european: false }
PUT /riak/airlines/ARG 1

db.save('airlines', 'ARG 2', { name: 'Aerolíneas Argentinas 2' , fleet: 2, european: false }
PUT /riak/airlines/ARG 2

db.count('airlines')
1

get not calling callback

I have a bucket called session with ~3000 keys like these:

"keys":["styjSNlsbwEIYCfgNINc9nku.CqdstcngbnJhTAns5M+m18Yau72bwzcc3EsGEI5vv10","G3RSuq4f1SFrVBK5i2eUyhOi.yOidtB6Vx7NQzIVDhpCwgoGrNPqWMbnykp473GhGdE","UJjx7wF28zSqqld36iQnLUFL.POmMSODsoBaJ5l7J9OhrnqvhtZoDgScgwyGtWzMNiE","HAqC3usgAMWBiGQkY21dttVW.rysityOevs+ENIlLuJqSZ6sF9sgZGUXBEOwba4M8rJw" ... ]

I have a function that clears all the keys in that bucket, it worked fine for other small buckets, with different (shorter and simple) key names:

clearAllSessions = function clearAllSessions(){
util.debug("[*] clearing all sessions from backend");

function cb(err, buffer, meta){
util.debug("[] clearing sessions");
if (err) throw err;
buffer.keys.forEach(function(key){
db.remove('session',key);
});
util.debug("[] cleared sessions");
}
db.get('session','?keys=true',cb);
//db.keys('session', cb);
}

The callback never gets called, and there is no error. I'm using http to talk to riak.

I've tried to make this call with keys=stream, but I get the error:
Error: Cannot convert response into application/json: Unexpected token { -- Response:
Is there an option like noJSON for get method?

May it be a problem in converting the response to JSON, or is the request being aborted?

cheers,
Daniel

[edit] I'm using node 0.4.2 and riak-js clone repo 0.3.6

404 statusCode

Hey, Frank, i faced a strange thing today.

I was doing something like:

client.head("bucket", "missingkey", {}, function(err, data, meta){
if( err ){
// it goes here if meta.statusCode == 404
}
});

it seems a bit wrong to me, because i didn't expect it as error, i was just checking for key existance and in my case it was a normal response. So to check if it's really 404 i gotta:
if( err ){
if( err.statusCode == undefined ){
// handle some error like riak is unreachable
}
else if( err.statusCode == 404 ){
// woot it's not found
}
else {
// something else
}
}

Shouldn't 404 be treated like normal response?

walk fails for non-JSON content

walk seems to work only if the returned objects are application/json. I'm using the data from the link walking tutorial, which is text/plain.

db.walk('people', 'sean', [["_", "friend"]])
POST /mapred
> { stack: [Getter/Setter],
  arguments: undefined,
  type: undefined,
  message: 'HTTP error 500: {"lineno":477,"message":"JSON.parse","source":"unknown"}',
  statusCode: 500,
  notFound: false }

My guess is that Riak.mapValuesJson barfs on the non-JSON content (text/plain). I'm surprised this POSTs to /mapred rather than GETting /riak/people/sean/_,friend,1

POST /mapred HTTP/1.1
Accept: multipart/mixed, application/json;q=0.7, */*;q=0.5
Host: localhost
Connection: close
Content-Type: application/json
Content-Length: 269

{"inputs":[["people","sean"]],"query":[{"link":{"bucket":"_","tag":"friend","keep":true,"language":"javascript"}},{"reduce":{"language":"erlang","module":"riak_kv_mapreduce","function":"reduce_set_union"}},{"map":{"name":"Riak.mapValuesJson","language":"javascript"}}]}

double error returned when riak endpoint is not up

output from airport-test.js when riak is not available:

macpro1:test siculars$ node airport-test.js
[riak-js] PUT /riak/test-airports/EZE
[riak-js] PUT /riak/test-airports/BCN
[riak-js] PUT /riak/test-airports/AMS
[riak-js] PUT /riak/test-airports/CDG
[riak-js] PUT /riak/test-airports/MUC
[riak-js] PUT /riak/test-airports/JFK
[riak-js] PUT /riak/test-airports/HKK
[riak-js] PUT /riak/test-airports/MEX
[riak-js] PUT /riak/test-flights/KLM-8098
[riak-js] PUT /riak/test-flights/AFR-394
[riak-js] PUT /riak/test-flights/CPA-112
[riak-js] PUT /riak/test-flights/IBE-5624
[riak-js] PUT /riak/test-flights/ARG-714
[riak-js] PUT /riak/test-flights/DLH-4001
[riak-js] PUT /riak/test-flights/AMX-1344
[riak-js] PUT /riak/test-flights/AMX-1346
[riak-js] PUT /riak/test-flights/KLM-1196
[riak-js] PUT /riak/test-flights/CPA-729
[riak-js] PUT /riak/test-flights/ARG-909
[riak-js] PUT /riak/test-flights/IBE-4418
[riak-js] PUT /riak/test-airlines/KLM
[riak-js] PUT /riak/test-airlines/AFR
[riak-js] PUT /riak/test-airlines/AMX
[riak-js] PUT /riak/test-airlines/ARG
[riak-js] PUT /riak/test-airlines/DLH
[riak-js] PUT /riak/test-airlines/IBE
[riak-js] PUT /riak/test-airlines/CPA
[riak-js] POST /mapred/
[riak-js] POST /mapred/
[riak-js] GET /riak/test-airlines/
[riak-js] GET /riak/test-airports/
[riak-js] GET /riak/test-flights/
[riak-js] ERROR: Couldn't reach localhost:8098
[riak-js] ERROR: Couldn't reach localhost:8098

pastie for readability: http://pastie.org/897596

Client.log should log to stderr

In general, stdout is where "informational" level output goes, like a w3c compatible log stream, for example. Debug logging should write to stderr.

Your code in Client.log currently just calls console.log. It should call console.error at minimum (ideally, it would be nice if I could pass you a stream for logging).

I can fork and send you a pull request if that's easier.

Saving an object fails the second time

Hi. I'm trying to update a object. Works the first time but not the second..

Create it..

> db.save("pers", "1", { title: "hans" })
PUT /riak/pers/1
false

> db.getAll("pers")
POST /mapred
false
> [ { meta: { bucket: 'pers', key: '1', vclock: 'a85hYGBgzGDKBVIsTO+1QzKYEhnzWBmuW5Qf48sCAA==' }, data: { title: 'hans' } } ]

Update it the first time, which works.

> db.save("pers", "1", { title: "peter" })
PUT /riak/pers/1
false

> db.getAll("pers")
POST /mapred
false
 > [ { meta: { bucket: 'pers', key: '1', vclock: 'a85hYGBgymDKBVIsTO+1QzKYEhnzWBmuW5Qf44MIszUnsezm+QOVeAKSyAIA' }, data: { title: 'peter' } } ]

but if I try to update it again

> db.save("pers", "1", { title: "andreas" })
PUT /riak/pers/1
false

it doesnt work:

> db.getAll("pers")
POST /mapred
false
> [ { meta: { bucket: 'pers', key: '1', vclock: 'a85hYGBgymDKBVIsTO+1QzKYEhnzWBmuW5Qf44MIszUnsezm+QOVeAKSyAIA' }, data: { title: 'peter' } } ]

I'm not sure if its a bug or if I just got something wrong.. Any ideas?

meaningful (bound to env) clientId

I really don't know what would be ideal.

But as a default there's probably something better than id-per-session (think i saw that in Basho's javascript client) or id-per-riak-client (what we currently have, riak-js).

What about grabbing the host name? We could end up with something like riak-js-Francisco-Treacys-MacBook-Pro.local

So I had this idea of pulling hostname while setting up the client, but it'd be more convenient to do that synchronously. Not sure.

client is useless after a connection error has occured

It doesn't seem to happen all the time though. Need to dig into it (check how other HTTP clients deal with this; would it make sense to replace the client?)

Also, emit 'error' event.

walk fails with object has no method 'link'

Hi Frank,

I am running node.js 0.2.4, riak 0.13.0, riak-js 0.3.0beta2 via npm. Following code taken from http://riakjs.org/:

db.save('airlines', 'KLM', {fleet: 111, country: 'NL'}, { links:
  [{ bucket: 'flights', key: 'KLM-8098', tag: 'cargo' },
   { bucket: 'flights', key: 'KLM-1196', tag: 'passenger' }]
});

db.walk('airlines', 'KLM', [["_", "flight"]]);

fails with:

TypeError: Object [object Object] has no method 'link'
    at [object Object].walk (/usr/local/lib/node/.npm/riak-js/0.3.0beta2/package/lib/http_client.js:112:15)
    at Object.<anonymous> (/home/factons/1.js:12:4)
    at Module._compile (node.js:462:23)
    at Module._loadScriptSync (node.js:469:10)
    at Module.loadSync (node.js:338:12)
    at Object.runMain (node.js:522:24)
    at Array.<anonymous> (node.js:756:12)
    at EventEmitter._tickCallback (node.js:55:22)
    at node.js:772:9

Thanks,
Ozgur

Key not returned in meta

The key generated by Riak during an POST request is not returned in the metadata.

keys=stream triggers error

db.get('test', undefined, {keys: "stream"})(sys.puts, function(e,m,r) { sys.puts('ERROR:' + r) })

ERROR:Couldn't parse it as JSON

This shouldn't happen (don't attempt to convert to JSON if it's streaming)

match up HTTP and PBC APIs

Make tests picking one client randomly out of multiple talking to different interfaces.

It'll reveal areas where the APIs need to be more even, or flaws in meta (which should be completely independent).

riak.get causes unrelated file-upload handling code to stall? (crazy, but important)

I know it sounds crazy and if one of my users reported this to me, I'd probably dismiss it as user error - and it may yet be a mistake I'm making and not realizing...

I have a test app (it's actually a hugely cut-down chunk of a much larger app) which lets users PUT a file onto a server. Earlier in the call stack, a riak.get is executed (in the full app, that is part of the authentication process). When the riak.get is avoided, the file upload works like a charm. When the riak.get is executed, the file upload stalls, but I have no clue why.

You can see the test program at https://gist.github.com/957127 (I didn't know if it would be couth to post a block that size in an issue...)

This issue is, unfortunately, blocking me only a day from a very important deadline, so I'd really appreciate whatever help you can offer.

emitter memory leak?

I'm getting this error on the latest node-proxy. Is this normal?

(node) warning: possible EventEmitter memory leak detected. 11 listeners added. Use emitter.setMaxListeners() to increase limit.
at Pool. (events.js:101:17)
at Object.proxyRequest (/usr/local/lib/node/.npm/http-proxy/0.3.1/package/lib/node-http-proxy.js:185:7)
at /opt/node/bb/lib/vhost.js:256:31
at /opt/node/bb/virt-dispatch.js:11:5
at Server. (/opt/node/bb/virt-dispatch.js:20:56)
at Server.emit (events.js:45:17)
at HTTPParser.onIncoming (http.js:862:12)
at HTTPParser.onHeadersComplete (http.js:85:31)
at Socket.ondata (http.js:787:22)
at Socket._onReadable (net.js:623:27)

Callback race conditions

It seems like the callbacks in riak-js return after the HTTP request has been sent, not when the response is received.

This means race conditions are created.

Check out this sample code gist for an example:

http://gist.github.com/519986

Authentication in getClient?

Is there a way to send authentication through the getClient() request? I know we can specify host and port, but what if we have http authentication running somewhere in the middle?

something like:

var db = require('riak-js').getClient({auth: 'key:secret', host: 'xx.xx.xx.xxx', port: '8080'});
or even
var db = require('riak-js').getClient({host: 'key:[email protected]', port: '8080'});

Riak-js + nginx

When using riak-js with nginx you can pop up with problem that there's no Host http request header is provided for http request, so nginx won't proxy it. Can you put a default Host header with value of host param given to getClient function?

getAll should provide a real Meta

At the moment, getAll provides a plain object with some properties, not a real Meta.

It should behave like a get does with an HTTP 300 Multiple Choices, so those Meta can be reused in subsequent operations.

Key filtering

Is there any support for that comming? I see a ranges branch in repo is it Key filtering?

riak search support

Get that in asap!

Using the 'Where' option

I may be using this wrong, but it worked before version 0.3.0beta6

db.getAll('users', { where: { email: req.body.user.email} }, function(err, users, meta) {});

This would return just the user objects with the specified email. Now it returns all users regardless of the email input.

riak-js-mapper

An ODM or whatever you wanna name it. Just like mongoose, visionmedia/rapid or Ripple.

I have already implemented something super basic with coffeescript classes, dunno how well it plays with js.

Anyway this functionality should belong to a project outside riak-js.

client pool

Ideally I want to build a wrapper around riak-js that will take care of:

• Holding several instances to make requests in parallel
• Automatically reconnecting to other nodes in the cluster when one goes down

The latter could also be achieved by using something like HAProxy but to start with we could support something super simple.

Should it belong to riak-js? Probably not.

script for protobuf/protobuf_for_node installation

Make it transparent to the user to install this dependency (which itself depends on protobuf). Hook it into an npm install script - or whatever they call it.

Possible memory leak

I tried importing fairly large amounts of data into riak using Node.js as a middle-man, my program continuously uses more and more memory. Cutting away the cruft, I could cut it down to this: https://gist.github.com/756279 (the setTimeout should probably be dropped in preference for just launching a new round of save()'s in the callback, when in_flight==0).

When executing the program on my machine, i get a memory usage like so (X-axis is seconds.):

Before it dies of excessive memory use.

I'm using the latest beta (4, I believe) + Node.js HEAD.

Database constructor data returned in meta on get()

When retrieving data using get(), the dictionary passed to getClient() is included in the returned meta-data.

var riak = require('riak-js'), sys = require('sys');

db = riak.getClient( { debug: false, someKey: 'someValue' });

db.save('foo', 'bar', {}, function(err, data) {
    db.get('foo', 'bar', function(err, data, meta) {
        console.log(sys.inspect(meta, false, 99));
        db.remove('foo', 'bar');
    });
});

Returns:

{ usermeta: {},
  debug: false,
  api: 'http',
  clientId: 'riak-js',
  binary: false,
  raw: 'riak',
  someKey: 'someValue', // <-- Should this be here?
  contentEncoding: 'utf8',
  links: [],
  host: 'localhost',
  accept: 'multipart/mixed, application/json;q=0.7, */*;q=0.5',
  responseEncoding: 'utf8',
  bucket: 'foo',
  key: 'bar',
  contentType: 'application/json',
  vclock: 'a85hYGBgzGDKBVIsTOunKmUwJTLmsTJMZX1yjC8LAA==',
  lastMod: 'Thu, 03 Mar 2011 12:29:09 GMT',
  etag: '"4RQh4xZBXMpfQQQvhVTO8K"',
  contentRange: undefined,
  acceptRanges: undefined,
  statusCode: 200 }

I don't know if this is by design, but I stumbled across it when I accidentally gave the constructor some unrelated data, that suddenly caused riak-js not to accept my modified header data. I have unfortunately not been able to reproduce the error reliably...

I have semi-randomly tried this on a few of the beta's, as well on @latests, all with the same result.

Problem running riak-js on npm 1.0rc / node 0.4.3

Installed forever 0.4.1 (which depends on winston, which depends on riak-js) using npm 1.0rc and on first run it gives such error:

% forever -v

node.js:134
throw e; // process.nextTick error, or 'error' event on first tick
^
Error: Cannot find module './lib/index'
at Function._resolveFilename (module.js:320:11)
at Function._load (module.js:266:25)
at require (module.js:348:19)
at Object. (/Users/jakub/node_modules/forever/node_modules/winston/node_modules/riak-js/index.js:1:80)
at Module._compile (module.js:404:26)
at Object..js (module.js:410:10)
at Module.load (module.js:336:31)
at Function._load (module.js:297:12)
at require (module.js:348:19)
at Object. (/Users/jakub/node_modules/forever/node_modules/winston/lib/winston/transports/riak.js:10:14)

Is it a bug in riak-js?

jQuery Version, Error with Headers

I'm getting an error in riak.js

headers is undefined
[line: 269] this.type = headers['content-type'];

Dug into the riak-jquery.js file and the helper method inside of .executeInternal

// helper
function getMeta(xhr) {
var meta = new Meta(), headers = xhr.getAllResponseHeaders();
meta.headers = {};
meta.statusCode = xhr.status;
if (headers.length) {
headers.split('\r\n').forEach(function(header) {
// header of type: Content-Length: 146
var i = header.indexOf(':');
meta.headers[header.substring(0, i).toLowerCase()] = header.substring(i+2);
});
}
var location = meta.headers['location'];
meta.key = location ? location.substring(location.lastIndexOf('/')+1) : options.key;
return meta;
}

the

var meta = new Meta() need parameters passed to it otherwise it throws and error and the javascript dies.

Insert without key

The Riak protocol supports the auto-generation of keys on insert if one isn't provided (simply posting to /riak/bucketname). I could be wrong but after looking at the docs/source, I found no way to do this in riak-js. Seems like it would be pretty trivial to add support for this. If I get any time to learn coffeescript I could take a look at implementing it I suppose.

ECONNREFUSED, Connection refused

I can't get riak-js to connect to Riak. I'm running a cluster and trying to connect to 192.168.1.103 but it seems that riak-js is always trying to connect somewhere else. I can use Riak's webinterface so I know the instance is up. What's the correct way to connect different IPs? I tried many different variations of getClient({host:'192.168.1.103'}) but it just doesn't seem to work.

Stream files through Luwak.

If I'm not mistaken, a major benefit of Luwak is dealing with very large files. Rather than having to hold these large files in memory, why not allow for the data to be streamed through the API. This would be pretty killer IMO.

Same api as normal, the library would just have to handle the stream internally. This can be done using util.pump();

  var readstream = fs.createReadStream("/path/to/large/file");
  db.saveLarge(<hash>, stream, function(err) {
      console.log('stream finished.');
  });

streaming API

Everything is buffered now.

Streaming data back to the client would be ideal for:

• big docs (i.e. video)
• Map/Reduce jobs
• etc