zendesk / radar

High level API and backend for writing web apps that use push messaging

radar

The real-time service layer for your web application

Documentation

See radar.zendesk.com for detailed documentation.

This is the project necessary for running a radar server. Documentation about building an app and using the client-side libraries is available at radar.zendesk.com. Browse radar client libraries and tools.

Installation

Requires: redis 2.8+, node.js 4+

###Installing from npm:

$ npm install radar

Programmatic usage

radar can be extended programmatically with custom code and middleware:

var http = require('http')
var { Radar } = require('radar')

var httpServer = http.createServer(function(req, res) {
  res.end('Nothing here.');
});

// Radar server
var radar = new Radar();
radar.attach(httpServer, { redis_host: 'localhost', redis_port: 6379 });

httpServer.listen(8000);

See also the sample directory in the radar repository.

Out-of-the-box usage

$ git clone [email protected]:zendesk/radar.git
$ cd radar
$ npm install
$ npm start

See radar.zendesk.com/server for additional usage and configuration documentation

Contributing

Running tests

$ npm test

By default, tests are run only against redis sentinel.

If you want to run against redis directly: $ npm run test-redis For direct redis and redis sentinel: $ npm run test-full

Workflow

• Fork http://github.com/zendesk/radar, clone, make changes (including a Changelog update), commit, push, PR

Copyright and License

Copyright 2016, Zendesk Inc. Licensed under the Apache License Version 2.0, http://www.apache.org/licenses/LICENSE-2.0

radar's Issues

Looks like tutorial code/docs are outdated again.

I'm going through the tutorial and a few things wouldn't work as described. First the Makefile, One of the lines in the make file is this @cat ./node_modules/radar_client/node_modules/engine.io-client/dist/engine.io.js >> public/radar_client.js But it throughs an error saying the directory could not be found. sure enough, there is no engine.io-client in the radar_client/dist/ folder. I changed the line to use the engine.io module that I had to download. this got me through the Makefile but when I try to run the Radarclient.alloc() function it never connects to the server. it returns a 404 error at the following url path /engine.io/?EIO=3&transport=polling&t=N1LIQMB and then returns this error

radar_client socket closed null transport error Error: xhr poll error
    at XHR.Transport.onError (radar_client.js:1751)
    at Request.<anonymous> (radar_client.js:1140)
    at Request.Emitter.emit (radar_client.js:3164)
    at Request.onError (radar_client.js:1322)
    at radar_client.js:1269

Im not sure how to fix this one.

Change certain APIs to be async

This is a tracking issue.
Right now we have some synchronous methods which do quite a bit of iteration (sometimes on multiple levels of references) synchronously, which is silly and unsafe.

• PresenceManager#getOnline
• PresenceManager#getClientsOnline
• While we're at it, address the #fixme on PresenceManager#addClient (use options object for named params)

Some places could be cleaned up with Promises:

• PresenceManager#fullRead

client_offline is sent multiple times on the instance from which a client disconnects

userData does not persist

When presence is updated the userData is lost.

Does Radar has client support for Android or iOS

I know there are android and ios libraries for socket.io. but since Radar is build on top of engine.io are there such libraries for Radar.
Thanks

Do you have an estimate of number of concurrent connections per RADAR server?

FYI: we did some experiment with socket.io servers and maxed 26000 concurrent connections, from which point we started getting disconnection from some clients, so I would say we could support 10,000 solid concurrent connections per socket.io servers on descent machines.

Do you have any benchmark on concurrent connections per RADAR servers?
Thanks

How to manage users?

Thanks for this great initiative. The API looks promising, it is clean and easy to use. The documentation is already quite extensive, but I miss the part describing how to manage accounts: how to add users, delete users, list users, etc. Am I overlooking something?

Change default expiry for presence per channel?

I have a couple of levels of presence for the project I'm working on.

1. Users online right now for the whole project
2. Users looking at a specific page

I'd like to change the default 15 second timeout for when disconnect events are sent to connected users on a per-channel basis.

For projects, I'd like the event to fire after 15 seconds of non-presence. For page presence, I'd like it to fire after 5 seconds.

I thought this was configurable via my resource type definition using the maxAgeSeconds policy:

  {
    name: 'projectPresence',
    expression: new RegExp('^presence:/rollbar/project/[0-9]+$'),
    type: 'Presence',
    policy: { cache: true, maxAgeSeconds: 15 },
    authProvider: {
      authorize: security.resourceTokenValidator
    }
  },
  {
    name: 'projectItemPresence',
    expression: new RegExp('^presence:/rollbar/project/[0-9]+/item/[0-9]+$'),
    type: 'Presence',
    policy: { cache: false, maxAgeSeconds: 5 },
    authProvider: {
      authorize: security.resourceTokenValidator
    }
  }

However, it looks like this 15 second number is hard-coded here:
https://github.com/zendesk/radar/blob/master/core/lib/resources/presence/presence_manager.js#L16

Latest engine.io?

Any chance of using latest engine.io?

AWS Cost and Spend Management

Can't access the site http://radar.zendesk.com/

Add ability to store arbitrary user data on presences.

this.store is undefined

I'm having a random error on radar/core/lib/resources/presence/presence_manager.js:249 calling this.store.userExists(userId).

Few times this.store.userExists(userId) is undefined, the complete backtrace is this:

TypeError: Cannot read property 'userExists' of undefined
1
at PresenceManager.setupExpiry (/home/deploy/www/moxrts/node_modules/radar/core/lib/resources/presence/presence_manager.js line 249 col 16)
if(this.store.userExists(userId)) {
2
at PresenceManager.handleOffline (/home/deploy/www/moxrts/node_modules/radar/core/lib/resources/presence/presence_manager.js line 228 col 10)
this.setupExpiry(userId, userType);
3
at PresenceManager.processRedisEntry (/home/deploy/www/moxrts/node_modules/radar/core/lib/resources/presence/presence_manager.js line 209 col 10)
this.handleOffline(clientId, userId, userType, message.explicit);
4
at Resource.Presence.redisIn (/home/deploy/www/moxrts/node_modules/radar/core/lib/resources/presence/index.js line 88 col 16)
this.manager.processRedisEntry(message);
5
at Server.handleMessage (/home/deploy/www/moxrts/node_modules/radar/server/server.js line 102 col 25)
this.channels[name].redisIn(data);
6
at emitTwo (events.js line 111 col 20)
7
at RedisClient.emit (events.js line 191 col 7)
8
at RedisClient.return_reply (/home/deploy/www/moxrts/node_modules/radar/node_modules/persistence/node_modules/redis/index.js line 694 col 22)
this.emit("message", reply[1].toString(), reply[2]); // channel, message
9
at ReplyParser.<anonymous> (/home/deploy/www/moxrts/node_modules/radar/node_modules/persistence/node_modules/redis/index.js line 321 col 14)
self.return_reply(reply);
10
at emitOne (events.js line 96 col 13)
11
at ReplyParser.emit (events.js line 188 col 7)

radar version: 0.10.1

Any suggestion about what can be causing this? Thanks

Document the rate-limiting and auth middlewares

There is nothing about how to use these middlewares or write some.

Some newby questioms

I've been reading through the Radar doc's and it all looks quite impressive, however I'm not sure if it fits my use case.

I am currently using Now.js which lets you do RPC between Browser and Server and vice-versa. I suspect it is the source of a specific problem I randomly see and it is no longer being developed, so I am looking at a replacement for it.

So far Autobahn.js looks pretty good, but now I've found Radar and want to throw it into the mix.

Q1) Do you think Radar could be used to replace an RPC package.
Q2) Can Radar be used without Redis. At present Redis isn't available for Windows and that is one of my targets.

Thanks,
Neville

Authorize message list sync/subscribe

Hi!

When the user publishes a message, I can send an object with the message that contains my authorization hash. Then, in the auth funcion defined in the policy I can verify that hash and return true to validate the request.

But, how can I pass the hash in a sync/subscribe request? The auth function is called but with no parameters.

Thanks!

Redis Auth

Seems to be no easy way to authenticate to the redis server... Am I missing something or am I just expected to edit the package?

Radar does not unsubscribe from resources in redis

Radar server keeps subscribing to new resources it is interested in. However, it never tracks when a resource is no longer needed and never unsubscribe from it in redis. This will cause performance issues after several days after seeing several resources later.

Custom server cannot access "Api".attach()

I followed the server doc on http://radar.zendesk.com/server.html but when it comes to Api.attach() fatal error occurred. Referencing index.js and server.js in this package, they are actually pointing to different things.

Problems with vanilla Radar

I have a super simple Radar server which is returning the following error (HTTP400) within the client:

XMLHttpRequest cannot load https://domain.com/engine.io/?EIO=3&transport=polling&t=1446831121358-1&sid=2Xk4XVFAB6lhh8haAAAz. Response for preflight has invalid HTTP status code 400

my server.js looks like this:

var url = require('url'),
    http = require('http'),
    Radar = require('radar').server,
    Api = require('radar').api;

var redis_url = process.env.REDIS_URL;
parsed_url = url.parse(redis_url)

var config = {
  redisUrl: redis_url,
  redis_host: parsed_url.host,
  redis_port: parsed_url.port,
  port: process.env.PORT,
  persistence: true,
}

var configuration = require('radar').configurator.load(config)

var server = http.createServer(function(req, res) {
  console.log('404', req.url);
  res.statusCode = 404;
  res.end();
});

Api.attach(server);

var radar = new Radar();
radar.attach(server, configuration);
server.listen(config.port);
console.log('Server is listening on port ' + config.port);

There's nothing here I would consider odd.

What's more I only see the HTTP 400 in the logs, with no exception. Is there any way to get Radar to log exceptions here? (nb. This is running on Heroku)

Clean up directory structure

• move server.js to /bin directory
• organize by layer / component
• clean up tests directory to separate various test types (unit, integration, functional)

Check in the node_modules folder

Support authenicated Redis

Not everyone leaves their redis instances open to the world. Add support for basic auth redis instances, such as those supplied by heroku-redis.

performance problem with large message data

I made a mini test and it seems radar is getting slow with data size. The two test cases.

1. Brute-force test: I upload the data from client to a web server (no radar here). The server responses then I send a radar message to the other client "available, donwload it". The client download it and send a radar message back: "done".

2. Radar test: I send a radar message with the data. And the other client responses "done".

Both tests run on my local machine, no internet connection used. I measured the total time.

Results:
very small data size: 1) 176 ms 2) 25ms
1M data size: 1) 176 ms 2) 1000ms

Why is it so slow? Is it normal?

`user_offline` messages are not being broadcast

@vanchi-zendesk @samshull I noticed in some instances that the user_offline message was not being broadcast. I think I've traced it to this code: https://github.com/zendesk/radar/blob/master/core/lib/resources/presence/index.js#L103-L104

It should be the following I believe:

if (Object.keys(this.subscribers).length == 0) {

engine.io version used would send out glibish on first handshake

Since the first request to the server would be a message to trying to upgrade from long-polling to websocket, however, that request simply returned as 400 bad request with the following output:

HTTP/1.1 400 Bad Request
Content-Type: application/json
Access-Control-Allow-Credentials: true
Access-Control-Allow-Origin: http://localhost:8000
Date: Thu, 23 Nov 2017 08:59:24 GMT
Connection: keep-alive

and the content given would have some extra bits:

00000000: 0009 07ff 307b 2273 6964 223a 2259 4835  ....0{"sid":"YH5
00000010: 6351 7653 4251 5139 4535 6164 3941 4141  cQvSBQQ9E5ad9AAA
00000020: 4322 2c22 7570 6772 6164 6573 223a 5b22  C","upgrades":["
00000030: 7765 6273 6f63 6b65 7422 5d2c 2270 696e  websocket"],"pin
00000040: 6749 6e74 6572 7661 6c22 3a32 3530 3030  gInterval":25000
00000050: 2c22 7069 6e67 5469 6d65 6f75 7422 3a36  ,"pingTimeout":6
00000060: 3030 3030 7d                             0000}

after upgading to newest engine.io verseion (3.1.4 at the time of this report), such glibish at the start of the response is no-longer there:

00000000: 3937 3a30 7b22 7369 6422 3a22 6649 4e59  97:0{"sid":"fINY
00000010: 4b4b 6c4f 6c65 746e 6a6b 697a 4141 4148  KKlOletnjkizAAAH
00000020: 222c 2275 7067 7261 6465 7322 3a5b 2277  ","upgrades":["w
00000030: 6562 736f 636b 6574 225d 2c22 7069 6e67  ebsocket"],"ping
00000040: 496e 7465 7276 616c 223a 3235 3030 302c  Interval":25000,
00000050: 2270 696e 6754 696d 656f 7574 223a 3630  "pingTimeout":60
00000060: 3030 307d                                000}

Haven't got to time to see if newest version of engine.io would break any existing APIs, but so-far there doesn't seem to have any problems.

How to turn on logging?

I've created my own server.js and I'm trying to debug it. I can't get the Radar logging to show in the console unless I run the radar/server.js directly.

I've tried including minilog and enabling it but the radar logs still do not appear.

Does Radar Presence support same user multi device/connections?

Please correct me if I'm wrong. The radar user presence is a toggle mode it either indicates online or offline. and in the docs you mentioned that it is changed either manually by calling the set function or by tcp disconnect/connection. So, if the user has multiple connections i.e. desktop-connection and another lets say mobile, In this case I would expect the offline would happen if both / all connections are lost.
Can you elaborate on this?
Thanks

Having trouble to get radar started

Hi guys,

I am trying to get radar running on my local dev.

I got this message after I typed command 'npm start'

Please help.

Cheers,

Daniel

Storing an object on a key will have [object Object] in the redis hash.

Notification works correctly because it sends JSON.stringify(message). However, if you do a .sync/.get on it, we will notice the issue.

Radar errors should identify the message that generated the error.

Radar server errors are sent to the client without any indication of where the error came from. For auth_invalid errors this lack of know the source of the issue prevents us from being able to automatically retry sending with a new and valid auth message.

We should include the original message in the error response so that the client and client-side libraries can decide if they would like to retry sending the message after an error.

Upgrade engine.io

The current dependency is to [email protected], we need to upgrade that to 0.7.9

Deprecate / remove API

The erstwhile radar api should be considered deprecated and will not be included in the 1.0.0 release.

Expose current configuration and the current socket's id via a managed API

//cc @vanchi-zendesk

How do I set CORS headers with Radar?

I'm running a radar server and a separate python server on different ports. The python server serves from http://localhost:7000/ and the radar server from http://localhost:8000.

I'm running into an issue while requesting pre-flight data for engine.io:

http://localhost:8000/?EIO=2&transport=polling&t=1408583423148-1

XMLHttpRequest cannot load http://localhost:8000/?EIO=2&transport=polling&t=1408583427160-2. No 'Access-Control-Allow-Origin' header is present on the requested resource. Origin 'http://localhost:7000' is therefore not allowed access. 

I don't see any docs on how to enable CORS for engine.io or radar. I'm using the vanilla radar setup, (from http://radar.zendesk.com/index.html):

var server = http.createServer(function(req, res) {
  console.log('404', req.url);
  res.statusCode = 404;
  res.end();
});

// attach Radar server to the http server
var radar = new Radar();

radar.attach(server, {
  redis_host: config.redis.host,
  redis_port: config.redis.port
});

server.listen(port);

I've tried adding headers in the createServer callback but I just get an infinite loop of requests for the EIO query I have above.

How do I set Access-Control-Allow-Origin headers with Radar?

Unified configuration

• clean up having ./configuration.js and ./configurator.js
• support all configuration as env vars
• validate configuration at startup
• where possible, delete code / farm out to other packages

Allow Radar to use a different redis database

Clean / define public programmatic interface

pseudocode:

// simple case
const Radar = require('radar')
const configuration = require('./config.json')

const server = new Radar(configuration)
server.ready.then(_ => console.log('radar started'))

// adding radar to an existing HttpServer instance
// this might have unexpected behavior due to middleware ordering, but is useful for things like testing
const Radar = require('radar')
const configuration = require('./config.json')
const http = require('http')

const server = http.createServer()
const radar = new Radar(server, configuration)

Remove all dependencies on 3rd party git repos

Tutorial code is bad

The following code from the tutorial:

var fs = require('fs'),
  url = require('url'),
  http = require('http'),
  Radar = require('radar').server;

var server = http.createServer(function(req, res) {
  console.log('404', req.url);
  res.statusCode = 404;
  res.end();
});

var radar = new Radar();

var configuration = {
  redis_host: 'localhost',
  redis_port: 6379,
  port: 5000
}
radar.attach(server, configuration);
server.listen(configuration.port);
console.log('Server is listening on port ' + configuration.port);

fails on startup:

throw new Error('No configuration provided'); on the radar.attach.

Using node 4.2.0 and radar 0.16.5

Accessing api.attach

Am I missing something or is the index.js returning the wrong module?
Api.attach(httpServer);
^
TypeError: Object # has no method 'attach

Replace Function#bind with lodash bind

v8 in particular has severe performance issues with bind. Eliminating it is a quick win.

Ultimately when we drop node 0.10.x support and switch to arrow functions, we can eliminate the need for bind in most situations.

radar.zendesk.com does not use HTTPS

Github Pages doesn't support HTTPS on custom domains. 😞

As a workaround could we remove the custom domain and setup a redirect from radar.zendesk.com to the github page url?

Question about clustering

Hello,

I am considering using Radar in a node server that uses clustering. Is there some configuration in Radar that makes it easier, or do I need to handle clustering myself?

N.

Adding extra data to presence notifications?

I'd like to add some data to each notification that is sent for presence updates.

e.g. I want to include the user's email address along with their user_id

I could just have the user_id be a string with each piece of data in it,

e.g.

RadarClient.configure({
    userId: String(userId) + '|' + email
});

but I'm hoping there is a more elegant way.

Ghost socket is kept open in one of our test

see PR #35