firehoseio / firehose Goto Github PK

If the difference between the clients last sequence and the current max sequence is greater than the items available nill messages will be returned.

This can occur when the buffer size is different from the default buffer size and is caused by indexing into the message list on a non-existent index. Example:

Real buffer size = 1
Client last_sequence = 5
Current sequence = 100
Calculated diff = 95
https://github.com/firehoseio/firehose/blob/master/lib/firehose/server/channel.rb#L41
message_list.length == 1
message_list[95] == nil
https://github.com/firehoseio/firehose/blob/master/lib/firehose/server/channel.rb#L51

I propose that after receiving the list from Redis we check the minimum available sequence number. Example:
Minimum available sequence = 100

if last_sequence < message_list[0].sequence
  last_sequence = message_list[0].sequence - 1

Hi,

I want to use the consumer as described in the README.

My consumer.ru file:

~/code/firehose_c% cat consumer.ru 
require 'firehose'
consumer = Firehose::Rack::Consumer.new do |app|
  app.http_long_poll.timeout = 20
end

run consumer

In my Gemfile I have locked rainbows to 4.4.3 and using the git checkout of firehose.

I start up rainbows now by calling:

rainbows consumer.ru -p 7474

Anyway if I try to connect via the Javascript library I am just getting this error message in the rainbows log:

127.0.0.1 - - [30/Oct/2013 07:19:46] "HEAD /LrMzsvbdNgA3RmrsjFuG HTTP/1.1" 500 70112 0.0232
Rack::Lint::LintError: rack.hijack must respond to call
    /Users/nirnanaaa/.rvm/gems/ruby-2.0.0-p247/gems/rack-1.5.2/lib/rack/lint.rb:20:in `assert'
    /Users/nirnanaaa/.rvm/gems/ruby-2.0.0-p247/gems/rack-1.5.2/lib/rack/lint.rb:459:in `check_hijack'
    /Users/nirnanaaa/.rvm/gems/ruby-2.0.0-p247/gems/rack-1.5.2/lib/rack/lint.rb:238:in `check_env'
    /Users/nirnanaaa/.rvm/gems/ruby-2.0.0-p247/gems/rack-1.5.2/lib/rack/lint.rb:43:in `_call'
    /Users/nirnanaaa/.rvm/gems/ruby-2.0.0-p247/gems/rack-1.5.2/lib/rack/lint.rb:37:in `call'
    /Users/nirnanaaa/.rvm/gems/ruby-2.0.0-p247/gems/rack-1.5.2/lib/rack/showexceptions.rb:24:in `call'
    /Users/nirnanaaa/.rvm/gems/ruby-2.0.0-p247/gems/rack-1.5.2/lib/rack/commonlogger.rb:33:in `call'
    /Users/nirnanaaa/.rvm/gems/ruby-2.0.0-p247/gems/rack-1.5.2/lib/rack/chunked.rb:43:in `call'
    /Users/nirnanaaa/.rvm/gems/ruby-2.0.0-p247/gems/rack-1.5.2/lib/rack/content_length.rb:14:in `call'
    /Users/nirnanaaa/.rvm/gems/ruby-2.0.0-p247/gems/rainbows-4.4.3/lib/rainbows/max_body.rb:66:in `block in call'
    /Users/nirnanaaa/.rvm/gems/ruby-2.0.0-p247/gems/rainbows-4.4.3/lib/rainbows/max_body.rb:59:in `catch'
    /Users/nirnanaaa/.rvm/gems/ruby-2.0.0-p247/gems/rainbows-4.4.3/lib/rainbows/max_body.rb:59:in `call'
    /Users/nirnanaaa/.rvm/gems/ruby-2.0.0-p247/gems/rainbows-4.4.3/lib/rainbows/process_client.rb:43:in `process_loop'
    /Users/nirnanaaa/.rvm/gems/ruby-2.0.0-p247/gems/rainbows-4.4.3/lib/rainbows/base.rb:32:in `process_client'
    /Users/nirnanaaa/.rvm/gems/ruby-2.0.0-p247/gems/unicorn-4.6.3/lib/unicorn/http_server.rb:632:in `worker_loop'
    /Users/nirnanaaa/.rvm/gems/ruby-2.0.0-p247/gems/rainbows-4.4.3/lib/rainbows/http_server.rb:45:in `worker_loop'
    /Users/nirnanaaa/.rvm/gems/ruby-2.0.0-p247/gems/unicorn-4.6.3/lib/unicorn/http_server.rb:500:in `spawn_missing_workers'
    /Users/nirnanaaa/.rvm/gems/ruby-2.0.0-p247/gems/rainbows-4.4.3/lib/rainbows/http_server.rb:61:in `spawn_missing_workers'
    /Users/nirnanaaa/.rvm/gems/ruby-2.0.0-p247/gems/unicorn-4.6.3/lib/unicorn/http_server.rb:142:in `start'
    /Users/nirnanaaa/.rvm/gems/ruby-2.0.0-p247/gems/rainbows-4.4.3/bin/rainbows:122:in `<top (required)>'
    /Users/nirnanaaa/.rvm/gems/ruby-2.0.0-p247/bin/rainbows:23:in `load'
    /Users/nirnanaaa/.rvm/gems/ruby-2.0.0-p247/bin/rainbows:23:in `<main>'
    /Users/nirnanaaa/.rvm/gems/ruby-2.0.0-p247/bin/ruby_noexec_wrapper:14:in `eval'
    /Users/nirnanaaa/.rvm/gems/ruby-2.0.0-p247/bin/ruby_noexec_wrapper:14:in `<main>'

And the browsers console just throws this:

XMLHttpRequest cannot load http://localhost:7474/LrMzsvbdNgA3RmrsjFuG. No 'Access-Control-Allow-Origin' header is present on the requested resource. Origin 'http://127.0.0.1:3000' is therefore not allowed access. 

If I start the consumer in thin I get this in my thin log:

[2013-10-30 06:31:39.590 #3869] DEBUG : WS connection `/LrMzsvbdNgA3RmrsjFuG` closing. Code: 1006; Reason ""

And an Upgrade header missing in my Browsers console.

Redis version:

Redis server v=2.6.16 sha=00000000:0 malloc=libc bits=64

Ruby:

~/code/firehose_c% ruby -v
ruby 1.9.3p392 (2013-02-22 revision 39386) [x86_64-darwin12.5.0]

Have I done something wrong or am I missing something?

Does Firehose currently support private channels?
What is the best way to restrict access so that anonymous users cannot read private messages?
Thank you!

Hi,

Imagine I have 3 Google Chrome/Firefox browser tab opened at once, every on the same page using Firehose. When I want to open fourth tab (on the same page) - it hangs and loading time is like 20 seconds. If I close third tab before fourth is fully loaded - it loads instantly. So connections block themselves somehow. I tried to run Rainbows! on 4 processes, but it doesn't help.

In my network tab I see websocketschannels@firehose, xhr, which gets 204 status after ~20s - I figured that's OK. Is is possible that browser blocks too many connections?

Stack: Firehose 1.4.0, Rainbows! 5.1.1 [no errors in logs],
Apache 2.4.25 [no errors in apache logs] config:

ProxyPassMatch /websockets http://localhost:7474 Keepalive=On acquire=3000 timeout=600
ProxyPassReverse /websockets http://localhost:7474

How I use Firehose:

new Firehose.MultiplexedConsumer(
     uri: websocket_host_no_protocol
     ssl: true
     channels:
       "some-channel-path":
         message: (msg) ->
           someMethod(msg)
   ).connect()

Rainbows logs:

[2017-10-03 15:18:17.989 #6669] DEBUG : HTTP multiplexing POST, subscribing [{:channel=>"/websockets/entry-models-2877/new-sensor-data", :last_message_sequence=>1}, {:channel=>"/websockets/activities-1/new-important-activity", :last_message_sequence=>0}]
[2017-10-03 15:18:17.990 #6669] DEBUG : No messages in buffer, subscribing. sequence: `0` consumer_sequence: 1
[2017-10-03 15:18:17.990 #6669] DEBUG : No messages in buffer, subscribing. sequence: `0` consumer_sequence: 0
[2017-10-03 15:18:18.370 #6669] DEBUG : HTTP published "{\"entry_model_id\":2935}" to "/websockets/entry-models-2935/new-sensor-data" with ttl "2"
[2017-10-03 15:18:18.371 #6669] DEBUG : Redis stored/published `{"entry_model_id":2935}` to list `firehose:/websockets/entry-models-2935/new-sensor-data:list` with sequence `1`
[2017-10-03 15:18:18.871 #6755] DEBUG : HTTP multiplexing POST, subscribing [{:channel=>"/websockets/user-23/update-notification-icon", :last_message_sequence=>0}, {:channel=>"/websockets/hierarchy-tree-23/refresh-node", :last_message_sequence=>0}, {:channel=>"/websockets/hierarchy-tree-23/remove-node", :last_message_sequence=>0}, {:channel=>"/websockets/entry-model-undefined/update-progress-table", :last_message_sequence=>0}, {:channel=>"/websockets/entry-model-undefined/refresh-entry-model", :last_message_sequence=>0}, {:channel=>"/websockets/entry-model-undefined/refresh-trans-table", :last_message_sequence=>0}, {:channel=>"/websockets/entry-model-undefined/refresh-trans-csv", :last_message_sequence=>0}, {:channel=>"/websockets/entry-model-undefined/refresh-transformation", :last_message_sequence=>0}]
[2017-10-03 15:18:18.872 #6755] DEBUG : No messages in buffer, subscribing. sequence: `0` consumer_sequence: 0
[2017-10-03 15:18:18.872 #6755] DEBUG : No messages in buffer, subscribing. sequence: `0` consumer_sequence: 0
[2017-10-03 15:18:18.872 #6755] DEBUG : No messages in buffer, subscribing. sequence: `0` consumer_sequence: 0
[2017-10-03 15:18:18.872 #6755] DEBUG : No messages in buffer, subscribing. sequence: `0` consumer_sequence: 0
[2017-10-03 15:18:18.872 #6755] DEBUG : No messages in buffer, subscribing. sequence: `0` consumer_sequence: 0
[2017-10-03 15:18:18.872 #6755] DEBUG : No messages in buffer, subscribing. sequence: `0` consumer_sequence: 0
[2017-10-03 15:18:18.872 #6755] DEBUG : No messages in buffer, subscribing. sequence: `0` consumer_sequence: 0
[2017-10-03 15:18:18.872 #6755] DEBUG : No messages in buffer, subscribing. sequence: `0` consumer_sequence: 0
[2017-10-03 15:18:19.621 #6755] DEBUG : Clearing old metrics

I have no idea how to deal with this. I would be grateful for any help!

Edit: Propably problem with apache configuration, ws request should get 101 status instead of 200.

firehose.io has the text "Firehose embraces RESTful design prinicpals"

It should be principles, not principals, unless this framework has a feature that hugs elementary school leaders.

The ws client should work without an explicitly defined connect and disconnect callback.

I can verify the same specs on Travis CI are failing locally. https://travis-ci.org/polleverywhere/firehose/jobs/5085390

At https://github.com/polleverywhere/firehose/blob/master/lib/firehose/assets.rb#L1 we require sprockets. But it is included only as a development dependency, which breaks deployment when running Bundler without development or test groups.

Should sprockets be a regular gem dependency? I'll submit a pull request for this trivial change if this is indeed the case.

I'm seeing some strange behavior with the Javascript client in particular. The network tab shows that some of the requests are taking around 10-15s or so to connect if it ever does (over ws://, with underscore and message parameters); where the request for the resource in question via curl (http://) completes immediately. I feel like I've got to be missing something.

For context -- I'm trying to use Firehose.io to hydrate some Backbone.js models. When a new model is created, it connects out to a firehose streaming resource; similarly for collections.

I'm expecting that it should get the latest copy of the resource immediately, as the curl client seems to. It's definitely not that all of the requests are like this, either -- but regularly I seem to be getting out-of-date resources.

(It just occurred to me the out-of-date issue may be orthogonal to the slow/no connection issue -- will keep poking locally... Really just wondering if any of this sounds symptomatic/familiar.)

I'm not sure this is a firehose problem, but I figured you may point me to the right direction. I get the following error:

WebSocket connection to 'ws://domain.com/firehose.json?last_message_sequence=' failed: Connection closed before receiving a handshake response

Looking at the source for the consumer, I can see that the comment said "probably firewall." The server is able to receive the response:

[2014-05-02 08:42:34.373 #10371] DEBUG : HTTP GET with last_sequence 0 for path /firehose.json with query "_=1399019839967" and params {"_"=>"1399019839967"}
[2014-05-02 08:42:34.373 #10371] DEBUG : exec returned: `` and `[]`
[2014-05-02 08:42:34.373 #10371] DEBUG : No message available yet, subscribing. sequence: `0`

I'm not sure why the client disconnects immediately after.

Thanks for your help!

According to http://caniuse.com/eventsource more browsers support Server-Sent Events to warrant their consideration as a transport for Firehose.

Change the Last-Message-Sequence client header to Last-Event-ID to support more like semantics in anticipation of an SSE Firehose transport. Also change all instances of "last message sequence" variable names in JS and Ruby to "last event id".

We should release this in two pieces:

1. Make Firehose 1.x series branch support both headers with a deprecation warning for the Last-Message-Sequence header.
2. Release a 2.x branch that removes Last-Message-Sequence and only has a Last-Event-ID header.

Our Firehose::Client::Producer code currently controls how long Redis keeps our message buffer in memory via the Cache-Control: max-age=#{n} header, which I'm pretty sure breaks the HTTP spec.

Use a more explicit header (like X-Firehose-TTL) to instruct Redis in seconds how long it should hang on to the message buffer before it expires.

TypeError: 'undefined' is not a function (evaluating 'this.transport.getLastMessageSequence()')

Consumer.prototype._upgradeTransport = function(ws) {
this.transport.stop();
=> ws.sendStartingMessageSequence(this.transport.getLastMessageSequence());
return this.transport = ws;
};

RubyGems.org doesn't report a license for your gem. This is because it is not specified in the gemspec of your last release.

via e.g.

spec.license = 'MIT'
# or
spec.licenses = ['MIT', 'GPL-2']

Including a license in your gemspec is an easy way for rubygems.org and other tools to check how your gem is licensed. As you can imagine, scanning your repository for a LICENSE file or parsing the README, and then attempting to identify the license or licenses is much more difficult and more error prone. So, even for projects that already specify a license, including a license in your gemspec is a good practice. See, for example, how rubygems.org uses the gemspec to display the rails gem license.

There is even a License Finder gem to help companies/individuals ensure all gems they use meet their licensing needs. This tool depends on license information being available in the gemspec. This is an important enough issue that even Bundler now generates gems with a default 'MIT' license.

I hope you'll consider specifying a license in your gemspec. If not, please just close the issue with a nice message. In either case, I'll follow up. Thanks for your time!

Appendix:

If you need help choosing a license (sorry, I haven't checked your readme or looked for a license file), GitHub has created a license picker tool. Code without a license specified defaults to 'All rights reserved'-- denying others all rights to use of the code.
Here's a list of the license names I've found and their frequencies

p.s. In case you're wondering how I found you and why I made this issue, it's because I'm collecting stats on gems (I was originally looking for download data) and decided to collect license metadata,too, and make issues for gemspecs not specifying a license as a public service :). See the previous link or my blog post about this project for more information.

I want Firehose to sit alongside the rest of my application (which is in Python).

My preferred use of it is for my python app to push updates to firehose, which pushes it to the browsers.

This seems like it would work except that I can't really figure out how to get the .js file needed. Is there a prebuilt version of it anywhere?

how about autentication? with faye there is a private_pub gem for use the autorhization in rails e faye

I installed firehose on an ubuntu machine with ruby 1.9.3 on it and started it using

RACK_ENV=production firehose server --host staging.mydomain.com --port 7373

If I try your example curl put firehose seems execute a script command on redis which fails with the response

[2013-02-19 22:02:04.160 #12854] ERROR : ERR unknown command 'script'
[2013-02-19 22:02:04.166 #12248] ERROR : reaped #<Process::Status: pid 12854 exit 1> worker=0
[2013-02-19 22:02:04.166 #12248] INFO : worker=0 spawning...
[2013-02-19 22:02:04.169 #13110] INFO : worker=0 spawned pid=13110
[2013-02-19 22:02:04.197 #13110] INFO : Rainbows! EventMachine worker_connections=400
[2013-02-19 22:02:04.197 #13110] INFO : EventMachine: epoll=true kqueue=false
[2013-02-19 22:02:07.395 #13110] INFO : Connected to Redis
[2013-02-19 22:02:07.395 #13110] INFO : Connected to Redis

Everything worked well in my dev env.

Do you have any suggestions what to try/do?

Thx so much for helping out.

The fallback strategy for the js client should account for a failed initial connection.

In your quick demo (readme and slides) you show writing to the queue after a commit. This doesn't work in the latest version of rails, unless you add require "net/http"

When starting firehose it always outputs

Starting 1.1.0 'Rockin' Reconnect', in development

is this really the development env?

If true, how can I start in production?

I dont understand the deployment guide :(

I am having a lot of trouble making Firehose work.

the em-hiredis is not finding the redis server:

[2017-01-24 16:19:12.337 #10] DEBUG : HTTP GET with last_sequence 0 for path /hi with query "cid=2039"
[2017-01-24 16:19:12.339 #10] INFO : [em-hiredis 127.0.0.1:6379] Reconnect failed
[2017-01-24 16:19:12.340 #10] INFO : [em-hiredis 127.0.0.1:6379] Reconnect failed
[2017-01-24 16:19:12.340 #10] INFO : [em-hiredis 127.0.0.1:6379] Reconnect failed
[2017-01-24 16:19:12.840 #10] INFO : [em-hiredis 127.0.0.1:6379] Reconnecting
[2017-01-24 16:19:12.840 #10] INFO : [em-hiredis 127.0.0.1:6379] Reconnecting
[2017-01-24 16:19:12.840 #10] INFO : [em-hiredis 127.0.0.1:6379] Reconnecting
[2017-01-24 16:19:12.841 #10] INFO : [em-hiredis 127.0.0.1:6379] Reconnect failed
[2017-01-24 16:19:12.841 #10] INFO : [em-hiredis 127.0.0.1:6379] Reconnect failed
[2017-01-24 16:19:12.841 #10] INFO : [em-hiredis 127.0.0.1:6379] Reconnect failed
[2017-01-24 16:19:13.341 #10] INFO : [em-hiredis 127.0.0.1:6379] Reconnecting
[2017-01-24 16:19:13.341 #10] INFO : [em-hiredis 127.0.0.1:6379] Reconnecting
[2017-01-24 16:19:13.341 #10] INFO : [em-hiredis 127.0.0.1:6379] Reconnect failed
[2017-01-24 16:19:13.342 #10] INFO : [em-hiredis 127.0.0.1:6379] Reconnect failed
[2017-01-24 16:19:13.342 #10] INFO : [em-hiredis 127.0.0.1:6379] Reconnecting
[2017-01-24 16:19:13.342 #10] INFO : [em-hiredis 127.0.0.1:6379] Reconnect failed
[2017-01-24 16:19:13.842 #10] INFO : [em-hiredis 127.0.0.1:6379] Reconnecting
[2017-01-24 16:19:13.842 #10] INFO : [em-hiredis 127.0.0.1:6379] Reconnecting
[2017-01-24 16:19:13.843 #10] INFO : [em-hiredis 127.0.0.1:6379] Reconnecting
[2017-01-24 16:19:13.843 #10] INFO : [em-hiredis 127.0.0.1:6379] Reconnect failed
[2017-01-24 16:19:13.843 #10] ERROR : Unexpected error when trying to GET last_sequence 0 for path /hi: #<EventMachine::Hiredis::Error: Redis connection in failed state>
[2017-01-24 16:19:13.846 #10] INFO : [em-hiredis 127.0.0.1:6379] Reconnect failed
[2017-01-24 16:19:13.846 #10] INFO : [em-hiredis 127.0.0.1:6379] Reconnect failed
[2017-01-24 16:19:13.847 #10] ERROR : Redis connection in failed state

But redis is running on a docker container and accessible:

redis-cli -h 127.0.0.1 -p 6379 ping
PONG

Resulting on:

curl "http://127.0.0.1:7474/hi?cid=2039"
curl: (52) Empty reply from server

I tried Firehose version 1.2.13 and 1.3.6, but the em-hiredis gem version is the same.

I read this: #28
this: #19
and this: mloughran/em-hiredis#24 (mine is not showing ERROR : Use pubsub client )

No lucky yet

When a client connects over WebSockets or HTTP, Firehose connects to Redis once per message. This is inefficient if there's more than one message that should be replayed to the client. The Firehose server should grab as many messages as it can from Redis, store in memory, and replay to the client with one connection.

• Change the deferrables at https://github.com/firehoseio/firehose/blob/master/lib/firehose/server/channel.rb#L46-L61 to succeed with an array of messages (instead of one message)
• Update WebSockets::DefaultHandler consumer at https://github.com/firehoseio/firehose/blob/master/lib/firehose/rack/consumer/web_socket.rb#L95 to loop through the array of messages
• Update WebSockets::MultiplexingHandler at https://github.com/firehoseio/firehose/blob/master/lib/firehose/rack/consumer/web_socket.rb#L169-L174 to loop through messages.
• Update HttpLongPoll::DefaultHandler callback at https://github.com/firehoseio/firehose/blob/master/lib/firehose/rack/consumer/http_long_poll.rb#L96-L97 to deal with multiple messages. Note that HTTP at the moment is only capable of handling one message at a time per request/response cycle. We could batch messages, but that would require a protocol change. Since WebSockets is so widespread its not worth delaying shipping this patch for adding a new protocol.
• Update HttpLongPoll::MultiplexingHandler - I can't find the code at the moment that does this, but make sure HTTP multiplexing loops through messages and streams them. This transport likely has the same issues as the DefaultHandler.

I'm guessing it is related to rack/rack#481 or more specifically, Rainbows! implementation of that API.

This is really a haproxy/web standards problem because safari uses a WS spec that doesn't work with haproxy.

IE8 and 9 are dying a slow death, but they work fine with HTTP long polling. Firefox, Chrome, and Safari have caught up enough to use the final draft of websockets, so lets remove the SWF polyfil to make the app less complex and more secure.

I have managed to start the firehose server with em-redis 0.2.1 and rainbows 4.5.0.

firehose server              
[2013-10-27 20:52:09.724 #30098] INFO : Starting 1.2.8 'Straight Stream', in development
[2013-10-27 20:52:10.574 #30098] INFO : listening on addr=0.0.0.0:7474 fd=7
[2013-10-27 20:52:10.575 #30098] INFO : worker=0 spawning...
[2013-10-27 20:52:10.576 #30098] INFO : master process ready
[2013-10-27 20:52:10.783 #30137] INFO : Rainbows! EventMachine worker_connections=400
[2013-10-27 20:52:10.788 #30137] INFO : EventMachine: epoll=true kqueue=false

However i can when i try:

curl "http://localhost:7474/hello"

I get curl: (52) Empty reply from server
I wonder if this is a em-redis or rainbows incompatibility problem

I'm having a lot of trouble with Firehose.io and the asset pipeline. Everything seems to go smoothly when config.assets.debug is true, but when I set to false and Sprockets attempts to concatenate everything together, for some reason certain modules don't seem to be getting loaded.

In particular, Firehose.Consumer is missing! Here's what the window.Firehose object looks like at the console in this context:

Object {version: "1.1.0", codeName: "Rockin' Reconnect", Transport: function, LongPoll: function}
  LongPoll: function LongPoll(args) {
  Transport: function Transport(config) {
  codeName: "Rockin' Reconnect"
  version: "1.1.0"
  __proto__: Object

I'm wondering if I'm just missing something here. I'm just including of the Firehose manifest at the end of my own Sprockets manifest with //= require firehose. Note I'm definitely seeing the Firehose.Consumer module in the concatenated Javascript -- I'm wondering if there's something different with how the WebSocket and Consumer modules reference/declare themselves?

Document Firehose protocol for publishers and subscribers over HTTP Long Polling and WebSockets in PROTOCOL.md. Ideally the message formats should:

• Be kept as consistent as possible between publishing and consuming
• Remain as consistent as possible with its representation on the server https://github.com/firehoseio/firehose/blob/master/lib/firehose/server/message_buffer.rb#L10 and the wire format
• Remain consistent over its transport (e.g. web sockets is a message per frame, long polling is 1 message payload per HTTP request, batch subscriptions over HTTP long polling is a JSON array of messages.

I propose the following format for publishing:

// Message from service published to Firehose
{
  "message": "Bee boop",
  "channel": "/greetings/from/mars",
  "ttl": 60
}

And the following for a client consuming a sequence of messages:

// Message from Firehose to consuming client
{
  "message": "Bee boop",
  "channel": "/greetings/from/mars",
  "sequence": 101
}

Instead of grabbing all the messages from the message buffer and calculating offsets in Ruby, do it all on Redis.

The MessageBuffer class is a good place to start. https://github.com/firehoseio/firehose/blob/message-buffer/lib/firehose/server/message_buffer.rb. Be sure the test suite passes for whatever is implemented in place.

The message emitted from this new class should be MessageBuffer::Message objects. This will be compatible with all channel consumers.

Preferably this upgrade is done in a way that's compatible with live production upgrades (e.g. the Lua scripts we load for publishing shouldn't corrupt the message buffer arrays during a rolling restart)

Some work started for this at https://github.com/firehoseio/firehose/tree/message-buffer-redis. This branch may not be necessary depending on how we calculate the offsets for Redis.

Implement ping pong frames and keep WS alive when idle. Maybe send ping frame after 20 seconds of inactivity?

http://www.whatwg.org/specs/web-apps/current-work/multipage/network.html#ping-and-pong-frames

I would like firehose to accept a batch of messages for various channels with a single HTTP request.

The POST body might look like this:

{
  "/my/unique/channel": [{"pay":"load"}, {"would":"have been the body"}],
  "/some/other/channel": [{"more":"data"}, {"that would":"have been the body"}]
}

• The array is an ordered set of what would have been in the request body if the requests were made separately.
• There can be multiple keys. I.e. we can update multiple Firehose channels with a single request.
• Ideally the server would be able to start processing the JSON while it is still being uploaded (i.e. stream it. Don't wait for the full POST body to be uploaded before processing the JSON.)
• Responses might look something like

{
  "/my/unique/channel": {success:2, error:0, last_sequence:4},
  "/some/other/channel": {success:1, error:1, last_sequence:9}
}

It might also be reasonable to return a 500 error in the case of an error, since we would generally expect all messages to be processed.
If any of the POSTed JSON is unexpectedly incorrect (for example, if one element in the array isn't a valid message to publish to a firehose channel), then we could return a 400.

The use cases for this are:

• Reduce the overhead of HTTPS handshakes and HTTP headers.
• Make it easier to write clients which:
  • throttle requests
  • drop unnecessary messages (see rational for this in https://github.com/polleverywhere/firehose/issues/35)

Hello!
I see in the docs that Thin and Rainbows is supported. I was wondering if anyone has had success with using Passenger? It would seem they support websockets as per (https://github.com/phusion/passenger-ruby-websocket-demo) but figured I'd ask if anyone here has tried before I go down a possibly futile rabbit hole :-)

Cheers

From @ProTip:

It appears to me that firehose pubsub occurs only once before Redis is polled again; am I reading this correctly?

1. web_socket calls subscribe on the channel
2. next_message is called. Redis is hit to get the sequence and message list. If the client is caught up the deferrable is subscribed to the pubsub. Deferrable is returned.
3. Deferrable callback is set(I'm not sure when the first opportunity for the deferrable to be resolved is, but it seems to me this should actually be created and the callback set then passed into next_message..)
   ...Later
4. Redis publishes and the callback is run. All deferrables subscribed to the channel are removed and resolved.
5. The deferrable callback calls subscribe again. GOTO 2. -> Redis is hit.

If this is the case then the pubsubbing is way less efficient and Redis gets hit way more than what I would have expected for a pubsub fanout.

To following this along a bit:

For a channel with 6k subscribers:

*Vote occurs -> Redis pubs
*6k deferrables are resolved
*Redis is pinged for the latest sequence and message_list 6k times(I think they are pipelined otherwise it would be 12k).

If not for the latency this situation appears to be trying really hard at N**2'ing Redis lookups.

Metrics on the following have been requested for Firehose:

Connection metrics

• Number of connections
• Number of connections established
• Number of closed connections

Channel metrics

• Number of catch-ups performed
• Number of "channels"
• Channels created
• Channels destroyed
• Channel pubs
• Per channel counters(subscribers,events,sizes,etc)

Redis commands

• Counts on events that cause Redis commands to occur?

Implement these as a module in Firehose. We'll use the same Firehose server to track these metrics.

Sorry if this is noisy/overlaps with the other issue I've filed but I just wanted to be clear about my core issue, which isn't with CORS in particular; it's that I can't even get the basic curl subscription behavior from the firehose server example on the homepage.

On the curl side, when I try to curl "http://localhost:7474/hello", I get acurl: (52) Empty reply from server` error, and curl exits. On the firehose side, I am seeing the following in the logs:

[2013-04-08 16:07:05.748 #55236] INFO : Starting 1.1.0 'Rockin' Reconnect', in development
[2013-04-08 16:07:05.832 #55236] INFO : listening on addr=0.0.0.0:7474 fd=7
[2013-04-08 16:07:05.832 #55236] INFO : worker=0 spawning...
[2013-04-08 16:07:05.834 #55236] INFO : master process ready
[2013-04-08 16:07:05.873 #55289] INFO : Rainbows! EventMachine worker_connections=400
[2013-04-08 16:07:05.873 #55289] INFO : EventMachine: epoll=false kqueue=true
[2013-04-08 16:07:08.656 #55289] DEBUG : HTTP GET with last_sequence 0 for path /hello with query "" and params {}
[2013-04-08 16:07:08.663 #55289] INFO : [em-hiredis 127.0.0.1:6379] Reconnect failed
[2013-04-08 16:07:08.663 #55289] INFO : [em-hiredis 127.0.0.1:6379] Reconnect failed
[2013-04-08 16:07:08.664 #55289] ERROR : Use pubsub client
[2013-04-08 16:07:08.668 #55236] ERROR : reaped #<Process::Status: pid 55289 exit 1> worker=0
[2013-04-08 16:07:08.669 #55236] INFO : worker=0 spawning...
[2013-04-08 16:07:08.705 #56329] INFO : Rainbows! EventMachine worker_connections=400
[2013-04-08 16:07:08.705 #56329] INFO : EventMachine: epoll=false kqueue=true

I maybe missing something even bigger here, but I just wanted to report what I'm seeing out-of-the-box.

First off, I'm not able to subscribe using curl; the server throws this error:

[2013-04-08 02:15:46.703 #15658] ERROR : Use pubsub client
[2013-04-08 02:15:46.707 #15655] ERROR : reaped #<Process::Status: pid 15658 exit 1> worker=0
[2013-04-08 02:15:46.708 #15655] ERROR : Invalid argument - setrlimit
[2013-04-08 02:15:46.708 #15655] ERROR : RLIMIT_NOFILE needs to be increased to >=406  before starting firehose master server

I'm also running into trouble connecting from a browser using the JS Firehose consumer. I seem to be getting CORS-style problems talking to Firehose. I'd be fine with using websockets as the transport if I can configure that (assuming that would let me work around whatever I'm seeing.) At any rate the errors I'm getting from a browser console are of the form

XMLHttpRequest cannot load http://localhost:7474/resources/1.json. Origin http://localhost:3000 is not allowed by Access-Control-Allow-Origin. 

A bit weird. I'm thinking I've missed something during setup, but would welcome any thoughts/help here. Thank you!

I'm having difficulty getting a firehose instance talking to redis over a unix domain socket.

I'm hoping this is possible as it would make my deployment strategy a bit simpler.

I've tried passing a few different things and get errors when I try to create a new subscriber, e.g., via curl [presumably the errors themselves are from em-hiredis?]: REDIS_URL=redis://tmp/redis.sock (I get ERROR: no connection); REDIS_URL=/tmp/redis.sock, REDIS_URL=unix:///tmp/redis.sock, REDIS_URL=redis:///tmp/redis.sock (in this case I get an error about not being able to convert nil into String; I'm wondering if this is an em-hiredis exception bubbling up complaining about the URL being malformed).

[Em-redis apparently provides support for this, as per some recent Github issues; I'm wondering if I'm missing something basic here. I've got redis bound to /tmp/redis.sock as in the URLs above, and can talk to it over that domain socket via redis-cli without issues.]

Curious to know how firehose.io can be scaled.

Can the Producer and Consumer be scaled across multiple compute instances behind a load balancer?

I've looked at the code but not 100% sure it is possible -- especially with regards to the Consumer.
Any clarification would be appreciated.

The client can detect a gap in the last_message_sequence and fire a "messagedropped" callback. This callback could be used by end users to send dropped message telemetry to a metrics server or have the client fill in the gaps by accessing their web servers.

Extract the Firehose Ruby client code from the firehose gem and move into its own repo, then release as the firehose-client gem. This will reduce the number of dependencies needed for projects that want to publish messages to Firehose.

Client gem should live at https://github.com/firehoseio/ruby

I'm doing some tests with firehose and found something I cannot solve.

In the Keynote you say that you use rabbitMQ which queues messages in order to send them after a reconnect.

This doesn't seems to work for me, as the exact same message arrives after every page reload for 2h now.

Can I turn off this "feature"?

Thx for helping me out

it should be "keeps client-side Javascript models in sync"

Firehose is currently hard-coded to hold 100 of the most recent publish messages per channel so that if a subscribes' connection drops, it can reconnect and receive the remaining messages. If a client is disconnected and 101 messages are published, a message will be dropped upon reconnection.

A 100 message buffer is not always desirable.

In many of cases the publisher concerned only about giving a client the latest representation of a resource that is published to Firehose; thus the buffer size should be 1. Then if a client connection drops for a few moments and 101 messages are published, 100 messages will be dropped and the client will simply get the most recent representation of that resource.

Firehose::Server::Publisher should accept the header X-Firehose-Buffer-Size: 1 where the X-Firehose-Buffer-Size number is greater than 1. Firehose should return a 400 Bad Request if a value less than 1 is specified.

The Firehose::Client::Publisher should emit the header X-Firehose-Buffer-Size: 10 when Firehose::Client::Publisher#publish(message, :buffer_size => 10).to('/my/resource') method is called.

Break the JS code out of https://github.com/firehoseio/firehose/tree/master/lib/assets/javascripts to https://github.com/firehoseio/firehose/javascript and package up as an NPM package (firehose is taken, so firehose-client?)

Can I send messages back to Firehose from the javascript consumer?