heroku / kafka-demo

twitter-display

A simple app that streams tweets containing a specified set of keywords to web browser clients.

Keywords are specified in the kafka-tweets app. They are read from a Kafka topic named 'test' from the 0th (zeroth) partition in that topic.

Development Setup

Node > v5.x required

npm install

Additionally these environment variables need to be defined:

• KAFKA_URL: A comma separated list of SSL URLs to the Kafka brokers making up the cluster.
• KAFKA_CLIENT_CERT: The required client certificate (in PEM format) to authenticate clients against the broker.
• KAFKA_CLIENT_CERT_KEY: The required client certificate key (in PEM format) to authenticate clients against the broker.

Development Running

npm run dev

Open http://localhost:3000 in a browser and watch tweets stream in...

Deploy

git clone [email protected]:crcastle/twitter-display.git
cd twitter-display
heroku create

You can define the below environment variables manually, or you can run this command to define them from another app that already has a Kafka cluster attached: heroku addons:attach my-originating-app::KAFKA (where "my-originating-app" is the app to which the cluster is already attached)

Or manually:

heroku config:set KAFKA_URL=
heroku config:set KAFKA_CLIENT_CERT=
heroku config:set KAFKA_CLIENT_CERT_KEY=

kafka-demo's Issues

Add page titles

Both HTML title and a title in the purple header.

Change labels for aggregate metrics

The "/s" and "per second", "per minute", and "per hour" labels are a little confusing.

Let's leave the "/s" on each metric, but change the text below so that there is:

• no text below the left column
• LAST MINUTE AVG below the middle column
• LAST HOUR AVG below the right column

bar chart y axis log scale

Add more colors to chart color palette

Currently there are four as defined here. Can you add two more? I think I am going to limit the number of keywords shown to five, but it would be good to have an extra just in case.

streamgraph: only animate path x values

http://bl.ocks.org/mbostock/1642874

issues with node version 7 and up

this doesn't work with node v7 and up

Style x and y axis

• no lines like in mockup
• place outside of chart border
• add labels below stats

related words: fix text not showing up

[streamgraph] fix streamgraph animation

maybe smooth out the curves more and prevent lines from shifting during animation

streamgraph: more elegant fix for mismatched data sizes

Kafka will sometimes backup and push different sized arrays for different topics. Since the streamgraph has to be normalized to always have the same number of data points and x values for each topic, this can cause problems. For now default data (the current time) is being pushed so it doesn't break, but there's probably a better way to fix that.

Reduce high CPU load

Found this on the google. Looks like you can define custom animations where you can specify the frame rate. Might a lower frame rate help? Also I'm wondering if resolving #10 will help too...

The worst consequence of this is that the tab with this page freezes on my computer (in Chrome 53 on OS X 10.11.6 on mid-2015 MBP Retina 15" 2.8GHz i7 16GB RAM). Others have said that their browser crashes (maybe safari?). My entire browser does not crash. And least bad is the fan turning on and machine getting hot, but that's still not great.

Document color palette switch process

Remove y-axis ticks and labels from streamgraph

The y-axis on this graph has confused a few people. I see how zero in the center could make sense, but let's just remove the numbers and ticks from this y-axis. Doing some quick googling for streamgraph shows that most people implement it without y-axis labels and ticks.

Otherwise, this one's looking great!

deploy!

Update README with instructions for deployment of all twitter-* repos

Change the rates that are shown

This one is mostly for me (Chris) to do, but if I change the JSON key, you'll need to update on your end. I'm not sure what these should be yet; we got some conflicting feedback. Margaret wanted to see them moving more. Then other feedback was to make them per second, per minute, and per hour to make it easier to explain. I think I'm going to go with the latter for now.

[consumer] force topics to consistent order before broadcast

Demo not showing anything

Hi, I set up the demo per the instructions here: https://heroku.github.io/kafka-demo/ but I'm not seeing any data on the app. Can someone help ensure I configured everything properly?

Here's my setup:

• data-ingest:
  • KAFKA_TOPIC=foobar
  • TWITTER_TRACK_TERMS=movies,hadoop,johnwick
• data-fanout:
  • KAFKA_TOPIC=foobar
  • TWITTER_TRACK_TERMS=movies,hadoop,johnwick
• data-aggregate:
  • KAFKA_CONSUMER_TOPIC=foobar
  • created a dyno for each of the track terms, and scaled down the web dyno to 0
• data-relatedwords
  • KAFKA_CONSUMER_TOPIC=foobar
  • created a dyno for each of the track terms, and scaled down the web dyno to 0
• data-visualization:
  • TWITTER_TRACK_TERMS=movies,hadoop,johnwick

I also attached the Kafka cluster to each of the five apps above. However, I'm still seeing empty on the app:

related words: animate group transform and bubble placement/size

Create GitHub Pages explanation page

Move all twitter-* repos to Heroku GitHub org

Improve streamgraph x-axis labels

The x-axis number changing is a little hard to understand quickly. I can think of two options to make this clearer... one option is to not animate the x-axis. Just have the numbers on a static axis go from 0:00 on the right to 0:59 on the left. No animation of the text. The other option would be to use the timestamp of the data point instead of the number of seconds since it was added to the chart. Again no animation of the text.

And another option I just thought of is leaving the animation but adding an x-axis label that says "Seconds Since Message Received". But if removing some of the animation here significantly reduces CPU load (#16), then I'd rather go with that option.