git clone [email protected]:heroku/kafka-tweet-producer.git
cd kafka-tweet-producer
You can create an app in one of the following two ways. If you're ok with Heroku's randomly generated app name
heroku create
If you want to create your own app name
heroku create my-awesome-app-name
- Create:
heroku addons:create heroku-kafka:beta-standard-0
- Attach:
heroku addons:attach my-originating-app::KAFKA
(where "my-originating-app" is an app to which the cluster is already attached) If you created a new cluster, wait until it is ready to use. This command will read you a quote from Franz Kafka when the cluster is ready to use.
heroku kafka:wait; say $(curl kafkafra.nz)
Create a Kafka topic and configure. I use the name test
here but you can use any valid Kafka topic name.
heroku kafka:create test --partitions 1
Creating a new topic takes some time. Use the same command "wait" command to wait until it's done, then set this environment variable with the name of the topic.
heroku kafka:wait; say $(curl kafkafra.nz)
heroku config:set KAFKA_TOPIC=test
heroku config:set KAFKA_HEAP_OPTS=-Xmx1g
heroku config:set KEYSTORE_PASSWORD=changeit
heroku config:set TRUSTSTORE_PASSWORD=changeit
heroku config:set CLASSPATH="/app/target/kafka-connect-twitter-0.1-jar-with-dependencies.jar"
To obtain the required keys, visit https://apps.twitter.com/ and Create a New App
. Fill in an application name & description & web site and accept the developer aggreement. Click on Create my access token
and populate the below environment variables with consumer key & secret and the access token & token secret.
heroku config:set TWITTER_CONSUMER_KEY=
heroku config:set TWITTER_CONSUMER_SECRET=
heroku config:set TWITTER_ACCESS_TOKEN=
heroku config:set TWITTER_ACCESS_TOKEN_SECRET=
Define environment variable containing terms to track in tweets (more details)
heroku config:set TWITTER_TRACK_TERMS=news,music,hadoop,clojure,scala,fp,golang,python,fsharp,cpp,java
Deploy to Heroku and scale-up the dyno type! (A dyno with at least 1GB of RAM -- i.e standard-2x -- is recommended.)
git push heroku master
heroku ps:scale web=1:standard-2x
This repo is mostly code from Eneco/kafka-connect-twitter but configured to be easily deployable to Heroku and use an Apache Kafka on Heroku cluster. The license for that code can be found here.
Additionally, check here to see what, if any, changes have been made to the Eneco/kafka-connect-twitter repo since its code was copied to this repo.