andreimaxim / river

A reliable streaming API implementation for PostgreSQL.

River

Simon: You're in a dangerous line of work, Jayne. Odds are you'll be under my knife again, often. So I want you to understand one thing very clearly: No matter what you do or say or plot, no matter how you come down on us, I will never, ever harm you. You're on this table, you're safe... 'cause I'm your medic. And however little we may like or trust each other, we're on the same crew. Got the same troubles, same enemies, and more than enough of both.

Now, we could circle each other and growl, sleep with one eye open, but that thought wearies me. I don't care what you've done, I don't know what you're planning on doing, but I'm trusting you. I think you should do the same. 'Cause I don't see this working any other way.

River: Also, I can kill you with my brain.

• from Firefly

River is a reliable streaming API implementation for PostgreSQL version 9.4 and above, using RabbitMQ as a message broker.

Architecture

PostgreSQL version 9.4 offers a set of non-standard SQL commands (NOTIFY and LISTEN) which can be used as an event publication system. A sample trigger that uses the pg_notify method is provided in the doc folder, with ample documentation to simply customization based on individual needs.

The river service listens to those notifications and pushes the payload to a RabbitMQ service.

Basic Usage

TBD

Configuration

The actual usage is quite simple, assuming you already have a properly configured Heroku Connect that syncs all the data from Salesforce into a Postgres database and a RabbitMQ add-on set up.

You need to have the following environment variables defined:

• POSTGRESQL_URL, which points to the Postgres instance mentioned above (defaults to localhost)
• POSTGRESQL_CHANNEL, the channel to listen to for notifications (defaults to rvr_events)
• RABBITMQ_URL, which points to the RabbitMQ add-on (defaults to localhost)
• TOPICS, which defines what tables to sync and how

There are two possible topic formats:

• A list of comma-separated values (e.g. account__c,sales__c)
• A list of key-value pairs (e.g. account__c:accounts,sales__c:sfdc_sales)

In the first case, the service will get events generated by the Postgres NOTIFY method on a specific table (like account__c or sales__c) and then place them into a RabbitMQ exchange named the same (account__c or sales__c).

If you want to have a name for the event names on the Postgres side and a different name for the RabbitMQ exchanges, use the second format. For example, if you define the channels to be account__c:accounts it will use the first value (account__c) as the source table and the put those events in the accounts exchange on the RabbitMQ side.

If no topics are specified, no events will be created on the RabbitMQ side.

Documentation

TBD

License

Soruce Copyright (c) 2019 Andrei Maxim and contributors. Distributed under the Eclipse Public License. See the file COPYING.

Logo (c) 2013-2019 Otis Frampton.