dosomething / message_broker_producer Goto Github PK

View Code? Open in Web Editor NEW

0.0 15.0 0.0 104 KB

Drupal 7.x producer functionality for the Message Broker system

License: Apache License 2.0

PHP 100.00%

message_broker_producer's Introduction

##Message Broker Producer

This Drupal 7 module provides producer access to the Message Broker / Quicksilver system.

Wrapper function for Object Library calls
Administration pages:
- RabbitMQ authentication configuration
- Add/Edit/Delete Productions
  - Productions are a collection of settings for connecting to a RabbitMQ server that define exchange and related queue settings.
- Queue status

Requirements:

Drupal 7
Message Broker PHP Library - https://github.com/DoSomething/messagebroker-phplib

Administration Access:

Once the module is enable goto: /admin/config/services/message-broker-producer

message_broker_producer's People

Contributors

Watchers

message_broker_producer's Issues

Update server config health check

Remove existing, non functional Rabbit server config listing.
Add health check for server config settings to confirm connection works:
http://10.241.0.14:15672/api/aliveness-test/dosomething

Reference RabbitMQ Management API: http://hg.rabbitmq.com/rabbitmq-management/raw-file/rabbitmq_v3_3_5/priv/www/api/index.html

Related:
#3

Make admin form collapsible / collapsed

UI nice to have.

Hide routing key and enable/disable settings in production creation form

When a new production entry is created hide the:

routing key
enable / disable settings
This settings will be made available in the edit state of the form where the exchange and related queues has been selected. The exchange type and queue bindings define what a valid production message key can be.

A production entry can only be enabled when a valid routing key is defined.

Add GPL2/3 license

While all DS repos are MIT licensed Drupal modules require GPL. Add GPL license file to this repo.

Add loggingQueue admin setting form item

In support of the addition of the loggingQueue admin form items will be needed to manage the RabbitMQ settings.

Note to self : create new release once this is merged to trigger addition of changes to next ds build.

Invalid link in error message

When Rabbit is not properly configured, you get this error message:

Failed to create MessageBroker library (messagebroker-phplib) instance. Check configuration settings in administration interface.

The link directs you to http://admin/config/services/message-broker-producer/mq-settings -- Should use a l function to output to correct path.

Transactional test entries

Add test links to the message_broker_producer module for each of the transactional types:

user_register
user_password
campaign_signup
campaign_reportback

to produce test transactions. Support payload formats for each of the transactional types as defined in dosomething_user.module dosomething_user_mbp_request()

Related Trello card: https://trello.com/c/EsFN1kwc

Coder review of .install file

Run the .install file through the Coder standards and fix the errors found unless there's a compelling reason not to.

Format RabbitMQ status details

@tz-earl as discussed, complete RabbitMQ status reports based on cURL API calls to the Rabbit Management API. See message_broker_producer_status() for the cURL calls and the related message-broker-producer-status.tpl.php template to format the output.

Don't forget to drush cc all whenever you make template changes.

Mailchimp Subscription only

Need to be able to subscribe to a Mailchimp Group ID but no transactional.

Add CRUD function for "productions"

Based on feedback from @aaronschachter : DoSomething/legacy-website#3240 (comment) add CRUD functikon for "productions". The function will be used by custom Drupal site specific hook_install() functions to automate the setup of productions in message_broker_producer.

Add loggingQueue

Request from @angaither for loggingQueue for consuming to log Message Broker activity to a Mongo db.

Display more wording if no Virtual Hosts, Servers, or Contexts found.

As implemented, the theme template just displays a header with nothing else if no Virtual Hosts are found.

If no Servers (listeners) or no Contexts are found, nothing at all is displayed.

Move produceTransactional out of messagebroker-phplib

See related issue: DoSomething/messagebroker-phplib#12

Move the producer logic called by the module out of the messagebroker-phplib library. The code specific to the producer tasks should live in the module rather in the general library.

Default message_broker_producer_queue_campaign_cache_binding_key value

The default form value for message_broker_producer_queue_campaign_cache_binding_key should be user.campaign.* not campaignCache.

Add "delete" to admin interface for Productions

Add general delete functionality for production entries in the message_broker_producer database table.
Add delete link from production listing table
Delete should have a confirmation step to ensure entries are not mistakenly removed.

Report missing messagebroker-phplib library

Calls to message_broker_producer_request() in Drupal 7 module should report missing messagebroker-phplib library to support correct installation.

This should also include a test for messagebroker-config.

Add support for binding_patterns

Add support for binding_patterns as an array vs a single binding_pattern as a string. When loading the queue config, if more than one binding_pattern is detected:

create additional queue entries / connections using each pattern.
don't define the consumer_tag, allow Rabbit to auto create with each binding as a consume option

Add vhost

Add vhost setting to admin.

Install Bug

DEBUG [15ad1e54] PHP Parse error: syntax error, unexpected T_IS_NOT_EQUAL in /var/www/beta.dosomething.org/releases/20141104153348/html/profiles/dosomething/modules/contrib/message_broker_producer/message_broker_producer.install on line 168
DEBUG [15ad1e54] Drush command terminated abnormally due to an unrecoverable error. [error]
DEBUG [15ad1e54] Error: syntax error, unexpected T_IS_NOT_EQUAL in

Sample message_broker_producer_request('produceTransactional', $payload) calls

Create outline of how to use message_broker_producer_request('produceTransactional', $payload) in New World app.

Examples for:

New User Registration
User Password Reset
Campaign Signup
Campaign Report Back

Example:

$payload = array(
    'activity' => 'campaign-signup',
    'event_id' => 8012345,
    'email' => '[email protected]',
    'merge_vars' => array(
      'FNAME' => 'First-Name',
      'CAMPAIGN-TITLE' => 'Campaign Test Title',
      'ACTION' => 'Some call to action',
      'STEP-ONE' => 'Step one from Do It',
      'STEP-TWO' => 'Step two from Do It',
      'STEP-THREE' => 'Step three from Do It',
  ),
);
message_broker_producer_request('produceTransactional', $payload);

Need a better way to refresh existing Production records if the config file changes

Currently the only way to do this, would be to go to the admin form for the production, e.g.
http://dev.dosomething.org:8888/admin/config/services/message-broker-producer/add-edit-production/transactional_campaign_group_signup

click the "Reload queues from config file.", and then re-de-select the queues to include.

Ideally, the record updates itself with the queues you want, but reading the config file and updating the entry. If you're changin the config file, the module should automatically find the update and make it... shouldn't have to manually refresh the production yourself.

Add message_broker_producer database table

Add database table to store "production" settings. The table will consist the columns:

machine_name : to be referenced by message_broker_producer_request()` calls. (Unique string).
config : Selected exchanges and the related queues. (serialized string)
status : Enable / disable entry (boolean)
last_update : The last time the config settings were refreshed from the mb_config.json file. (integer timestamp). The table will reference what exchange / queue settings to include in the message_broker_producer_requests() based on the machine_name sent in the request.
Create hook_schema() entry in .install
hook_uninstall() to remove table
enable_enable() to confirm module installation
Update hook_requirements to include messagebroker-config.

Related Issue:
#32

Coder review of .module

Monitor RabbitMQ queue status

From the Drupal module: admin/config/services/message-broker-producer lists the RabbitMQ status for:

// List queues
rabbitmqctl list_queues

// List exchanges
rabbitmqctl list_exchanges

// List bindings
rabbitmqctl list_bindings

Add timezone to date values in MB payload

As a part of the process of converting Message Broker transactions to include time zone values in date settings for message payloads - needed for affiliate sites where transactions / activities will take place in different timezones.

In addition to activity_timestamp:
https://github.com/DoSomething/message_broker_producer/blob/master/message_broker_producer.module#L186 Add:

`activity_date` => date("c") 
2004-02-12T15:19:21+00:00

Rename productionType

Low priority task but a nice to have.

Per our walkthrough yesterday, the productionType variable is storing what RabbitMQ Exchange to send the request to. The code would be a bit more readable if %productionType was renamed to something like $exchange_name (keep lowercase / underscores for Drupal standards?)

Hardcoded DS config file

This is more of a placeholder for discussion and not high priority (or even low priority?), but if this module is to be used by any external non DS use, there wouldn't be much use for the DS Message Broker config library.

Module could potentially be altered to just read a file from a URL stored in a variable.

Datatable storage for configuration

Also a nice to have, but possibly not necessary. Adding an issue for possible enhancement but mainly for documentation / discussion.

The variables used for Producer Configuration (admin/config/services/message-broker-producer/mq-settings) could be implemented using a datatable. This way we wouldn't need to add a new fieldset to add new queues, but would simply add a new record to a queues datatable.

Refactoring to use this architecture would make the module a bit more friendly towards open-source / usage by others.

Campaign Reportback PhpAmqpLib error

WHen reporting back to https://staging.beta.dosomething.org/node/17

Warning: fwrite() expects parameter 1 to be resource, null given in PhpAmqpLib\Wire\IO\StreamIO->write() (line 61 of /var/www/beta.dosomething.org/releases/20140408203302/html/profiles/dosomething/libraries/messagebroker-phplib/vendor/videlalvaro/php-amqplib/PhpAmqpLib/Wire/IO/StreamIO.php).
Warning: fwrite() expects parameter 1 to be resource, null given in PhpAmqpLib\Wire\IO\StreamIO->write() (line 61 of /var/www/beta.dosomething.org/releases/20140408203302/html/profiles/dosomething/libraries/messagebroker-phplib/vendor/videlalvaro/php-amqplib/PhpAmqpLib/Wire/IO/StreamIO.php).

Log failed MessageBroker exceptions

In https://github.com/DoSomething/message_broker_producer/blob/0.2.4/message_broker_producer.module#L246

output Exception $e with watchdog to help for debugging

Add "update" to admin interface for Productions

Add "update" of CRUD admin functionality to edit production entry in message_broker_producer database table.

Load existing production entry in message_broker_producer_production_options_form() based on machine_name passed in URL. Ex: admin/config/services/message-broker-producer/add-edit-production/%
Add link to production listing for each entry that loads entry in edit form
In production entry edit form add checkbox form elements to queues assigned to the exchange to allow for loading of specific queues in production entry.

Producer Tests

Ways to trigger message_broker_producer_request() test:

Add link to admin page
Drush command

Remove campaign cache exchange and queue

Further steps in removing the campaign cache api functionality as it's been replaced by https://www.dosomething.org/api/v1/content/:nid

See
DoSomething/legacy-website#3014
https://jira.dosomething.org/browse/MB-36

Rename database

Rename the database message_broker_producer to message_broker_producer_productions.

Support campaign_group_signup transactions

Add support for campaign_group_signup as campaign.signup.transactional.

See:
DoSomethingArchive/message-broker#55

Add support for "cacheCampaign" transactions

Add support for '"produceCampaign" transactions.

Add support for connecting to "cacheExchange" and "cacheQueue"
Pass payload to exchange$messageBroker->publishMessage($message) call.
Refactor message_broker_producer_request() to support "produceTransactional" and "produceCampaignCache" $method.

See for the addition of related functionality:
DoSomething/legacy-website#1844

PHP error when connecting to MessageBroker fails

PHP error when connecting to MessageBroker fails.

See:
DoSomething/legacy-website#1749

If status is not "ok", show current host, port, etc. instead.

If the initial call to RabbitMQ does not return a status of "ok", then show the current host, port, user, and vhost as recorded in the database. Skip the other api calls and suppress the normal output.

Support for vhost setting. Currently it appears the cURL calls are connecting to the default ("/") vhost.

Refactor message_broker_producer_request()

Refactor message_broker_producer_request() to use the production settings in the message_broker_producer table.

rename $productionType to $production where $production is an entry in the message_broker_producer table that defines the exchange and queues to send the request to.
connect / define $config values to the exchange and queues based on the settings in production entry requested
remove the $param['activity'] logic and use production entries that include $routing_key values

Provide shorter time-out for curl calls

Figure out what would be an appropriate shorter time-out for the curl api calls, and implement it.

Add "create" of CRUD admin functionality

Add "create" of CRUD admin functionality for message_broker_producer_production_options_form() to add production entry to the message_broker_producer database table.

Create form elements at admin/config/services/message-broker-producer/production-options -> message_broker_producer_production_add_update()
- production_name (string) - text field
- exchange (string) - select
- routing key (string) - text field
- status (boolean) - checkbox
  Note: Use pass-thru function to allow the passing of the machine_name to target entries for update "U" functionality.
Validate values submitted for production entry:
- machine_name must be unique
- An exchange must be selected
Add SQL in message_broker_producer_production_options_form_submit() to support SUBMIT to message_broker_producer database table.

Make productions exportable as a feature

Implement integration with Features module:
https://www.drupal.org/node/581526

Remove Admin Exchange and Queue setting form

Remove admin interface based exchange and queue settings. The settings will be managed by the values in mb_config.json.

Remove admin form elements
Provide admin interface to select exchange and related queues
Store selection in Drupal variables table
List selected exchange and related queue settings for reference.

New user registration transaction details will be queued (user.registration.*) in userAPIRegistrationQueue to be consumed by mbc-userAPI-newUser
User activity in campaigns (sign up and report backs) will be queued (campaign..) in userAPICampaignActivity to be consumed by mbc-userAPI-campaignActivity

See:
Campaign activity: DoSomethingArchive/message-broker#22
New Users: DoSomethingArchive/message-broker#35

Binding key default value

Adjust binding key default value for userAPIregistration queue.

Add message_broker_producer_payloads table

Add support for message_broker_producer_payloads table.
- Key/value table row values define message payload values for message_broker_producer_productions entry based on production_id.
Add production_id to message_broker_producer_productions
Admin interface to manage Key/value/production_id values.

Notes:

support for token replacement using ['token']['<name>'] = <value> in message_broker_producer-request() call

Load mb_config.json

Load exchange and queue settings from mb_config.json
Adjust admin interface to present possible exchange / queue settings for use by the module when producing entries in a RabbitMQ system.