dosomething / quicksilver-node

A collection of Node.js based micro-service applications that make up the Message Broker system for DoSomething.org

MessageBroker-Node

A collection of Node.js based micro-service applications that make up the Message Broker system for DoSomething.org.

• mb-logging-api: An API to send logging data for persistent storage. Currently the persistent storage is a MongoDB.

####Monitoring: https://app.ghostinspector.com/suites/565f1b51811dd7846273df6f

quicksilver-node's Issues

Test coverage of /v1/campaign

Add test coverage for verbs of endpoints:

• /
  • GET
• /v1
  • GET
• /v1/campaign
  • POST
  • GET
  • DELETE

Refactor to sync mb-logging-api vs mb-digest-api

Based on successful development of mb-logging-api, update mb-digest-api.

Test coverage

Add test coverage using supertests and Mocha for each of the endpoints:

• router.get('/'
• router.get('/v1'
• router.post('/v1/imports'
• router.post('/v1/imports/summaries'
• router.route('/v1/user/activity')
  • .post
  • .get

Test Coverage: /api/v1/user/activity

Adjustments to application to update endpoint logic based on test development.

• /api/v1/user/activity:
  • it('GET: Lookup of missing user activity log entries returns 404 response code.'
  • it('POST: User activity log entry returns 200 response code and OK message.'
  • it('GET: Lookup of user activity log entry returns 200 response code. Returned data is formatted as expected.'
  • it('DELETE: Test user activity log entry returns 200 response code.'
  • it('GET: Lookup of deleted user activity log entry returns 404 response code.'

Modernize mb-user-api

• Restructure using Express 4
• Test coverage of all endpoint verbs.

Support Config in Environment Variables

• Add support for missing mb_config.json file, fall back to default settings.
• Add support for environment variables as source of configuration settings
• if missing, create logging directory

Reference - must read and re-read

• 12 Fractured Apps
  • https://medium.com/@kelseyhightower/12-fractured-apps-1080c73d481c

Example of using environment vars:

log.Println("Overriding configuration from env vars.")
if os.Getenv("APP_DATADIR") != "" {
    config.DataDir = os.Getenv("APP_DATADIR")
}
if os.Getenv("APP_HOST") != "" {
    config.Host = os.Getenv("APP_HOST")
}
if os.Getenv("APP_PORT") != "" {
    config.Port = os.Getenv("APP_PORT")
}
if os.Getenv("APP_USERNAME") != "" {
    config.Username = os.Getenv("APP_USERNAME")
}
if os.Getenv("APP_PASSWORD") != "" {
    config.Password = os.Getenv("APP_PASSWORD")
}
if os.Getenv("APP_DATABASE") != "" {
    config.Database = os.Getenv("APP_DATABASE")
}

Works with Docker

$ docker run --rm \
  -e "APP_DATADIR=/var/lib/data" \
  -e "APP_HOST=203.0.113.10" \
  -e "APP_PORT=3306" \
  -e "APP_USERNAME=user" \
  -e "APP_PASSWORD=password" \
  -e "APP_DATABASE=test" \
  app:v2

AfterSchool

A logging support for user imports from After School.

Language support in GET requests.

• Support user_language and campaign_language in mb-user-api GET returned results from calls to /user and /users.
• Filter GET requests by user_language and campaign_language

Related:

• DoSomethingArchive/mbc-digest-email#74
• https://trello.com/c/zg8gUAnc

User Transactional Activites

• Add route to support POSTs of transactional requests: router.route('/v1/user/transactional').
• Add collection to support logging user transactional requests:
  • `var userTransactionalsCollectionName = 'user-transactional';
  • schema:

    var userTransactionalLoggingSchema = new mongoose.Schema({
      logged_date : { type: Date, default: Date.now },
      email : { type : String, trim : true },
      source : {
        type : String,
        trim : true,
        enum: ['US', 'CA', 'UK', 'ID', 'BR', 'MX']
      },
      activity : {
        type : String,
        lowercase : 1,
        trim : true,
        enum: ['user_register', 'user_password', 'campaign_signup', 'campaign_reportback']
      },
      activity_date : { type: Date, default: Date.now },
      activity_details : { type : String }
    });
    userTransactionalLoggingSchema.set('autoIndex', false);

    models.userTransactionalModel = mongoose.model(userTransactionalCollectionName, userTransactionalLoggingSchema);

• Module: user-transactional
  • UserTransactional.prototype.post = function(req, res) {

Test Coverage: /api/v1/import/summaries

Adjustments to application to update endpoint logic based on test development.

• /api/v1/imports/summaries:
  • it('GET: Lookup of missing import summary log entries returns 404 response code.'
  • it('POST: Import log summary entry returns 200 response code and OK message.'
  • it('GET: Lookup of summary import log entry returns 200 response code. Returned data is formatted as expected.'
  • it('DELETE: Test summary log entry returns 200 response code.'
  • it('GET: Lookup of deleted import summary log entry returns 404 response code.'

Remove config settings from repo

The /config directory contains sensitive configuration settings. Remove from repo.

Add Risingstack monitoring

Test node Microservice monitoring tool Trace based on:

*Monitoring Microservices Architectures: Enterprise Best Practices
https://trace.risingstack.com/app/#/infrastructure/570f096d9acadf1100a94e0e/topology

Monitor forever Process

Add automated monitoring to ensure http://subscriptions.dosomething.org is functional. The application lives at 10-100-25-44/usr/bin/nodejs mbc-user-subscriptions-server.js which runs as a daemon:

$ PORT=4711 forever start mbc-user-subscriptions-server.js

Create/Add Campaigns Key

When campaign data is cached create/add an index key with a list of all campaigns cached.

• When campaign is added through POST /campaign create / add to the index key of all the campaign cached values: [nid]-[language]-[expiry timestamp].

  Storing Sets: Sets are similar to lists, but the difference is that they don’t allow duplicates.

  The first item of the array represents the name of the key while the rest represent the
  elements of the list.

client.sadd(['tags', 'angularjs', 'backbonejs', 'emberjs'], function(err, reply) {
    console.log(reply); // 3
});

Reference:

• http://www.sitepoint.com/using-redis-node-js/

Related:

• #2

Test Coverage: / GET /v1/imports

• Add coverage to ensure origin_start and origin_end are returning expected results.
• Confirm logged_date is the current date, not the default 1970-something date

Get Campaigns Key

Create an endpoint to GET a list of all the campaign nid-languages being stored.

• GET /campaign/index

  To retrieve the members of the set, use the smembers() function as following:

client.smembers('tags', function(err, reply) {
    console.log(reply);
});

Reference:

• http://www.sitepoint.com/using-redis-node-js/

Related:

• #1

Allow wildcards in GET /imports requests

Allow wild cards in name string for origin for both a greater than / equal values and a less than value.

Add support drupal_mobile for logging existing users

Support drupal_mobile as a logged value for existing users on the /v1/imports endpoint.

mb-logging-api Test Coverage

Complete test coverage:

• /api/v1/imports:
  • it('POST: Import log entry returns 200 response code and OK message.'
  • it('GET: Lookup import log entry returns 200 response code and expected content.'
  • it('DELETE: Import log entry returns 200 response code and expected OK response.'
  • it('GET: Lookup of missing import log entry returns 404 response code.'
  • it('GET: Lookup of import log entry returns 200 response code. Returned data is formatted as expected.'
  • it('DELETE: Test import log entry returns 200 response code.'
  • it('GET: Lookup of deleted import log entry returns 404 response code.'
• /api/v1/imports/summaries:
  • it('GET: Lookup of missing import summary log entries returns 404 response code.'
  • it('GET: Lookup of summary import log entry returns 200 response code. Returned data is formatted as expected.'
  • it('DELETE: Test summary log entry returns 200 response code.'
  • it('GET: Lookup of deleted import summary log entry returns 404 response code.'

See commented out stubs in test.js.