topcoder-platform / onboarding-processor

Processor to act on various platform events to update the onboarding statuses of new users

Onboarding API Checklist

Dependencies

• Nodejs(v14+)
• Kafka
• Informix
• Java 1.8

Configuration

Configuration is at config/default.js. The following parameters can be set in config files or in env variables:

• PORT: port number the health check dropin listening on
• LOG_LEVEL: the log level
• KAFKA_URL: comma separated Kafka hosts
• KAFKA_CLIENT_CERT: Kafka connection certificate, optional; if not provided, then SSL connection is not used, direct insecure connection is used; if provided, it can be either path to certificate file or certificate content
• KAFKA_CLIENT_CERT_KEY: Kafka connection private key, optional; if not provided, then SSL connection is not used, direct insecure connection is used; if provided, it can be either path to private key file or private key content
• KAFKA_GROUP_ID: the Kafka group id
• MEMBER_API_URL: the member api url
• STANDARD_TERMS_ID: The Topcoder standard terms id
• NDA_TERMS_ID: The NDA terms id
• PROFILE_UPDATE_EVENT_COUNTRY_FIELD_NAME : The name of the field which corresponds to country value in the message generated by update profile event; default value is payload.competitionCountryCode
• TERMS_USER_AGREEMENT_TOPIC: The kafka topic on which the processor will listen to terms agreement events
• USER_TAXFORM_UPDATE_TOPIC: The Kafka topic to which to listen to user tax form updated events
• UPDATE_MEMBER_PROFILE_TOPIC: Topic for receiving member profile update events
• CREATE_MEMBER_PROFILE_TRAIT_TOPIC: Topic for receiving member profile trait creation events
• UPDATE_MEMBER_PROFILE_TRAIT_TOPIC: Topic for receiving member profile trait update events
• DELETE_MEMBER_PROFILE_TRAIT_TOPIC: Topic for receiving member profile trait removal events
• UPDATE_PROFILE_PICTURE_TOPIC: Topic for receiving update profile picture events
• auth0.AUTH0_URL: Auth0 URL, used to get TC M2M token
• auth0.AUTH0_AUDIENCE: Auth0 audience, used to get TC M2M token
• auth0.AUTH0_CLIENT_ID: Auth0 client id, used to get TC M2M token
• auth0.AUTH0_CLIENT_SECRET: Auth0 client secret, used to get TC M2M token
• auth0.AUTH0_PROXY_SERVER_URL: Proxy Auth0 URL, used to get TC M2M token
• auth0.TOKEN_CACHE_TIME: Auth0 token cache time, used to get TC M2M token
• PAYMENT_METHODS_PROCESSOR_CRON_EXPRESSION: The cron expression for the user payment methods processor, default value is * * * * * (every minutes, used for development only)
• MODIFIED_PAYMENT_METHODS_TIMEFRAME_DAYS: The timeframe expressed in days for which to get the updated user payment methods from informix database, default value is 3 which means that when the job runs it will get the user payment methods records modified in the last 3 days
• INFORMIX: This configuration object contains the configuration parameters for Informix database connection, the confguration parameters defined in this object are self-explanatory, for more details refer to config/default.js
• ID_VERIFICATION_PROCESSOR_CRON_EXPRESSION: The cron expression for the id verification processor, default value is * * * * * (every minutes, used for development only)
• PAUSE_ID_VERIFICATION: If this value is true then while the job is running, check if it is true, if so just log the log message The id verification is currently paused
• FETCH_LOOKER_VERIFIED_MEMBER_TIMEFRAME_DAYS: The timeframe expressed in days for which to fetch looker api, default value is 3 which means that when the job runs it will get the id verification records in the last 3 days
• lookerConfig: This configuration object contains the configuration parameters for looker api, for more details refer to config/default.js

Local Dependencies setup

1. Navigate to the directory local

2. Run the following command

   docker-compose up -d

This will setup both Kafka and Informix database with Topcoder databases all set

Local deployment

0. Make sure to use Node v14+ by command node -v. We recommend using NVM to quickly switch to the right version:

   nvm use

1. From the project root directory, run the following command to install the dependencies

   npm install

2. To run linters if required

   npm run lint

   To fix possible lint errors:

   npm run lint:fix

3. Local config

   In the onboarding-processor root directory create .env file with the next environment variables. Values for Auth0 config should be shared with you on the forum.
   

   # Auth0 config
   AUTH0_URL=
   AUTH0_AUDIENCE=
   AUTH0_CLIENT_ID=
   AUTH0_CLIENT_SECRET=
   AUTH0_PROXY_SERVER_URL=

   • Values from this file would be automatically used by many npm commands.
   • ⚠️ Never commit this file or its copy to the repository!

4. Start the processor and health check dropin

   npm start

5. Run unit tests

   npm run test
   npm run test:cov

Local Deployment with Docker

To run the processor using docker, follow the below steps

1. Navigate to the directory docker

2. Rename the file sample.api.env to api.env

3. Set the required configuration parameters in the file api.env.

   Note that you can also add other variables to api.env, with <key>=<value> format per line.

4. Once that is done, run the following command

   docker-compose up

5. When you are running the application for the first time, It will take some time initially to download the image and install the dependencies

onboarding-processor's Issues

Traits API returns profile_picture: false even if Profile picture is uploaded

Steps
Go to any users profile settings : https://www.topcoder-dev.com/settings/profile/
Upload a profile picture
Call {{URL}}/members//traits?traitIds=onboarding_checklist API
See the response

As soon as uploading a profile picture, it will return profile_picture: true but after sometimes it is set to false

example user: taasintake6

Store the date as timestamp for id_verification onboarding step

it seems we are using different date format for id_verification steps's date field compared to other places. It should use timestamp (in millis) instead of string. We can just convert the string verification_date from the looker to the timestamp by parsing it as date and then converting it to millis assuming the 00:00:00 as time for the verification date.

Update Tax Form Receiver to generate v5 bus events for docusign document completion events

Overview

In this challenge, we will update the https://github.com/topcoder-platform/terms-receiver, to parse the DocuSign envelope and raise a new event on Topcoder bus

Objective

As criteria to receive the payments by a member from Topcoder, the member has to submit the tax form, please check https://www.topcoder.com/thrive/articles/payment-policies-and-instructions for more detail. For this, we are using DocuSign. There are two types of tax form W-9 or a W-8BEN. When the member submits the tax form, DocuSign calls our webhook at terms-receiver code base. Since these forms are embedded on the Topcoder platform using legacy codebase, we don’t have mapping terms set up on the database. But we need a way to update member traits whether the member has submitted the tax form or not.

For this, we will update the DocuSign webhook handler to capture whether the form submitted are related to Tax form or not. If they are related to tax then raise the terms.notification.user.taxform.updated

Individual Requirements

• Update the services/DocusignService.js
• After Line number 82, we need to check DocumentStatuses[0].DocumentStatus[0].TemplateName on the webhook payload whether the values are W-9(TopCoder) or W-8BEN(TopCoder). If the name is either of them, then raise the event with following payload

userId: The topcoder user Id based on user email
taxForm: DocumentStatuses[0].DocumentStatus[0].TemplateName
Handle: topcoder handle
created: new Date()

• The template name should be configurable
• The event name should be configurable
• Provide guidance on how to test this

Implement onboarding trait for ID Verification step

for tracking purposes

Remove strict validation from legacyId for terms event payload

We are seeing too many instances of following error:

We should get rid of mandatory legacyId validation for the terms event as we are not using the legacyId anywhere in our process.

Implement onboarding trait for profile completeness

for tracking purposes

Updates/deletes to profile settings are not shown in the response

Steps
Go to any users profile settings : https://www.topcoder-dev.com/settings/profile/
Update/delete an existing trait (ex- bio)
Call {{URL}}/members//traits?traitIds=onboarding_checklist API
See the response

the updated/deleted data are still displayed as true in the response

example user: taasintake6
Deleted data - bio & skills

Remove title field for now from the profile completeness check

As we don't have the field to capture the title in member profile, it does not make sense to keep counting the title field for the completeness of the profile.

Implement onboarding trait for Payment method setup

for tracking purposes, fyi @sdgun

Add address to the profile completeness onboarding step check

We missed to add this while building profile completeness check.

Create traits for standard and nda terms sign

Welcome message flips from handle to name

On loading - the flip from handle to name could be avoided :

Onboard.mov

None of the profile updates are returned in the response for new user

Steps
Go to a new user's profile settings who doesn't have any traits yet : https://www.topcoder-dev.com/settings/profile/ ( ex. user - gigonboard)
Update the profile settings like bio, education work etc
Call {{URL}}/members//traits?traitIds=onboarding_checklist API
See the response

The response is empty

Onboarding trait save fails because ES consistency in member api

There is existing issue with member api and its ES indexing which causes the trait data to not reach to the ES index many times. And when this happens, primary database (DynamoDB) and ES have different states of the same data which causes troubles in further updating the traits for that particular user. Here is what happens:

• User or onboarding processor adds some trait for the first times e.g. adding the education for the first time
• Due to unknown reason in member api, it saves it to the dynamodb but not to the ES index (most recent issue for this is logged here but this is consistently there since the member api started supporting traits.
• Now, when user or our onboarding, fetches the trait, it does not receive it from the GET endpoint and hence it tries to make a POST call instead of PUT or PATCH.
• member api internally looks at the primary source of truth (dynamodb) to verify if the requested trait exists or not and it founds it in the DB
• now the member api errors out because caller is not allowed to make a POST call when the trait already exists.

Validate the fields for the profile completeness when any other field is updated

At the moment if user updates any setting in profile, it would update only skills flag. It would be better to validate other fields(profile picture, profile bio, education, work experience & language) also for the profile completeness when any other field is updated.

Have a feature to disable ID verification

• Introduce a configurable variable PAUSE_ID_VERIFICATION
• default value false
• If this value is true then while the job is running, check if it is true, if so just log the log message The id verification is currently paused.
• Note the job will still run but not call the looker API and update the traits.

Update processor to handle the docusign events generated by terms-receiver

onboarding processor has to consume the events generated by #3 and update the user traits accordingly.

When delete skills and make an update settings, the skills flag is still true in the response.

Steps
Delete all existing skills of a user from profile settings page
Make an update to any other setting(bio, education etc),
Call the traits API

Result
skills flag is still true in the response.

Expected
Skills should be false