dcsil / jerryrat

Team repository for GithubClient

JerryRat

JerryRat's URL: https://herokujerryratapp.herokuapp.com/
Note: When you are running our app for the first time, you may need to wait for a few seconds. Please be patient.

JerryRat provides marketing solutions to various industry unicorns across the globe. Your products matter and clients are also important to us, why not kick-start the journey with us in furthering enchanting campaigns that best fit to your blueprint? It's our duty to be with you, on the road to success!

Our team name "JerryRat" is inspired by "tomcat" is a free and open-source software which provides a HTTP web server environment written in Java which scripts in Java can run on. Additionally, there is a famous cartoon "Tom and Jerry", we invert the name of "TomCat" to "JerryRat" with this cartoon. It looks cute, isn't it? :)

Introduction

• People
• Diversity
• Team Principles
• WriteUp
• Architecture

Our web app will be deployed on Heroku and is going to use Django as the framework. Apache Spark will be used to process and analyze the data that will be fed to the ML model. Since the user may change their campaign plan frequently, we are going to use Redis to save this data while saving personnel info (which is used to train the ML model) in Amazon Aurora. We will use XGBoost to train a model which takes processed personal information as input and outputs whether this person is worth calling for a promotion. The trained model will be updated regularly based on the most recent data from the database.

The app will output prospective clients to be called but will only generate the list in CSV. The task dispatching should be done by the companies to their call centers offline

Roadmap

The short, medium, and long term goals for project development are written in the Milestone of GitHub issues.

URL

https://herokujerryratapp.herokuapp.com
Note: When you are running our app for the first time, you may need to wait for a few seconds. Please be patient.

Sample Heroku logs

The log message above is generated by logDNA, an add-on in Heroku as a logger for messages created during running our application. Here is an example of automatic training. This log shows a user starts training the model, one cycle of training finishes, and then end training the model. Later the model is saved and the new accuracy is computed. This happens all in the background, and the training process will not be shown in the front end.

Sample Sentry Alerts

This is the Sentry the app hooked up to and some exceptions it recieved.

Write up

Please click here for step-by-step instructions.

Tutorials

JerryUp Campainging Master helps you find out the best candidate clients to call in your campaigns, and give clear views on the client population.

Sign Up

Please sign up before you start. Just follow the instruction here to create your account.

Log In

Enter your email address and password to log in.

Data Entry Page

The data entry page is used to upload the client data. Users can click on the “Browse” button to browse the file in the local machine and upload it when confirmed. There are some column descriptions below that explain the exact format the users need to follow, and what each column means. We only accept csv file here. Once clicked on “Upload File”, the data will be fed into the database for analysis or other further usage.

Analytics Dashboard

There are 7+n graphs in our analytics dashboard. All graphs are bar charts, and the first 7 graphs show how the client conversion rates are different over varying x values, for 7 metrics (client spectrum vs yes/no). In addition, users could add more stacked bar charts about relationships between user profile spectrums (client spectrum 1 vs client spectrum 2), simply by clicking on the add button, and specifying the title and x, y axis in the pop-out window. For each graph, users could toggle the legend to make designated labels visible/invisible. There are also some small add-ons in the bottom widgets, including saving the graph, general zoom-in, window zoom-in, reset, etc. Users could also see the detail (x, y) values by hovering the cursor on the graph. Whiling deleting a graph, just click on the red “X” button on the upper right corner.

Calling Operations

Worthy Client List is a list of clients that have higher probabilities to subscribe to the products as evaluated by the model, and they are the prospective targets to be contacted. The page will show the client numbers, their names and numbers. Users could select the clients they want to call and dispatch the tasks to the call center. (Just a button indicating an imaginary connection, need external API to connect to the cal center)

Model Controls

In the model control page, you can configure your model performance based on your demands like your user data size. You can input the learning rate of the model, the maximum decision tree size of the model (ourmachine learning model is decision tree based), the number of rows of data used to train the model per cycle, and the period to train the model once. All the settings are shown in the table, once you submit a new set of settings, the table will be updated. And you can also start training your model automatically and periodically in the background. To start training, click the button "start training", there is a reminder (red color text) tells you it is get trained at now. To end training, click the button "end training", there is a reminder (green color text) tells you it is not get trained yet.

The Heroku has some problem with Python threading, so the content of the colored reminder might be wrong, but if you click “start training”，the model will be trained automatically. If you stop traing and click “stop training”, the automatic training will be stopped in a period and then the page crashes, you can just refresh the page and work as usual. For more instructions for this, please read the description in model control page.

jerryrat's Issues

test

GitHub action

We may run out of free serving time, and we have to solve that.
We may charge for that.

First step

To-do list:

1. Company name.
2. A first version of your logo.
3. Team Folder with profiles.
   Include any relevant links to profiles or documents such as your GitHub profile, LinkedIn, Resume, Bio, Twitter profiles, etc
   Be fun, write a small paragraph to help your teammates get to know you more
4. Team Principles
5. Leader election

For more details, please refer to https://csc491.dcsil.ca/assignments/a1/

After doing all of this, remember to create a release and submit it at https://learnsoftware.engineering/my_assignments by 11:59:59pm Eastern Time on September 16th.

Roadmap: medium term plan

Django integer list model

Need to figure out some ways to build integer list field in Django models

Tasks today

In this week, we will issue a document about inclusion and diversity specifying following:

1. Should be at least 1 page of writing, 12pt font, single spaced. Write as much as you feel is needed to accurately capture the rest of these requirements.
2. Identify where your team’s backgrounds and expertise may benefit the project
3. Identify where your team’s backgrounds and expertise are not enough for this project
4. Given (3), identify some subject matter experts you think would help fill this gap. Describe how.
5. Identify where your team lacks diversity
6. Describe how the lack of diversity in certain areas could affect the company, such as cultural blindspots
7. Describe how people in other diverse groups benefit your team’s vision

For more information, please view this guideline: https://csc491.dcsil.ca/assignments/a2/

Xgboost compatability on different systems

ClearDB has problem

ClearDB's MySQL version is 5.7 which is not very up to date, this causes some problems with SQL syntax

UI updated needed for Data entry page

implement analysis history for each user account

I think there should be the implementation of user account system and each user should have their own analysis histories.

Database (parse data)

Next step, enable table file (e.x., csv, xlsx) parsing to sql table format and upload it to database.

"Active" in navbar

If we use the "active" css class in the nav bar, the "active" effect or page jumping cannot work normally.
Maybe we have to disable "active" at first.

Tool tips for model configuration

Configurations may be unclear for users, need more tips!

Suggestions about user friendly configuration interface

Possibly we can let users be able to adjust the model and Datapipe in the configuration interface.

Suggestion about control panel page

It would be better if the Control Panel page could provide a user interface to change the model training iteration and discard the period of the calling history.

Psql database permission problem

Privileges have been granted to the user in SQL script, however, permission is still denied.

Parameter choices

Now I have done a function on loading configs and model, and I am working on parameters setting, so what parameters should we choose for model training? And how to make it work well with model configuration and analysis dashboard?

Multiple database

Now we successfully connect to the remote database and deployed on Heroku.

The next step is to setup a local database for test purpose.

Production infrastructure

setup production environment and script/bootstrap.

Roadmap: long term plan

Test case

Write new test cases on our functions, include but not limit to:
data fetch and other

Add a data pipeline

Add the Spark data pipeline for transferring the data from database to model for training and prediction

Configure graph feature

Will need to figure out how to change the configs (xaxis, yaxis, title) of the existing graphs. Maybe try rerendering the graph created at the specific timestamp (with a specific webpage coord).

clarify how each component coordinate

Create state machine about how each component of our software work together.

log out button needed

I did not find log out button in the design page, and I think we'd better to have it.

Model configuration page

We need to add a page for parameter customization, I have enabled this in my train function implementation.

Read in data does not include name and number.

Read in data is not raw data, instead, it's processed data. This should not be the case. Data entry should be altered to read raw data that includes names and numbers. Following that would be some updates for the data pipeline.

Complete the login and registration system

Design frontend and database (backend) for this functionality

Roadmap: short term plan

Feed the actual calling history data from jaws db

Need to establish data transmitting pipeline to feed actual calling history data from the jaws db.

• First need to test on django mysql api to directly feed them in
• Second try implementing the Spark middleware to feed fine-processed data
• Further need to investigate how data should be processed for each axis, like aggregation (sum, count, max/min, etc) mechanism and algebraic preprocessing.

Logging system

We adopted "logDNA" add-on on Heroku for logging system of our APP. We need to spend time on learning how to use it.

Useful links:
https://elements.heroku.com/addons#logging
official doc for logdna: https://devcenter.heroku.com/articles/logdna

prototype

Please update our prototype at: https://www.figma.com/file/iB1guAmoGctNuU5yupB3KZ/491-prototype1?node-id=0%3A1

Connect with database

Now we are trying to connect our data entry to database, so we can actually store something in the database.

Strange Syntax error when creating database in psql

Syntax error when using backslash for any commands such as "\gexec"

Model test

TODO for now

1. add model test function
2. Read local preprocessed data directly for training or testing

Data pipe line design

Which part of the data should be processed and what kind of raw data should the app take in. In other words, how flexible does the app need to be?

suggestion about data storage system

It is better to enable each user to set their own data storage system.

Can we post our setup commands on the readme page?

So everyone can follow the command and setup locally.

Add ML model

Add the machine learning model (XGBoost) for customer segmentation
Train and fine-tune the model

Add the horizontal scrollbar and frame border

• Want to include combos inside each section as a frame with a right border, and all combo cards can only be seen within this frame
• Want to add a horizontal scrollbar under each section

Unable to save model

Need to achieve showing list of graphs but they need to be saved first. But Django complained about missing tables on either sqlite3 or jawsdb. Got to come up with ways to bypass it or create tables for them.

Css issue and styling the graph config pop-out window

• Css changes are not applied to my side, not sure why but other teammates could try on their end.
• Need to spend some time making the graph config pop-out more stylish by playing with css.

Unable to properly place the delete and grouping buttons on the campaign combo card

Intend to place the grouping/delete buttons on the right-most/left-most of the combo card header area, and make them float in decent margins, but failed.

Heroku database charge

We may need to pay for larger storage and more queries allowed per hour.

Hiding overflow middle layer corners when it is in absolute position

The outer wrapper has rounded corners and need the middle graph layer corners to be hidden, but it doesn't work when it has absolute position.

Transfer to Heroku

In order to run on Heroku, we have to transfer to PostgreSQL from MySQL, we have to do the following:

1. Redeploy CI for PSQL
2. Test migration
3. Reconnect Heroku to psql (I delete psql connection before)

Deploy MySQL CI

Delete graph feature

Need to investigate how graph deletion works, especially retriving the plotly graph object id from the template and removing it from the graph object collection.

Some workflows have no defined timeouts

This repo has workflows that have no defined timeouts.

GitHub Actions gives the DCSIL organization up to 3000 minutes per month. To ensure a single build doesn't spend hours eating up those minutes, we limit each job to 20 minutes.
Please add a timeout to your jobs. We will disable your actions until you add them.

Docs: https://docs.github.com/en/actions/learn-github-actions/workflow-syntax-for-github-actions#jobsjob_idtimeout-minutes
Example:

jobs:
  build:
    runs-on: ubuntu-latest
    timeout-minutes: 20 # Add this line here

Affected Workflows:

Workflow Path	Job	Re-enable
.github/workflows/CI-test.yml	tests	https://github.com/dcsil/jerryrat/actions/workflows/CI-test.yml

How to use model

1. periodically train the model
2. how to make prediction on runtime with our model?