The purpose of the project was to gain business intelligence and bolster profit margins through intelligence gained from large scale opinion mining also known as sentiment analysis. To solve a growing need to understand the increasing demands that “big data” trends present. The client uses sentiment analysis models on a regular basis and would like to gain actionable insights regarding data trends. Visualisations are presented to gather information quickly and efficiently on sentiment classification, these are portable visualisations represented in a web application format. In addition, the calibrated models will perform sentiment analysis without labelled data.
The project uses the Sci-kit learn machine learning library to address the research question: what the best performing machine learning classifier and evaluating it with the best suitable method. Experiments with different classifiers identified a Stochastic Gradient Descent iterative learner that regularises the best performing linear model which was Logistic Regression as the most suitable model for sentiment analysis, in agreement with literature. The project is then implemented within a traditional software setting – a web application.
Many skills were adopted on the project some of which encompass; Literature review skills, Sci-learn, Pandas data structures, metrics to validate a classifier, Machine learning technicalities and vocabulary, Heuristic evaluations, Iterative learning model application architecture, Automation skills, Unit/integration testing, Database design, the Django web framework, python data structures, Pagination with search features, JavaScript and Chartjs.
The final project relied on drawing together concepts from machine learning, natural language processing and applying them to a software product. The results and outcomes of the experiment and were a highly accurate, precise machine learning classifier that enables simple and interpretable visualisations of results in a user-friendly display validated through client feedback, heuristic evaluations and user questionnaires. Future investigation would include real time feedback on sentiment this would cascade change to the training architecture. Deep Neural Network investigations for model improvement would also be considered.
python3 -m pip install --upgrade pip
Recommended: python3 -m venv /path/to/new/virtual/environment, source env/bin/active
pip install -r requirements.txt --upgrade
=== UBUNTU / DEBIAN ===
sudo apt-get install python3-tk
Data can be downloaded from https://nijianmo.github.io/amazon/index.html
Run the pp.sh
script
-
sudo apt update
-
sudo apt install postgresql postgresql-contrib
-
download pgadmin4 - https://www.pgadmin.org/download/pgadmin-4-python/
-
pip install psycopg2
-
Change the following config files; pg_hba.conf, postgresql.conf.
-
To find;
-
$ psql -U postgres
postgres=# SHOW config_file;
-
Alternatively:
$(ls /etc/postgresql/*/main/pg_hba.conf)
$(ls /etc/postgresql/*/main/postgresql.conf)
-
postgresql.conf: add this line listen_addresses = '*'
-
In pg_hba.conf - first line should go:
#Database administrative
host all postgres 127.0.0.1/32 trust -
Also add this below
#TYPE DATABASE USER CIDR-ADDRESS METHOD
host all all 0.0.0.0/0 md5
-
sudo -u postgres -i
## log into postgres user -
postgres=# systemctl start postgresql
-
And
postgres=# systemctl stop postgresql
postgres=# systemctl restart postgresql
. To stop and restart the database. -
Start pgadmin and go to the ip address to create database with the config settings defined in the config dictionary found in ./DBFuncs/dbConfig.py or configure your own!
- Run the
python3 create_database.py
found in the root directory.
- Run
python3 review_upload.py
found in the root directory. Reviews should now be in the db.
The algorithms used are based from a thorough Literature review.
- Run
classifier_main.py
found in ./app/ directory. This saves a model to file/database which can be used for iterative learning.
- To run the web application: python3 SentimentApp/manage.py runserver