thusharan / wordwranglers

A collaborative project on text mining undertaken as part of the post-graduate program.

How to work on this project?

Cloning the project

The project can be either downloaded as a zip file from the git repository WordWranglers or cloned using the following git command:

git clone https://github.com/ThusharaN/WordWranglers.git

The project can then be opened in Visual Studio Code or any other IDE.

Setting up the project

Install all the necessary dependencies

pip install -r requirements.txt

Do a test run to check if the environment has been setup correctly

python3 app/src/test_run.py

Note: Instructions for installing git can be found here. Most Mac/Linux machines will have git pre-installed.

Adding Dependencies

Dependencies can added to the current virtual environemt by running the following command

pip install <package_name> && pip freeze > requirements.txt

Making changes

A pull request needs to be created in order to push any changes to code.

After cloning, create a local branch to work on.

git checkout master
git checkout -b <your_branch_name>

In case a change has been made to hello.py, run the following commands to commit the changes

git status // Gives the paths of the files that have been updated
git add <file_path>/hello.py
git commit -m <some_commit_message>

In case new packages were installed, add the updated requirements.txt to the commit. Push your branch to GitHub

git push origin <your_branch_name>

Raise a pull request through GitHub. If the request shows conflicts, run the following commands locally

git fetch origin
git rebase origin/master

Fix all the conflicts in the files; add, commit and push.

Note: More git commands can be found here

Running the program

Since the classifier script resides within the src folder of the repository, make sure to move into this folder before running the train and/or test commands

cd WordWranglers/app/src

Run the following command in the console to train the model:

python3 question_classifier.py --train --config [config_file_name]

and the following command to test the model:

python3 question_classifier.py --test --config [config_file_name]

The placeholder [config_file_name] can be any of the following files to train and test the model with different configurations:

• bilstm_pretrained.yaml: Train the test classifier with pretrained word embeddings using GloVe with the sentence representation given by BiLSTM
• bilstm_random.yaml: Train the test classifier with randomly generated word embeddings with the sentence representation given by BiLSTM
• bow_pretrained.yaml: Train the test classifier with pretrained word embeddings using GloVe with the sentence representation given by Bag-Of-Words (or BoW)
• bow_random.yaml: Train the test classifier with randomly generated word embeddings with the sentence representation given by Bag-Of-Words (or BoW)
• ensemble.yaml: Train the test classifier with pretrained word embeddings using GloVe with the sentence representation given by an ensemble learner which utilisez both LSTM and Bag-Of-Words

Debugging the program

The program can be debugged using the VS Code debugger. The configurations have been added as part of the .vscode/launch.json file