alexhan46 / studyfind-ai-web-crawler

AI Web Crawler 1.0

AI Web Crawler is a python tool that downloads data about available research studies, formats it, and uploads the data to a database.

Coded in Python v3.8.5

Relesae Notes

v1.0.0 (20/11/2020)

New Features

• #7 #8 #9 #15 #25 Download research studies by crawling clinicaltrials.gov=
• #6 #26 Schedule crawling tasks to run on a recurring basis
• #27 #33 Use NLP to create a brief summary and a list of keywords for each crawler
• Automatically upload the data to the specified Firebase database
• #34 Use multithreading for superior performance (up to 7 studies per sec)

Bug Fixes

• The crawler didn't run if there was http request errors(fixed)

Installation Guide

Requirements

• Python 3 (recommended 3.8.5)
• Pip (included with Python 3)
• The Firebase JSON provided by the development team

Installation

1. Download the code using git or straight from GitHub
2. To install dependencies, execute the following command where the code was downloaded

   pip install -r requirements.txt

3. Run the following commands to install other required dependencies

  python -m nltk.downloader stopwords
  python -m nltk.downloader universal_tagset
  python -m spacy download en

4. Place the Firebase JSON into the same folder

Usage

There are 2 ways to execute the crawler

A. Using the admin panel to schedule recurring crawls

python manage.py runserver

To use the admin panel, you must be an authorized user, with access to the login information

b. Manually executing the crawler

python crawler.py

Known Limitations

• When updating studies that have already been crawled, there is a limitation placed by clinicaltrials.gov, causing some studies to be left out. In our testing we have never gotten close to this limit as long as the crawler is executed daily.

Contributors ✨

Meet our team mates

Adeeb Zaman

Heejoo Cho

Jonathon Sisson

Juntae Kim

Alex Han

studyfind-ai-web-crawler's Issues

Test pysummarization library

Test Gensim TextRank library

Test and evaluate rake-nltk python library

Create crawling engine

Lay foundation for the rest of the tasks involving the crawler

Finalize architectural design

The team will work together to go over and finalize the architectural design of the product.

Test polyglot library

Create Admin Panel

Create admin panel using Django which allows user to run and schedule the crawler, as well as configure the websites.

Get complete list of updated studies

Implement and trial NLP library

test

Implement and trial text summarization NLP algorithm

Get ID numbers of studies

Find a way to get a list of ID numbers for all new studies from clinicaltrials,gov

Test spaCy library

Test TextBlob library

Implement better error handling

Test pke library

Manual testing

Evaluate pyate NLP library

scraping only newly added studies without searching previously one repeatedly

Download and format data

Given an ID number of a study, download the data from clinicaltrials.gov, and format/prune the data to fit the client's specifications

Connect crawler to admin panel

Test sumy python library

test issue

create boilerplates for our project

Research libraries

Since the team is fairly new to NLP, we will spend some time researching appropriate NLP and web crawling libraries so we can implement those features in later sprints.

Create readme file to have installation guide

Implement unit tests

Create database and export data

Create database and upload formatted data to the database.

Test nltk library

Implement multithreading