vishwajeet-hogale / multilex-workflows

Workflows using luigi

Luigi Workflows for Multilex

Commands

• luigid --port = 8082 : Runs a luigi vizualizer server on port number 8082
• python -m luigi --module [file_name] [ClassName-Task] --local-scheduler
  • file_name : Enter the name of the file without .py extension
  • ClassName : Enter the name of the class

Initial setup

• git clone [url of repo]
• pip install -r requirements.txt
• morning.bat
• evening.bat
• Paste final xlsx file into Output folder
• night.bat

Workflows

• LoggingWf : S1 data workflow
• SecS1WF : Union of two daily log reports and automated mail

News assigner + data upload

• form
• take final report as input
• file will be sent to the backend
• read this file, and upload to database
• upload to drive

deals with deep learning

• train
• predicts

multilex-workflows's Issues

Correct navigable string is removing a few rows

The except block which has
err.append(i) in correctnavigablestring function in scraper.py

Try to understand why it is being excluded and what has to be done to include them as well

Bert model to classify if the article is an ipo article or not

Setting up a dashboard for all news sources from the log files

Filter based on links: remove junk links

Use of power bi to do the data analysis

Script to add brief log file to the table created

Improve FIlterfunction

Create a logging table in the database

Run pylint and fix scraper.py and improve code quality

Analyse the news sources. Update with new words