josep-pujol / pdm

To build containers do:

1. docker-compose up airflow-init
2. docker-compose up

pdm's Issues

Remove SQLAlchemy from `psql_api.py` completely

Remove the use of SQLAlchemy from the psql_api module completely as it is not required at all. psycopg2 is more than capable of handling anything we need to do and will also be more efficient.

Move defaults from ENV Variables into the INIT method instead of in the parameters of INIT

The defaults would be safer to be in the body of the init function as one line IF statements.

Break `get_tweets` into smaller functions

There is currently too many things being done by the get_tweets method of the TwitterClient class. Break it into smaller methods each focusing on only one thing.

E.g.

1. Fetch tweets and return list of them
2. Parse tweets if a parse is specified or simply return them if one is not
3. Method that builds the set of tweet ids and returns them

Check that catching too generic an exception `except Execption` reraises errors otherwise pipelines will fail silently

There are numerous places where exceptions that are too broad are being captured. It prints an error message but the exception is handled and thus the Airflow pipeline will simply run as if everything is normal.

Remove these try catches so that the exceptions raised fail the python code and are logged in the Airflow DAG as an error with the stack trace or reraise them if you wanted to add your own message to the error.

Logic can be simplified to only check for twttest that do not have their tweet_id in the set

Remove Psql `insert_jason` method and replace it with a more generic method such as `insert_rows`

Remove the insert_json method and replace it with a method that is more generic and which can take a list of dictionaries of all the rows and insert it in bulk.

Remove unused methods on TwitterClient class

The clean_tweet_text method does not seem to be used in the TwitterClient class. Either it should be used or it should be removed.

Method inside a method not required

There is no need to have a method inside the get_tweets method

We could simply use the if statement instead.