Rewrite export_dag.py to use PythonOperator instead of BashOperator and update README about ethereum-etl-airflow

[evgeniuz]

I've set up basic structure using PythonOperator, proceeding with moving all exports to python. It's tricky to use click.Commands that are exported from ethereumetl.cli: they are not callable in itself (pallets/click#40), which requires either setting up context manually and invoking it or using CliRunner from testing utilities. Both approaches are rather unreliable and/or complicated, so I'm going to use instead undecorated functions which can be accessed using command.callback. This will disable click parameter validation, but overall should be more reliable way to run jobs.

[gitcoinbot]

Issue Status: 1. Open 2. Started 3. Submitted 4. Done

---

This issue now has a funding of 300.0 DAI (300.0 USD @ $1.0/DAI) attached to it as part of the Ethereum Foundation fund.

• If you would like to work on this issue you can 'start work' on the Gitcoin Issue Details page.
• Want to chip in? Add your own contribution here.
• Questions? Checkout Gitcoin Help or the Gitcoin Slack
• $34,861.24 more funded OSS Work available on the Gitcoin Issue Explorer

[gitcoinbot]

Issue Status: 1. Open 2. Started 3. Submitted 4. Done

---

Work has been started.

These users each claimed they can complete the work by 1 week, 6 days ago.
Please review their action plans below:

1) evgeniuz has been approved to start work.

Will reimplement using PythonOperator, add logging and documentation and test it.

Learn more on the Gitcoin Issue Details page.

[medvedev1088]

@evgeniuz send me an email to [email protected] from your Google email. I will give you access to GCP project.

[gitcoinbot]

@evgeniuz Hello from Gitcoin Core - are you still working on this issue? Please submit a WIP PR or comment back within the next 3 days or you will be removed from this ticket and it will be returned to an ‘Open’ status. Please let us know if you have questions!

• reminder (3 days)
• escalation to mods (6 days)

Funders only: Snooze warnings for 1 day | 3 days | 5 days | 10 days | 100 days

[gitcoinbot]

@evgeniuz Hello from Gitcoin Core - are you still working on this issue? Please submit a WIP PR or comment back within the next 3 days or you will be removed from this ticket and it will be returned to an ‘Open’ status. Please let us know if you have questions!

• reminder (3 days)
• escalation to mods (6 days)

Funders only: Snooze warnings for 1 day | 3 days | 5 days | 10 days | 100 days

[gitcoinbot]

Issue Status: 1. Open 2. Started 3. Submitted 4. Done

---

@evgeniuz due to inactivity, we have escalated this issue to Gitcoin's moderation team. Let us know if you believe this has been done in error!

• reminder (3 days)
• escalation to mods (6 days)

Funders only: Snooze warnings for 1 day | 3 days | 5 days | 10 days | 100 days

[evgeniuz]

Reimplemented according to specs. Couple of notes:

• since ethereumetl uses default logger no logging configuration is needed: default logger is captured by airflow;
• only data folder is mounted from bucket, all others are discarded when task ends, so using /tmp/ for temporary files is ok (no need to do manual cleanup);
• in the PR there are few TODOs that make DAG run only once and export output into export-dev folder instead of dev - these are for easier debugging and to avoid overwriting export folder until changes are complete.

Steps left to do:

• update README;
• test load_dag.py.

[evgeniuz]

@medvedev1088 Fixed minor issue with template formatting in load_dag, will update README, but otherwise ready for review.

[gitcoinbot]

Issue Status: 1. Open 2. Started 3. Submitted 4. Done

---

Work for 300.0 DAI (300.0 USD @ $1.0/DAI) has been submitted by:

1. @evgeniuz

@ceresstation please take a look at the submitted work:

• PR by @evgeniuz

---

• Learn more on the Gitcoin Issue Details page
• Want to chip in? Add your own contribution here.
• Questions? Checkout Gitcoin Help or the Gitcoin Slack
• $44,518.51 more funded OSS Work available on the Gitcoin Issue Explorer

[gitcoinbot]

⚡️ A *Polyglot Programmer* Kudos has been sent to @evgeniuz for this issue from @ceresstation. ⚡️ The sender had the following public comments: Awesome work as always evgeniuz! Nice work @evgeniuz! Your Kudos has automatically been sent in the ETH address we have on file.

[gitcoinbot]

Issue Status: 1. Open 2. Started 3. Submitted 4. Done

---

The funding of 300.0 DAI (300.0 USD @ $1.0/DAI) attached to this issue has been approved & issued to @evgeniuz.

• Learn more on the Gitcoin Issue Details page
• Questions? Checkout Gitcoin Help or the Gitcoin Slack
• $44,114.00 more funded OSS Work available on the Gitcoin Issue Explorer