one-data-cookie / mds4all-superset

Minimalistic and free Modern Data Stack, hence for all. Visualisation layer.

This part of mds4all is archived for now as it is built on Heroku who cancelled their free tier in 2022.

---

mds4all

Minimalistic and free modern data stack, hence for all.

• one-data-cookie/mds4all-elt:

  • Warehousing: Google BigQuery
  • Orchestrating: GitHub Actions
  • Ingesting: Meltano
  • Transforming: dbt
  • Cataloging: dbt docs on GitHub Pages

• one-data-cookie/mds4all-superset:

  • Visualising: Superset on Heroku

• other:

  • Code-editing: VS Code with:
    • BigQuery Runner
    • dbt Power User
    • dbt Language Server

mds4all-superset

Visualisation layer of mds4all.

Setup

# Set up basics
$ pip install apache-superset
$ pip install psycopg2
$ pip install pybigquery # for BigQuery connection
$ pip freeze > requirements.txt # or just limit to installed above
$ echo "web: superset db upgrade" > Procfile
$ echo "python-3.8.2" > runtime.txt
# add configs to superset_config.py

# Generate secret key
$ python
>>> from cryptography import fernet
>>> fernet.Fernet.generate_key()

# Set up Heroku app
$ heroku login
$ heroku create mds4all-superset
$ heroku config:set SECRET_KEY=<SECRET_KEY>
$ heroku config:set PORT=<PORT>
$ heroku addons:create heroku-postgresql:hobby-dev
$ git push heroku main
$ heroku logs –-tail # check for errors

# Initialise Superset
$ heroku run bash
$ export FLASK_APP=superset
$ superset fab create-admin
$ superset init
$ exit

# Finalise app
$ echo 'web: gunicorn "superset.app:create_app()"' > Procfile
$ git push heroku main
$ heroku logs --tail # check for errors

# Open app
$ heroku open

Limitations

• Sleep after 30 mins of inactivity
• 512 MB RAM
• 4 workers
• Gunicorn server
• No cache (like Redis)
• Database storage with 10K rows and 20 MB

Inspiration

• Chizzy-codes/NBA_Dashboard
• dugjason/superset-on-heroku
• zi-nt/superset-on-heroku