centerforcollectivelearning / dive-backend Goto Github PK

Codebase for DIVE backend (server, worker, and ORM)

License: GNU General Public License v3.0

Python 94.58% Shell 0.06% HTML 5.36%

dive-backend's Introduction

DIVE Backend

The Data Integration and Visualization Engine (DIVE) is a platform for semi-automatically generating web-based, interactive visualizations of structured data sets. Data visualization is a useful method for understanding complex phenomena, communicating information, and informing inquiry. However, available tools for data visualization are difficult to learn and use, require a priori knowledge of what visualizations to create. See dive.media.mit.edu for more information.

Development setup involves the following steps:

Installing system dependencies
Setting up postgres
Setting up rabbitMQ
Starting and entering virtual environment
Installing python dependencies
Migrating database
Starting celery worker
Starting server

Install System Dependencies (Linux / apt)

$ sudo apt-get update && sudo apt-get install -y postgresql git python2.7 python-pip build-essential python-dev libpq-dev libssl-dev libffi-dev liblapack-dev gfortran libxml2-dev libxslt1-dev rabbitmq-server

Install System Dependencies (Mac / brew)

Install Homebrew if you don't already have it. Then, run the following code:

$ brew install postgres
$ brew install rabbitmq

OR Install postgres.app

Install postgres.app by following the instructions here: (http://postgresapp.com/).

Download and open the app to start postgres.

Setup postgres

Make sure that you have a postgres server instance running:

postgres -D /usr/local/pgsql/data >logfile 2>&1 &
sudo -u postgres -i

Create the dive database by running:

$ createuser admin -P
$ createdb dive -O admin

Start RabbitMQ AMQP Server

Add rabbitmq-server executable to path (add PATH=$PATH:/usr/local/sbin to ~/.bash_profile or ~/.profile)
Run the server as a background process sudo rabbitmq-server -detached
Create a RabbitMQ user and virtual host:

$ sudo rabbitmqctl add_user admin password
$ sudo rabbitmqctl add_vhost dive
$ sudo rabbitmqctl set_permissions -p dive admin ".*" ".*" ".*"

Install and Enter Virtual Python Environment

Installation: See this fine tutorial.
Starting virtual env: source venv/bin/activate.

Install Python Dependencies

Within a virtual environment, install dependencies in requirements.txt. But due to a dependency issue in numexpr, we need to install numpy first.

$ pip install -U numpy && pip install -r requirements.txt

Start Celery Worker

Start celery worker: ./run_worker.sh
Start celery monitor (flower): celery -A base.core flower

Database Migrations

Follow the docs. The first time, run the migration script.

python migrate.py db init

Then, review and edit the migration script. Finally, each time models are changed, run the following:

$ python migrate.py db migrate
$ python migrate.py db upgrade

Run API

To run development Flask server, run python run_server.py.
To run production Gunicorn server, run ./run_server.sh.

Deployment

Set environment variable before running any command:

$ source production_env

Building Docker Images

conda env export > environment.yml
conda env create -f environment.yml

conda list -e > conda-requirements.txt
conda create --name dive --file conda-requirements.txt

dive-backend's People

Contributors

Stargazers

Watchers

dive-backend's Issues

Missing dependencies

To name a few: pandas, flask, yaml, flask_compress, flask_cors, flask_login, flask_sqlalchemy etc.

Maybe this was never tried to be installed into a newly installed virtual machine like mine? I used an 18.04 LTS Ubuntu Server, completely fresh VM. If you need a more full list, I am happy to provide it, but then I would like you to promise that you'll merge it in :)

specs-api: /visualizations endpoint doesn't support histogram-type data

data-ingestion: save specs

Visualizations specs with multiple viz types are all the same type

specs-api: val:val specs always have q:q typeStructure regardless of actual type structure

specs-api: /visualizations endpoint doesn't support ind:val specs

Error saving document with extended ascii chars

If I try to save the a document with this title "Analizando la distancia y la duración de los viajes", I get the following error.

{"message": {"title": "'ascii' codec can't encode character u'\xf3' in position 35: ordinal not in range(128)"}}

I see the error in the browser console, it does not appear in the front end.

https://dive.media.mit.edu/exportedstory/1721

Website's registration broken

Both Firefox and Chrome complain about CORS and registration does not succeed. I have all blocking/adblocks/privacy badger/etc. turned off. Here is the Chrome console output:

register:1 Failed to decode downloaded font: https://d33wubrfki0l68.cloudfront.net/a571dd4a0c7e19690e66cf444b06be148f0f5ebb/b5b4a/assets/fonts/segoe-ui/segoeuisl.ttf
register:1 OTS parsing error: OS/2: Failed to parse table
register:1 Access to fetch at 'https://dive-api.media.mit.edu/auth/v1/anonymous_user' from origin 'https://dive.media.mit.edu' has been blocked by CORS policy: No 'Access-Control-Allow-Origin' header is present on the requested resource. If an opaque response serves your needs, set the request's mode to 'no-cors' to fetch the resource with CORS disabled.
dive-api.media.mit.edu/auth/v1/anonymous_user:1 Failed to load resource: net::ERR_FAILED
register:1 Uncaught (in promise) TypeError: Failed to fetch
register:1 Access to fetch at 'https://dive-api.media.mit.edu/auth/v1/register' from origin 'https://dive.media.mit.edu' has been blocked by CORS policy: Response to preflight request doesn't pass access control check: No 'Access-Control-Allow-Origin' header is present on the requested resource. If an opaque response serves your needs, set the request's mode to 'no-cors' to fetch the resource with CORS disabled.
dive-api.media.mit.edu/auth/v1/register:1 Failed to load resource: net::ERR_FAILED
bundle.js:1 Registration failed TypeError: Failed to fetch
register:1 Failed to decode downloaded font: https://d33wubrfki0l68.cloudfront.net/a571dd4a0c7e19690e66cf444b06be148f0f5ebb/b5b4a/assets/fonts/segoe-ui/segoeuisl.ttf
register:1 OTS parsing error: OS/2: Failed to parse table

unable to buld docker

Downloading https://files.pythonhosted.org/packages/7c/ef/79873517838910e8f1bd5610ab3de380a3886675103072727266eec91bbc/dbf-0.94.003.tar.gz (79kB)
ERROR: Command errored out with exit status 1:
command: /usr/local/bin/python -c 'import sys, setuptools, tokenize; sys.argv[0] = '"'"'/tmp/pip-install-9f0aorln/dbf/setup.py'"'"'; file='"'"'/tmp/pip-install-9f0aorln/dbf/setup.py'"'"';f=getattr(tokenize, '"'"'open'"'"', open)(file);code=f.read().replace('"'"'\r\n'"'"', '"'"'\n'"'"');f.close();exec(compile(code, file, '"'"'exec'"'"'))' egg_info --egg-base pip-egg-info
cwd: /tmp/pip-install-9f0aorln/dbf/
Complete output (8 lines):
Traceback (most recent call last):
File "", line 1, in
File "/tmp/pip-install-9f0aorln/dbf/setup.py", line 4, in
import dbf
File "/tmp/pip-install-9f0aorln/dbf/dbf.py", line 339
print args
^
SyntaxError: Missing parentheses in call to 'print'
----------------------------------------
ERROR: Command errored out with exit status 1: python setup.py egg_info Check the logs for full command output.
The command '/bin/sh -c pip install -r requirements.txt' returned a non-zero code: 1

HTTP500: SERVER ERROR - The server encountered an unexpected condition that prevented it from fulfilling the request. (Fetch) DELETE - https://dive-api.media.mit.edu/projects/v1/projects/1643

Is DIVE project active ??

I see not much movement in git repo for DIVE since June 2018.
Is the project in "Active" state ?

Also, how much data DIVE can handle for regression or correlation ??