pjhale / agr

Source code for the Alliance of Genome Resources web portal

Alliance of Genome Resources (AGR)

This software is written to support the goals of Alliance of Genome Resources

Getting Started

These instructions will get you a copy of the project up and running on your local machine for development and testing. This includes both the default configuration and instructions on how to customize your environment. This configuration allows user to have flexible development environment.

Contents

• Prerequisites
• Installing
• Configuration
  • Webapp
  • API
  • Indexer
  • ElasticSearch
• Running the Development Enviroment
  • Webapp
  • API
  • Indexer
  • ElasticSearch
• Running the Production Enviroment
• API Usage
• Running the tests

Prerequisites

• NodeJS
• Python 2.7
• Python Package Index (pip)
• Virtualenv
• Elasticsearch

  Starting with Python 2.7.9, pip is included by default with the Python binary installers. 
  
  make sure the executables npm, node, pip, and virtualenv are accessible from /usr/local/bin  (MAC, Linux)
  if not create symbolic links as needed
  
  Make sure /usr/local/bin is in your PATH (MAC, Linux)

Create a virtualenv for isolating the python dependencies:

	> mkdir -p ~/.virtualenvs/agr
	> # The prototype currently requires Python2
	> # Assuming virtualenv and python2 are in your PATH
	> virtualenv -p python2 ~/.virtualenvs/agr

• Webapp requires (NodeJS)
• API requires (python, pip, virtualenv, Elasticsearch)
• Indexer requires (python, pip, virtualenv, Elasticsearch)

Local ES hosted INDEX see elasticsearch setup for more info

Installing

To run a full install of all sub directories run the following:

	> git clone https://github.com/alliance-genome/agr.git
	> cd agr
	agr> source ~/.virtualenvs/agr/bin/activate
	(agr) agr> make -C webapp install
	(agr) agr> make -C webapp build
	(agr) agr> make -C api install
	(agr) agr> make -C indexer install

Configuration

Webapp

• API_URL A url pointing to the server that is hosting the /api endpoint for the frontend
• DEV_SERVER_UI_PORT Used to specify the local port that the WEBPACK server will be listening on

These parameters will default to host: "localhost" and port "2992". If you want to override them, you can do so through setting the environment variables. One such configuration a UI developer might be want to use is something along the lines of:

	(agr) agr> cd webapp
	(agr) agr/webapp> export API_URL=http://dev.alliancegenome.org
	(agr) agr/webapp> export DEV_SERVER_UI_PORT=12345

After running this set of commands the webpack-dev-server will be hosting the website at the URL http://localhost:12345 and will be pulling data from the API running on dev.alliancegenome.org.

API

The following variables can be used to customize your development environment:

• "PRODUCTION" Controls the start up of the flask server. If set to "false", debugging will be turned on and the server will crash upon first encountered error. If set to "true" the server runs the WSGIServer.
• "ES_AWS" controls whether or not SSL will be used on the connection to ES_HOST.
• "ES_HOST" points the flask server to the running Elastic Search instance.
• "ES_INDEX" indicates the index the flask server will be using to get its data.
• "API_PASSWORD" Not used at the moment, but will be the password controlling access to the writable API's

	(agr) agr> cd api
	(agr) agr/api> export PRODUCTION=true
	(agr) agr/api> export ES_AWS=true
	(agr) agr/api> export ES_HOST="search-es1-oyqxarxm2djn35dfodzniituhe.us-west-2.es.amazonaws.com"
	(agr) agr/api> export ES_INDEX=es_username
	(agr) agr/api> export API_PASSWORD="api_password"

The defaults for these variables are set to:

• PRODUCTION = false
• ES_AWS = false
• ES_HOST = "http://localhost:9200"
• ES_INDEX = searchable_items_blue
• API_PASSWORD = "api_password"

Indexer

• ES_AWS controls whether or not SSL will be used on the connection to ES_HOST.
• ES_HOST points the indexer to the running ElasticSearch instance.
• ES_INDEX indicates which index the indexer will be using to put documents into.

	(agr) agr> cd indexer
	(agr) agr/indexer> export ES_AWS=true
	(agr) agr/indexer> export ES_HOST="search-es1-oyqxarxm2djn35dfodzniituhe.us-west-2.es.amazonaws.com"
	(agr) agr/indexer> export ES_INDEX=es_username

The defaults for these params assume ElasticSearch is running on "localhost".

ElasticSearch

There is no configuration for ElasticSearch please see the elasticsearch setup for more info

Running the Development Enviroment

All of the following steps are not necessary, if developing only one part of the system. The frontend (webapp) can be pointed to a different API server and the webpack dev server will run on its own without the API, indexer, or ES running.

Simularly, if only the API needs to be developed then one can start the API and set the ES_HOST variable to the location of an instance that already has data and there is no need to run the webapp, indexer, or ElasticSearch instance.

If developing the indexer, go through the local setup of the ElasticSearch and point the indexer to run against the ElasticSearch server running on localhost, without having to run the API or webapp.

Webapp

In a seperate terminal window run the following commands:

	(agr) agr> make -C webapp run
	npm start

	> [email protected] start /Users/oblod/git/agr/webapp
	> webpack-dev-server --history-api-fallback --hot --inline

	[HPM] Proxy created: /api  ->  http://localhost:5000
	 http://localhost:2992/
	webpack result is served from /assets/
	content is served from dist
	404s will fallback to /index.html
	The react/require-extension rule is deprecated. Please use the import/extensions rule from eslint-plugin-import instead.
	The react/wrap-multilines rule is deprecated. Please use the react/jsx-wrap-multilines rule instead.

Now the server is running on port 2992 and is redirecting "/api" with proxy to localhost:5000 which should be hosted by the flask server.

API

In a seperate terminal window run the following set of commands:

	(agr) agr> make -C api run
	PRODUCTION= API_PASSWORD='api_password' ES_HOST='127.0.0.1:9200' ES_AWS= ES_INDEX='searchable_items_blue' python src/server.py
	 * Running on http://127.0.0.1:5000/ (Press CTRL+C to quit)
	 * Restarting with stat
	 * Debugger is active!
	 * Debugger pin code: 326-018-460

The flask server is now running on localhost:5000, serving /api requests.

Indexer

In a seperate terminal window run the following commands:

	(agr) agr> make -C indexer index
	cd src && ES_HOST='127.0.0.1:9200' ES_AWS= ES_INDEX='searchable_items_blue' python fetch_save_index.py
	ES_HOST: 127.0.0.1:9200
	ES_INDEX: searchable_items_blue
	ES_AWS: 
	...
	...
	...
	>

After the indexer has run, the Elastic search instance at localhost:9200/searchable_items_blue will be loaded with data that is now available to the API.

ElasticSearch

For running a local ElasticSearch instance see the elasticsearch setup for more info.

Running the Production Enviroment

In a production style running the application. Skip the section for running the webapp and use nginx to server the webapp/dict folder as the root of the site. The nginx configuration is as follows:

server {
        listen 80;
        root webapp/dist;
        index index.html;
        server_name hostname.alliancegenome.org;
        error_page 404 =200 /index.html;
        location /api {
                 proxy_set_header Host $host;
                 proxy_set_header X-Real-IP $remote_addr;
                 proxy_pass http://localhost:5000;
        }
}

Setup the hostname to whatever hostname you are wanting to use. Error page is set to allways return the index.html page and a 200 status. Change the /api url to point to the running flask server.

API Usage

Once you have the API up and running, with it pointed at a valid ES instance that has all loaded data, you should be able to run the following command and get the same results.

	> curl http://dev.alliancegenome.org/api/gene/MGI:98341
	{
		"category": "gene",
		"crossReferences": [ { ... }],
		"dataProvider": "MGI",
		"dateProduced": "2017-02-10T08:53:14-05:00",
		"description": null,
		...
		"geneLiteratureUrl": "http://www.informatics.jax.org/reference/marker/MGI:98341?typeFilter=Literature",
		"geneSynopsis": null,
		"geneSynopsisUrl": "https://en.wikipedia.org/wiki/SnRNP70",
		"gene_biological_process": [ ... ],
		"soTermName": "protein_coding_gene",
		"species": "Mus musculus",
		"symbol": "Snrnp70",
		"taxonId": "10090"
	}

Plans to make the API documentation available at /api.

Running the tests

The first three examples of running tests, run tests on the code in the directory that is being tested. The "api tests", does integration tests across the system (webapp, api, elasticsearch)

	(agr) agr> make -C webapp test
	(agr) agr> make -C api test-py
	(agr) agr> make -C indexer test
	# If the full system is up and running then the following can be run
	(agr) agr> make -C api tests