refinery-platform / refinery-higlass-docker

"Refinery-ified" flavor of the higlass-docker (https://github.com/hms-dbmi/higlass-docker/) project

 Refinery Platform

• Additional information about how to administer and develop Refinery can be found in the wiki
• Production deployments require access to Amazon Web Services
• Refinery supports the latest version of Chrome (Linux and OS X), Firefox (Linux and OS X), and Safari (OS X)

Installing and Launching for Development

Prerequisites

• Install Git (2.19.0+), Vagrant (2.2.0+) and Virtualbox (5.2.20+)
• Add SSH key to your GitHub account
• Note: this procedure has only been tested on local development machines running OS X 10.10+

Configure and Load Virtual Machine

$ git clone [email protected]:refinery-platform/refinery-platform.git
$ cd refinery-platform
$ vagrant up

The above step should take about 15 minutes depending on the speed of your machine and Internet connection. If you get an error, simply retry by:

$ vagrant provision

Open http://192.168.50.50:8000/ in your web browser.

Configure Deployment Environment on the Host

Create a Python 2.7 virtual environment (optional but recommended, assumes virtualenvwrapper is installed):

$ mkvirtualenv -a $(pwd) refinery-deployment

Install deployment tools (assumes header files for Python are installed):

$ pip install -r deployment/requirements.txt

Install Pre-Commit Hooks

Use fabricrc.sample to update or initialize Fabric configuration, for example:

$ cp fabricrc.sample ~/.fabricrc

To pull the latest code and update Refinery installation:

$ fab vm update

Refinery Operations on the VM

Connect to the initialized VM:

$ vagrant ssh
$ workon refinery-platform
$ ./manage.py [command]

Log in to Refinery (http://192.168.50.50:8000/) with the default guest user account (username: guest, password: guest).

Log in to Django admin UI (http://192.168.50.50:8000/admin/) with the default superuser account (username: admin, password: refinery).

Please see installation notes for more details, including information on how to configure Galaxy for this setup.

Troubleshooting

• Refinery deployment requires a lot of external dependencies. You might have to run vagrant provision repeatedly to install all dependencies successfully. Any errors in the output of vagrant provision indicate that you have to re-run the command.
• If you run into a build error in OS X when trying to install Fabric: export C_INCLUDE_PATH=/usr/local/include
• If you have a VPN connection running, you may need to disconnect and reconnect before you can access the VM. In some cases you may have to reboot the host machine.
• To make sure all the required services are running after the VM was restarted or shut down, you need to provision again: vagrant reload --provision or vagrant up --provision

refinery-higlass-docker's Issues

Support BIGWIG track types

NIls made a point that this is something we want to support

Remove directories on host when a container is purged?

With the introduction of DockerContainerSpec.extra_directories, directories get created on the host system upon a run().

Containers may fill these up withs lots of data and they should probably be removed during a purge_by_label().

Use the newest higlass

• This project is pretty out of date, theres been a lot of development on higlass since I've worked on this initial implementation. It would be good to spend some time and harden this tool.

• Have some sort of automated build process that will create and tag new refinery-higlass-docker images when Higlass itself is updated.

Can't load tracks with with latest Chrome

Ref: higlass/higlass#287

HiGlass is calling the viewconf api relative to Refinery's base url

@pkerpedjiev Something in my container is hitting: /api/v1/viewconfs/?d=default but returning a 404 because it is trying to access this resource relative to Refinery's base url rather than going through our HTTPProxy.

This is w/ the latest higlass-docker image from today.

I get a 200 response by manually accessing it:
200 - http://192.168.50.50:8000/visualizations/TestLISTVisualizationHiGlass-launch-53599e6b-ff8a-4039-af24-7f695835fe5e/api/v1/viewconfs/?d=default

Console Error

hglib.js:1 XHR finished loading: GET "http://192.168.50.50:8000/api/v1/viewconfs/?d=default".
send @ hglib.js:1
get @ hglib.js:1
(anonymous) @ hglib.js:1
i @ hglib.js:12
Ct @ hg-launcher.js? [sm]:2155
(anonymous) @ hg-launcher.js? [sm]:2184
(anonymous) @ hg-launcher.js? [sm]:5
(anonymous) @ hg-launcher.js? [sm]:6

Let me know if you need anymore info and thanks for the help!

Wait somehow until default viewconf is available

http://192.168.50.50:8000/visualizations/HiGlass-39b4cd52-f64f-42ac-b11d-ea7eb76245a0/api/v1/viewconfs/?d=default will 500 sometimes because the backend isn't in a ready state.

We're already not starting nginx until tilesets have been ingested, so we could just check through Django ORM that the Default ViewConf is available as well

Start with bad data, never get 200

Refinery visualizations always need to return a 200 eventually, even if it's just to say there's an error.

When running it from my demo, with input a CSV:

,a,b,c
x,1,0,0
y,0,1,0
z,0,0,1

... I just get this error repeated in the log:

/usr/lib/python2.7/dist-packages/supervisor/options.py:296: UserWarning: Supervisord is running as root and it is searching for its configuration file in default locations (including its current working directory); you probably want to specify a "-c" argument specifying an absolute path to a configuration file for improved security.
  'Supervisord is running as root and it is searching '
2018-05-11 16:11:17,633 CRIT Supervisor running as root (no user in config file)
2018-05-11 16:11:17,637 INFO supervisord started with pid 7
2018-05-11 16:11:18,641 INFO spawned: 'uwsgi' with pid 10
2018-05-11 16:11:18,642 INFO spawned: 'on_startup' with pid 11
2018-05-11 16:11:19,645 INFO success: uwsgi entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
2018-05-11 16:11:19,645 INFO success: on_startup entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
2018-05-11 16:11:21,924 INFO exited: on_startup (exit status 1; not expected)
2018-05-11 16:11:22,928 INFO spawned: 'on_startup' with pid 19
2018-05-11 16:11:23,929 INFO success: on_startup entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
2018-05-11 16:11:24,939 INFO exited: on_startup (exit status 1; not expected)
2018-05-11 16:11:25,942 INFO spawned: 'on_startup' with pid 21
2018-05-11 16:11:26,946 INFO success: on_startup entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)

High dependance on file naming

See here: https://github.com/scottx611x/refinery-higlass-docker/blob/master/on_startup.py#L48

Files probably don't always need to have multires in their filename

Check if .hitile is being ingested the right way

me:

Also: I see .cool working, but not .hitile… Any reason it would be different? Or is this a bug? cf: #2358

pete:

Are you ingesting it with --filetype hitile --datatype vector?

html variable hides html module

    html = '''
                <html><head><title>Error</title></head><body>
                <p>An error has occurred: There may be a problem with the
                input provided. To report a bug, please copy this page and
                paste it in a <a href="{url}">bug report</a>. Thanks!</p>
                <pre>{stack}</pre>
                </body></html>'''.format(
        url='https://github.com/refinery-platform/'
            'refinery-higlass-docker/issues/new',
        stack=html.escape(error_str))

Note the reuse of html... this causes:

Traceback (most recent call last):
  File "higlass-server/on_startup.py", line 191, in <module>
    main()
  File "higlass-server/on_startup.py", line 161, in main
    error_page(e)
  File "higlass-server/on_startup.py", line 179, in error_page
    stack=html.escape(error_str))
UnboundLocalError: local variable 'html' referenced before assignment

dependencies incompatibility

$ pip install -r requirements-test.txt
...
docker 3.2.1 has requirement requests!=2.18.0,>=2.14.2, but you'll have requests 2.12.3 which is incompatible.
...

But I guess whatever the incompatibility is, it hasn't been causing us problems?

Friendlier on launch

Right now we just get an empty screen on launch... Even the grey control bar isn't there on first load.

Perhaps work with Pete and see if there could be a default to bring up a file selector onload if the UI is otherwise empty? Or load tracks as horizontals?

(Probably two distinct issues here: some kind of please-wait during ingest, and some sort of welcome-and-select files once you're in.)

Bed file / Annotation Data Support?

Nils mentioned this today

• Is this even an option in parent higlass?
• There appears to be annotation tracks from the public data source, but many other things that we don't want as well. We currently remove this here: https://github.com/refinery-platform/refinery-higlass-docker/blob/master/Dockerfile#L18

Get rid of `get_python_server_url()`

This method is pretty hard to come back and understand what its doing.

• Check out: the solution you have in docker_igv_js and do something like that 👍