vecnet / simulation-manager

VecNet Simulation Manager

Vecnet Metadata Catalog

This application provides the Vecnet Metadata Catalog. It handles the curation and indexing of the data generated by the Vecnet cyberinfrastructure.

Dependencies

• Fedora Commons 3.6
• Solr 4.3
• Redis (version?)
• Postgresql or other SQL database
• nginx
• chruby Ruby version manager

The SETUP file has detailed steps on installing the platform on a bare RHEL machine.

Deployment

First, your public ssh key needs to be put on the server. Ask Don to do this. To deploy to QA:

cap qa deploy

To deploy to Production:

cap production deploy

To deploy from branch

cap <environment> deploy -S branch=<branch name>

To deploy new nginx config. This will reload nginx.

cap <environment> vecnet:update_nginx_config

Other server admin tasks

To rebuild the Fedora object store:

sudo service tomcat6 stop
cd /opt/fedora/server/bin
sudo FEDORA_HOME=/opt/fedora CATALINA_HOME=/usr/share/tomcat6 ./fedora-rebuild.sh
# choose option 1 to rebuild the resource index
sudo FEDORA_HOME=/opt/fedora CATALINA_HOME=/usr/share/tomcat6 ./fedora-rebuild.sh
# choose option 2 to rebuild the SQL database
sudo service tomcat6 start

To resolarize everything...it will take a LONG time to complete.

chruby 2.0.0-p353
RAILS_ENV=qa bundle exec rake solrizer:fedora:solrize_objects

To load and build the MeSH trees run. This will run for a while (~0.5--1 hours)

chruby 2.0.0-p353
RAILS_ENV=qa bundle exec rake vecnet:import:mesh_subjects vecnet:import:eval_mesh_trees

To resolrize with mesh synonyms...it will take a LONG time to complete.

chruby 2.0.0-p353
# This builds the synonyms.txt file if needed.
# you could skip this if synonyms did not change
RAILS_ENV=qa bundle exec rake vecnet:solrize_synonym:get_synonyms FILE=solr_conf/conf/synonyms.txt
#copy this file to solr core
sudo  cp solr_conf/conf/synonyms.txt /opt/solr-4.3.0/vecnet/conf/synonyms.txt
#copy schema and solrconfig
sudo  cp solr_conf/conf/schema.xml /opt/solr-4.3.0/vecnet/conf/schema.xml
sudo  cp solr_conf/conf/solrconfig.xml /opt/solr-4.3.0/vecnet/conf/solrconfig.xml
#change owner to be tomcat
sudo chown tomcat:tomcat -R /opt/solr-4.3.0
#restart solr
sudo service tomcat6 restart
#resolrize all objects
RAILS_ENV=qa bundle exec rake solrizer:fedora:solrize_objects

To ingest Citation to qa/Production #Copy endnote file to file to /opt/endnote and make sure everyone can read sudo cp /from/path/to/endnote/file /opt/endnote sudo chmod -r 755 /opt/endnote #Copy pdf to /opt/citation_file/<createfolder_with_endnote_file_name> and make sure everyone can read sudo cp -r /from/path/to/endnote/pdf/* /opt/citation_file/ sudo chmod -r 755 /opt/citation_file/ #Execute citation task as app user sudo su app cd /home/app/vecnet/current chruby 2.0.0-p353 RAILS_ENV=production bundle exec rake vecnet:import:endnote_conversion ENDNOTE_FILE=/opt/endnote/ ENDNOTE_PDF_PATH=/opt/citation_files:/opt/citation_files/

Initializing new production environment

1. Do system setup as in SETUP file
2. Get capistrano deploy working to new site
3. on production machine:

• setup ruby: chruby 2.0.0-p353
• Setup mesh terms: RAILS_ENV=production bundle exec rake vecnet:import:mesh_subjects vecnet:import:eval_mesh_trees
• Migrate user table: See below
• Resolrize: RAILS_ENV=production bundle exec rake solrizer:fedora:solrize_objects
• Migrate fedora objects: RAILS_ENV=production bundle exec rake vecnet:migrate:batch_to_collection

7. Done!

NCBI Terminalogy

Work in progress. After running rake db:migrate the following task will download the NCBI taxonomy from the following location

ftp://ftp.ncbi.nih.gov/pub/taxonomy/taxdump.tar.gz

and ingest the terms into the database.

rake vecnet:import:ncbi_taxonomy

There are about 1,091,096 terms (November 2013).

Gather repository contents for statistics

OUTFILE=~/repo-stats-20130916.csv RAILS_ENV=production bundle exec rake vecnet:dump_statistics

Pubtkt Authentication

The site uses the pubtkt authentication scheme, which uses a signed cookie for every request. For development, a dummy login to create a pubtkt is provided (class DevelopmentSessions). But, first, a public/private key pair needs to be generated and installed.

rake pubtkt:generate_keys
mv pubtkt.pem config/pubtkt-development.pem
mv pubtkt-private.pem config/pubtkt-private-development.pem

And that should be enough for development. There are also utility rake tasks for creating and verifying tickets:

1. To create a ticket on the comand line:

   $ P_KEY=pubtkt-private.pem P_UID=dbrower P_VALIDUNTIL=3456789012 P_TOKENS='dl_librarian,dl_write' rake pubtkt:create uid=dbrower;validuntil=3456789012;tokens=dl_librarian,dl_write;sig=MCwCFHiaErA+7lHoHxbSUIZaSnmTovIPAhRf4RxtrmArBMD8CBnZaUM/yWI+Cw==

The valid until date above has the date July 16, 2079 in the Unix epoch, so the ticket should not expire while you are using it. 2. To validate tickets from the command line:

$ P_KEY=pubtkt-private.pem P_TICKET='uid=dbrower;validuntil=3456789012;tokens=dl_librarian,dl_write;sig=MCwCFF1/aaSbtrxN9PLrZE1XvLH5SIWQAhRXN8AHevzPMFbMuIIlOwuCLTZDPw==' rake pubtkt:verify
Ticket text: uid=dbrower;validuntil=3456789012;tokens=dl_librarian,dl_write
Ticket sig : MCwCFF1/aaSbtrxN9PLrZE1XvLH5SIWQAhRXN8AHevzPMFbMuIIlOwuCLTZDPw==
Sig Valid? : true
Expired?   : true

simulation-manager's Issues

Enhance GET for a simulation to include percent done

From redmine, ticket 6361

Revise the GET method of Simulation resource to accept query parameter (or some flag in the request's JSON body) that indicates that the % done is to be returned if simulation's status == running model.

Note: In the OpenMalaria Portal project, Greg D has written code to parse the model's standard output to get its current percent done. See the while loop in tasks.py module -- https://redmine.crc.nd.edu/redmine/projects/openmalaria/repository/entry/WorkerPackage/Task/tasks.py#L60

Deploy and configure on Windows HPC cluster

Moved from redmine, ticket 6368
To see if it'll run on Windows system.

Finish implementing the ability of a sim-group to have default output-URL

Moved from redmine, ticket 6367
This would enable each individual simulation to NOT have an output URL in the execution request. Currently, the same output URL is being used for all simulations.

A simulation's output URL would be filled out from the group's default output URL when the simulation description was written to the simulation's working directory. That way, the run_simulation.py script does not have to do any additional work, and can remain unchanged.

Add cancel functionality to the REST API

Moved from redmine, ticket 6364
How to implement as URL? Possible options are:

• PATCH with body = { action : cancel }
• POST to /api/v1/sim-groups/{id}/cancel/
• DELETE HTTP method
  ** This may not be appropriate because we don't want to delete the simulation or group resource. We just want to stop its execution.

Implement caching of input files

Transferred from redmine, ticket 5947

A cache for input files on the cluster is a performance optimization. (Add link to its Google design doc once it's created?)

Implement a GET method for simulation's output files

Moved from redmine, ticket 6366
Possible URL would refer to the simulation's output files as a collection of subordinate resources:

• GET /api/v1/simulations/1/output/

This could be used by VecNet CI to retrieve a simulation's output files if the attempt to POST them by Simulation Manager fails. It would be alternative to requesting the Simulation Manager to re-POST the files (issue #6365).

test

test body

Get the status of a simulation group

Moved from redmine, ticket 6362
Above a certain threshold (#), the status is represented as aggregate stats:

• Number of simulations submitted (in queue)
• Number of simulations running
• Number of simulations finished (successfully) - output received
• Number of simulations failed (error occurred)

Below the threshold, the status for each simulation in the group is returned:

• simulation's id_on_client : status code

If status code == running model, then also include % done.

Probably want an option (a flag in request body, or query parameter) to get individual statuses for all simulations in the group regardless of the threshold.

Null simulation group crashes the submit_group script

Transferred from redmine, ticket #6213

When the "simulation_group" key is an execution request is "null", the submit_group script crashes. The script assumes that the group will have a member called "simulations". So this exception is raised:

File ".../submit_group.py", line 76, in main for simulation in execution_request.simulation_group.simulations: AttributeError: 'NoneType' object has no attribute 'simulations'

Customize the administrative panel for groups and simulations

Moved from redmine, ticket 6371
Both list views for groups and simulations should show various useful columns (e.g., ids, status, etc.)

Move new Python packages out of this project

Moved from redmine, ticket 6527
There are 3 new Python packages under development in this project (source:/trunk/new-packages@394):

• crc_nd.django
• crc_nd.pyutils
• vecnet.simulation

The objective is to move these packages out of this project, and to establish each one as its own project, with its own repository and issue tracking system.

Investigate auto-generated documentation for REST API’s

Moved from redmine, ticket 6369
For example, using the "django-tastypie-swagger":https://pypi.python.org/pypi/django-tastypie-swagger Python package.

Add started_when and stopped_when date-time fields to Simulation data model

Moved from redmine, ticket 6363
These fields are needed by the VecNet CI to know when a simulation started, and how long it took to run. They should be assigned automatically when the simulation's status is updated with a PATCH request:

• When status becomes STARTED_SCRIPT, set the started_when field to the current time
• When status becomes SCRIPT_DONE or OUTPUT_ERROR or SCRIPT_ERROR, set the stopped_when to the current time

Fix Python 2.6 incompatibility with remote_tests

Transferred from redmine, #6344

The remote tests package has the following issues when run with Python 2.6:

When no module names are provided on the command line, the main module attempts test discovery. However, test discovery was introduced in Python 2.7
The TestCase.assertRegexEquals method was also introduced in Python 2.7
These should be fixed so the tests can run on Python 2.6

Implement "resend model output" to REST API

Moved from redmine, ticket 6365
How to implement as an URL? The options are:

• PATCH with body = { action : resend-output }
• POST to /api/v1/sim-groups/{id}/resend-output/

This request is only valid if the status is OUTPUT_ERROR. If the re-POSTing of the output files succeeds, then the simulation's status become SCRIPT_DONE.

Add links on the home page

Moved from redmine, ticket 6370
The home page should have links to:

• The Django admin page
• The API schema