radusuciu / ctesi

Automated processing of chemoproteomics data

Ctesi

Ctesi is a project meant to simplify processing and quantification of chemoproteomic mass spectrometry data in the Cravatt lab. The following are automated:

• Conversion of .raw files to .ms2 and mzXML formats
• Submission to an instance of IP2 maintained at The Scripps Research Institute
• Quantification of data with in-house software (cimage)

Dev setup notes

Ctesi requires a copy of the cimage-minimal project, to be placed alongside it.

On development instances you must create a file config/secrets.override.yml. This will by default not be entered into version control. It must contain credentials for accessing Active Directory. There is an example file in config/secrets.yml.

You must also mount an NFS share that is accessible by a cravatt-rawprocessor (not included here) instance. An example mount commmand is included in config/mount-nfs.example.sh. This command must be issued with root privileges.

ctesi's Issues

Experiments start getting processed even if uploads fail

Verify that experiments with the same name can be resolved

This may require changes to ip2api.. can either give searches unique names (under the same experiment name on ip2), add a unique identifier to the experiment name, or store search ids in the experiment table.

Difficulty downloading large datasets

Large datasets seem to time out as the zip operation takes too long.

Timeout when uploading larger files

Better handling of cancelling/removing runs

Provide temporary tokens for zip file downloads without username and pass

Propagate experiment deletion to ip2

Add option to send email on completion of processing

Experiments sometimes require hitting the remove button twice to actually be removed

Do not auth against AD unless local auth fails

Conversion to mzXML sometimes fails partially

Make better use of flask_principal

Would be nicer than a lot of this:

experiment = api.Experiment.query.get(experiment_id) 
if experiment.user_id == int(current_user.get_id()) or current_user.has_role('admin'):
   // something that requires permissions

Re-run individual parts of pipeline

Probably would be best to break up tasks into smaller chunks. Should also consider allowing tasks to just retry.

"Remember me" functionality for ip2

Need to modify ip2api to add a method to login via a stored cookie. On page load check if cookie still works and if not, require that the user passes their username and password once again.

Do not refresh status page if it is currently not in view

Do not error out experiment if email notification fails to send.

Should just retry sending the email with increasing cooldown times instead.

Allow retry for individual file uploads

ip2 credentials sometimes go stale transparently

Though there's a check for the presence of credentials in the session prior to rendering of the main page, these credentials aren't actually checked. The end result is sometimes the input boxes for these credentials pop up after the user has submitted.

Perhaps checking credentials before rendering every request is a bit much, but perhaps the experience of them being rejected/stale can be made a bit less jarring.

Maintain diff mod name when using preset list

Status page sometimes loads empty

Database lock error when updating experiment status

Task ctesi.core.tasks.convert_task[f17e0507-8f93-4405-a099-ddac4fe54a69] raised unexpected: OperationalError('(sqlite3.OperationalError) database is locked',)
Traceback (most recent call last):
  File "/home/ctesi/ctesi/venv/lib/python3.5/site-packages/sqlalchemy/engine/base.py", line 724, in _commit_impl
    self.engine.dialect.do_commit(self.connection)
  File "/home/ctesi/ctesi/venv/lib/python3.5/site-packages/sqlalchemy/engine/default.py", line 462, in do_commit
    dbapi_connection.commit()
sqlite3.OperationalError: database is locked

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/home/ctesi/ctesi/venv/lib/python3.5/site-packages/celery/app/trace.py", line 382, in trace_task
    R = retval = fun(*args, **kwargs)
  File "/home/ctesi/ctesi/venv/lib/python3.5/site-packages/celery/app/trace.py", line 641, in __protected_call__
    return self.run(*args, **kwargs)
  File "/home/ctesi/ctesi/ctesi/core/tasks.py", line 83, in convert_task
    status_callback=lambda s: update_status(experiment_id, 'converting', meta=s),
  File "/home/ctesi/ctesi/ctesi/core/convert.py", line 22, in convert
    status_callback(status)
  File "/home/ctesi/ctesi/ctesi/core/tasks.py", line 83, in <lambda>
    status_callback=lambda s: update_status(experiment_id, 'converting', meta=s),
  File "/home/ctesi/ctesi/ctesi/api/__init__.py", line 71, in update_experiment_status
    db.session.commit()
  File "/home/ctesi/ctesi/venv/lib/python3.5/site-packages/sqlalchemy/orm/scoping.py", line 153, in do
    return getattr(self.registry(), name)(*args, **kwargs)
  File "/home/ctesi/ctesi/venv/lib/python3.5/site-packages/sqlalchemy/orm/session.py", line 943, in commit
    self.transaction.commit()
  File "/home/ctesi/ctesi/venv/lib/python3.5/site-packages/sqlalchemy/orm/session.py", line 471, in commit
    t[1].commit()
  File "/home/ctesi/ctesi/venv/lib/python3.5/site-packages/sqlalchemy/engine/base.py", line 1643, in commit
    self._do_commit()
  File "/home/ctesi/ctesi/venv/lib/python3.5/site-packages/sqlalchemy/engine/base.py", line 1674, in _do_commit
    self.connection._commit_impl()
  File "/home/ctesi/ctesi/venv/lib/python3.5/site-packages/sqlalchemy/engine/base.py", line 726, in _commit_impl
    self._handle_dbapi_exception(e, None, None, None, None)
  File "/home/ctesi/ctesi/venv/lib/python3.5/site-packages/sqlalchemy/engine/base.py", line 1413, in _handle_dbapi_exception
    exc_info
  File "/home/ctesi/ctesi/venv/lib/python3.5/site-packages/sqlalchemy/util/compat.py", line 265, in raise_from_cause
    reraise(type(exception), exception, tb=exc_tb, cause=cause)
  File "/home/ctesi/ctesi/venv/lib/python3.5/site-packages/sqlalchemy/util/compat.py", line 248, in reraise
    raise value.with_traceback(tb)
  File "/home/ctesi/ctesi/venv/lib/python3.5/site-packages/sqlalchemy/engine/base.py", line 724, in _commit_impl
    self.engine.dialect.do_commit(self.connection)
  File "/home/ctesi/ctesi/venv/lib/python3.5/site-packages/sqlalchemy/engine/default.py", line 462, in do_commit
    dbapi_connection.commit()
sqlalchemy.exc.OperationalError: (sqlite3.OperationalError) database is locked (Background on this error at: http://sqlalche.me/e/e3q8)

Check how 403 Unauthorized responses are handled throughout the app

Recently switched to 'nice' html error pages. There may be api consumers that expect a json response with an error property.

Individual file uploads should go green as they return successfully

Do not allow multiple submits of the same experiment

Some sort of admin view

tmp folder not getting cleared can cause issues

Should probably just clear the folder when the app starts.

Allow file upload directly from cravault

Retry retrieving of dta-select following ip2 job finish

Verify IP2 credentials before allowing user to upload files

Respect user choices for ip2 login memory and email notifications

Possible issues with this after implementing batched file upload.

Make diff mod symbol assignments transparent to user

Otherwise if multiple diff-mods are entered it may be unclear to user which mass was assigned to which symbol

Create separate celery queues

Cues that require little processing such as IP2 submission should be allowed higher concurrency than intense cues like running cimage.

Output processing actions to text file for later review

Provide more options for search customization

For example:

• Max number of internal diff mods