cmbi / whynot2 Goto Github PK

View Code? Open in Web Editor NEW

0.0 8.0 2.0 26.63 MB

CSS 5.33% Python 63.37% HTML 30.20% Shell 0.11% Dockerfile 0.99%

whynot2's Introduction

Requirements

a mongodb server: https://www.mongodb.com/
python pip: https://pypi.org/project/pip/
cmbi databanks installed locally: https://github.com/cmbi/databanks.git

installing

run: pip install -r requirements:
set the passwords and hostnames for the database in "whynot_web/default_settings.py" and "update_settings.py" call 'storage.authenticate' too if the database is password protected.
run install.py

Usage

Use ./crawl.py to update the database with new entries. Use ./annotate.py to check for missing entries and annotate a reason why it is missing. Run ./run_site.sh to run the whynot website.

whynot2's People

Contributors

Watchers

Forkers

jonblack cbaakman

whynot2's Issues

Improve docker use

Update the dockerfiles and docker-compose files to match hope, also utilising the scripts for running a development and test environment. This means:

Use depends_on intead of links in docker-compose file
Add a dev docker-compose file and run_dev.sh script
Add a run_tests.sh script
Put the command in the docker-compose file

When a comment string is too long to fit in the table, whynot does not add any newlines. Instead, the text just intersects with other text outside the table. Example "BUSTER: probably full/mixed anisotropic refinement. Not all B-factors could be reproduced from ANISOU records" on this page: http://www.cmbi.ru.nl/WHY_NOT2/search/pdbid/4diw

statistics page shows wrong information

Several data on the page http://www.cmbi.ru.nl/WHY_NOT2/statistics/ are wrong.

Most recent annotations shows a list of nulls
Total number of entries is 248083, too low
Total number of entries equals total number of annotations
Most recent files list is empty

No lists of entries are generated

The request http://www.cmbi.ru.nl/WHY_NOT2/resources/list/PDB_REDO_PRESENT returns an empty file instead of a list of existing entries. On the website, there is no (obvious) way to generate lists of existing or missing entries.

Crawl failing

The script crawl.py on chelonium fails to update the whynot mongo database. Presumably the port number has changed and the mongo database runs in a docker container now.

resources only generated at whynot-web startup

Some downloadable resources only get created when the whynot web application starts. Like: http://www.cmbi.ru.nl/WHY_NOT2/resources/list/BDB_VALID

Those resources are generated here: https://github.com/cmbi/whynot/blob/3bc307ebd54c5b81adf75a2569b22cbdabfd0786/whynot-web/src/main/java/nl/ru/cmbi/whynot/databank/ListInitializer.java#L46

This means that those resources for newly added databanks, don't show up until the web-application is restarted.

Individual data files for PDBFINDER

Single data files are available for PDBFINDER entries. Now the PDBFIND.TXT.gz file containing all entries is provided (e.g. on the search results page), this should be changed into the individual data file.

Search pdbid does not work.

A URL like http://www.cmbi.ru.nl/WHY_NOT2/search/pdbid/1crn used to search for all entries related to 1crn. Now it results in "No value specified for parameter pdbid".

Coding standards

Use pep8 and flake8 to make the code confirm to coding standards.

Remove dsspcmbi

The repository has the executable scripts/dsspcmbi which is used to determine why a DSSP or DSSP_REDO file doesn't exist.

Executables shouldn't be included in version control. If whynot depends on an external program, we should be that into the deployment environment.

What source was used to compile this executable?

Write unit and integration tests

There are currently zero tests for whynot. This is not good as it means we have little trust in the code, and changes can introduce regressions without detection.

Database username and password are hardcoded

The username and password for the mongo database are hardcoded. This is insecure, especially in a public repository.

When is an entry considered missing?

Currently an entry is considered missing if it doesn't exist in a databank but does exist in its parent. For example, if there is no entry in HSSP for 1crn but there is an entry for 1crn in its parent, DSSP, then 1crn is missing from HSSP

This is, however, only one level deep. What happens if the entry exists in MMCIF but in neither DSSP nor HSSP? The above definition labels it missing in DSSP but says nothing about HSSP. I assume it inherits the missing state from the parent in this case?

It can get even more complicated. What happens if an entry is in MMCIF but annotated in DSSP? I assume in this case the HSSP entry inherits the annotation of the entry in DSSP?

@drlemmus, @cbaakman

Make circle diagrams clickable again

In the previous version of why_not frontend, a page for listing entries was shown whenever the user clicked for a group of entries. Like: dssp, present, giving all the present entries in the dssp databank.

These selections were available by clicking either the circle diagram itself or by clicking the names. (present, valid, annotated, etc.)

It would be usefull to have this option back, so that I can see which entries in the databanks still need annotation/build.

Show pie chart as two concentric circles

Show the pie chart as two concentric circles where:

Center shows missing and present
Outer shows annotated and unannotated aligned with the inner missing segment, and valid and obsolete aligned with the inner present segment.

See issue #34.

This isn't possible with chart.js but can be done with d3.js.

loading image never goes away

For the whynot website, an AjaxLazyLoadPanel is used to load the pie chart. A small animated image is displayed while the chart is being loaded. However, this image doesn't go away once the pie chart is there.

mongo

A new branch was recently created: https://github.com/cmbi/whynot/tree/mongo

It's an experiment where postgresql is replaced by mongodb and java is replaced by python.

An advantage is that changes to the database are faster, sessions don't expire and some webpages load faster.
However, the restful services and pages that process large quantities of data, take more time.

Maybe python/mongodb is just too slow or maybe there are still some tricks possible to speed it up.

Replace scripts/dsspcmbi

The executable scripts/dsspcmbi is a copy of and unknown version of mkdssp that outputs more useful information for why not. Currently it's unknown where the source code for this variant is.

Ideally we want to include the changes in https://github.com/cmbi/xssp as an option. At the moment, any changes to xssp will not be reflected in the executable being used here.

Uncomment doesn't remove comments

When I wanted to replace a comment for 50 entries by a different comment, I put these 50 entries in a whynot syntaxed textfile in the uncomment directory (for the comment to replace) and the comment directory (the new comment to give them).

The annotater placed the new comments, however the old ones were not removed. Though the annotater reported it so.

Comments page is empty

The page at http://www.cmbi.ru.nl/WHY_NOT2/comment/ doesn't show any comments. It should give a list of all comments in the mongo database, the number of entries per comment and the most recent annotation.

How to comment entries missing in PDB?

It's possible for there to be an entry in mmCIF but not in the PDB. These will show up as UNANNOTATED. Should/Can we comment these?

Exception: no such databank: PDB_REDO

The following error appears in the logs:

[2016-12-01 07:06:54,761] ERROR in app: Exception on /resources/list/PDB_REDO_UNANNOTATED/ [GET]
Traceback (most recent call last):
  File "/usr/local/lib/python3.5/site-packages/flask/app.py", line 1988, in wsgi_app
    response = self.full_dispatch_request()
  File "/usr/local/lib/python3.5/site-packages/flask/app.py", line 1641, in full_dispatch_request
    rv = self.handle_user_exception(e)
  File "/usr/local/lib/python3.5/site-packages/flask/app.py", line 1544, in handle_user_exception
    reraise(exc_type, exc_value, tb)
  File "/usr/local/lib/python3.5/site-packages/flask/_compat.py", line 33, in reraise
    raise value
  File "/usr/local/lib/python3.5/site-packages/flask/app.py", line 1639, in full_dispatch_request
    rv = self.dispatch_request()
  File "/usr/local/lib/python3.5/site-packages/flask/app.py", line 1625, in dispatch_request
    return self.view_functions[rule.endpoint](**req.view_args)
  File "/usr/src/app/whynot/frontend/routes.py", line 137, in resources
    for entry in get_entries_from_collection(databank_name, collection):
  File "/usr/src/app/utils.py", line 196, in get_entries_from_collection
    return get_unannotated_entries(databank_name)
  File "/usr/src/app/utils.py", line 400, in get_unannotated_entries
    for entry in get_missing_entries(databank_name):
  File "/usr/src/app/utils.py", line 366, in get_missing_entries
    raise Exception("no such databank: " + databank_name)
Exception: no such databank: PDB_REDO

Annotations cannot be accessed

On the website, the overall list of comments is empty. For individual databanks there is no link to get all the annotations. Request through the API such as http://www.cmbi.ru.nl/WHY_NOT2/webservice/rs/annotations/PDB_REDO/1abc give an error 404.

Comment subcategories

WHY NOT comment sub-categories would improve the user experience in case categories are present that can initially be grouped under the same message without seeing the accession codes (until the user clicks on the message for a subdivision and the accession codes).
Example: experimental method in case X-ray is needed (NMR, EM, SAXS etc is more informative than 'NOT X-RAY').
Initial view:

Experimental method (n)

Secondary view:

Experimental method: NMR (n)
1aaa 1aab 1aac 1aad ...
...

Experimental method: EM
2aaa 2aab 2aac 2aad ...
...

etc.

Another option would be to 'fold' large categories by default (>500 entries for example)

incomplete comment files

When the comment file for annotated BDB entries is requested. The user recieves an incomplete file from download.

annotater exits with null pointer exception

Yesterday, I ran the annotater twice. (master branch, 2.3) In both cases it ended like this:

15:44:56,888  INFO Annotater:33 - Annotater start.
15:44:58,355  INFO Converter:133 - Optimizing file /srv/scratch/whynot2/comment/fraud_150818.txt
15:44:58,399  INFO Annotater:84 - Adding annotations in fraud_150818.txt.optimized
15:44:58,399  INFO Annotater:128 - COMMENT: Empty comment: Adding 0 annotations
15:44:59,837  WARN Annotater:108 - Skipping annotation for mmCIF,1g40: No present parent
15:44:59,837  WARN Annotater:108 - Skipping annotation for mmCIF,1g40: No present parent
15:45:01,218  WARN Annotater:108 - Skipping annotation for mmCIF,1g44: No present parent
15:45:01,218  WARN Annotater:108 - Skipping annotation for mmCIF,1g44: No present parent
15:45:02,605  WARN Annotater:108 - Skipping annotation for mmCIF,1l6l: No present parent
15:45:02,605  WARN Annotater:108 - Skipping annotation for mmCIF,1l6l: No present parent
15:45:03,983  WARN Annotater:108 - Skipping annotation for mmCIF,1rid: No present parent
15:45:03,983  WARN Annotater:108 - Skipping annotation for mmCIF,1rid: No present parent
15:45:05,376  WARN Annotater:108 - Skipping annotation for mmCIF,1y8e: No present parent
15:45:05,376  WARN Annotater:108 - Skipping annotation for mmCIF,1y8e: No present parent
15:45:06,741  WARN Annotater:108 - Skipping annotation for mmCIF,2a01: No present parent
15:45:06,741  WARN Annotater:108 - Skipping annotation for mmCIF,2a01: No present parent
15:45:08,117  WARN Annotater:108 - Skipping annotation for mmCIF,2hr0: No present parent
15:45:08,117  WARN Annotater:108 - Skipping annotation for mmCIF,2hr0: No present parent
15:45:09,499  WARN Annotater:108 - Skipping annotation for mmCIF,2ou1: No present parent
15:45:09,499  WARN Annotater:108 - Skipping annotation for mmCIF,2ou1: No present parent
15:45:09,500  INFO Annotater:138 - COMMENT: Structure is fraud: Adding 0 annotations
15:45:09,507  INFO Converter:133 - Optimizing file /srv/scratch/whynot2/comment/wi-list-pdb-150818.txt
15:45:09,511  INFO Annotater:84 - Adding annotations in wi-list-pdb-150818.txt.optimized
15:45:09,511  INFO Annotater:128 - COMMENT: Empty comment: Adding 0 annotations
15:45:24,597  INFO Annotater:128 - COMMENT: Not enough residues: Adding 7 annotations
15:46:40,379  INFO Annotater:138 - COMMENT: Too many residues: Adding 37 annotations
15:46:42,549  INFO Converter:133 - Optimizing file /srv/scratch/whynot2/comment/wi-list-redo-150818.txt
15:46:42,574  INFO Annotater:84 - Adding annotations in wi-list-redo-150818.txt.optimized
15:46:42,574  INFO Annotater:128 - COMMENT: Empty comment: Adding 0 annotations
15:46:49,699  INFO Annotater:128 - COMMENT: Too many C-alpha-only residues: Adding 1 annotations
15:46:51,287  INFO Annotater:128 - COMMENT: Bioinformatics unfriendly aspects: Adding 2 annotations
15:47:05,111  INFO Annotater:128 - COMMENT: Percentage bad residues too high: Adding 2 annotations
15:47:40,374  INFO Annotater:128 - COMMENT: Too many bad residues: Adding 13 annotations
15:49:59,483  INFO Annotater:128 - COMMENT: Too many residues: Adding 91 annotations
15:56:22,793  INFO Annotater:138 - COMMENT: Not enough residues: Adding 253 annotations
15:56:24,906  INFO Converter:133 - Optimizing file /srv/scratch/whynot2/uncomment/scenes-pdb-ss2_20150818.txt
15:56:24,915  INFO Annotater:147 - Removing annotations in scenes-pdb-ss2_20150818.txt.optimized
15:56:24,915  INFO Annotater:197 - COMMENT: Empty comment: Removing 0 annotations
15:56:25,382  WARN Annotater:171 - Skipping annotation for PDB_SCENES_ss2,4r78: Entry not found
15:56:25,382  WARN Annotater:171 - Skipping annotation for PDB_SCENES_ss2,4r78: Entry not found
15:56:25,382  WARN Annotater:171 - Skipping annotation for PDB_SCENES_ss2,4rir: Entry not found
15:56:25,382  WARN Annotater:171 - Skipping annotation for PDB_SCENES_ss2,4rir: Entry not found
15:56:25,383  WARN Annotater:171 - Skipping annotation for PDB_SCENES_ss2,4tkd: Entry not found
15:56:25,383  WARN Annotater:171 - Skipping annotation for PDB_SCENES_ss2,4tkd: Entry not found
15:56:25,383  WARN Annotater:171 - Skipping annotation for PDB_SCENES_ss2,4wig: Entry not found
15:56:25,383  WARN Annotater:171 - Skipping annotation for PDB_SCENES_ss2,4wig: Entry not found
15:56:25,383  WARN Annotater:171 - Skipping annotation for PDB_SCENES_ss2,4wtp: Entry not found
15:56:25,383  WARN Annotater:171 - Skipping annotation for PDB_SCENES_ss2,4wtp: Entry not found
15:56:25,383  WARN Annotater:171 - Skipping annotation for PDB_SCENES_ss2,4wtr: Entry not found
15:56:25,383  WARN Annotater:171 - Skipping annotation for PDB_SCENES_ss2,4wtr: Entry not found
15:56:25,383  WARN Annotater:171 - Skipping annotation for PDB_SCENES_ss2,4wts: Entry not found
15:56:25,383  WARN Annotater:171 - Skipping annotation for PDB_SCENES_ss2,4wts: Entry not found
15:56:25,383  WARN Annotater:171 - Skipping annotation for PDB_SCENES_ss2,4z5q: Entry not found
15:56:25,383  WARN Annotater:171 - Skipping annotation for PDB_SCENES_ss2,4z5q: Entry not found
15:56:25,383  WARN Annotater:171 - Skipping annotation for PDB_SCENES_ss2,5ccj: Entry not found
15:56:25,383  WARN Annotater:171 - Skipping annotation for PDB_SCENES_ss2,5ccj: Entry not found
15:56:25,383  WARN Annotater:171 - Skipping annotation for PDB_SCENES_ss2,5cvi: Entry not found
15:56:25,383  WARN Annotater:171 - Skipping annotation for PDB_SCENES_ss2,5cvi: Entry not found
15:56:25,383  WARN Annotater:171 - Skipping annotation for PDB_SCENES_ss2,5cwa: Entry not found
15:56:25,383  WARN Annotater:171 - Skipping annotation for PDB_SCENES_ss2,5cwa: Entry not found
15:56:25,383  WARN Annotater:171 - Skipping annotation for PDB_SCENES_ss2,5czg: Entry not found
15:56:25,383  WARN Annotater:171 - Skipping annotation for PDB_SCENES_ss2,5czg: Entry not found
15:56:25,383  WARN Annotater:171 - Skipping annotation for PDB_SCENES_ss2,5d1j: Entry not found
15:56:25,383  WARN Annotater:171 - Skipping annotation for PDB_SCENES_ss2,5d1j: Entry not found
15:56:25,384  INFO Annotater:216 - COMMENT: Error creating YASARA scene: Removing 0 annotations
15:56:25,560  INFO Converter:133 - Optimizing file /srv/scratch/whynot2/uncomment/scenes-redo-ss2_20150818.txt
15:56:25,614  INFO Annotater:147 - Removing annotations in scenes-redo-ss2_20150818.txt.optimized
15:56:25,615  INFO Annotater:197 - COMMENT: Empty comment: Removing 0 annotations
15:56:25,955  WARN Annotater:171 - Skipping annotation for REDO_SCENES_ss2,4wtp: Entry not found
15:56:25,955  WARN Annotater:171 - Skipping annotation for REDO_SCENES_ss2,4wtp: Entry not found
15:56:25,955  INFO Annotater:216 - COMMENT: Error creating YASARA scene: Removing 0 annotations
15:56:25,961  INFO Converter:133 - Optimizing file /srv/scratch/whynot2/uncomment/scenes-pdb-iod_20150818.txt
15:56:25,982  INFO Annotater:147 - Removing annotations in scenes-pdb-iod_20150818.txt.optimized
15:56:25,983  INFO Annotater:197 - COMMENT: Empty comment: Removing 0 annotations
15:56:26,042  WARN Annotater:171 - Skipping annotation for PDB_SCENES_iod,5c5p: Entry not found
15:56:26,042  WARN Annotater:171 - Skipping annotation for PDB_SCENES_iod,5c5p: Entry not found
15:56:26,042  WARN Annotater:171 - Skipping annotation for PDB_SCENES_iod,5cv7: Entry not found
15:56:26,042  WARN Annotater:171 - Skipping annotation for PDB_SCENES_iod,5cv7: Entry not found
15:56:26,042  WARN Annotater:171 - Skipping annotation for PDB_SCENES_iod,5cv8: Entry not found
15:56:26,042  WARN Annotater:171 - Skipping annotation for PDB_SCENES_iod,5cv8: Entry not found
15:56:26,042  INFO Annotater:216 - COMMENT: Error creating YASARA scene: Removing 0 annotations
15:56:26,164 ERROR Annotater:60 - File uncomment/wi-list-redo-iod_20150722.txt is empty and should probably be removed: Skipping it for now.. 
15:56:26,164 ERROR Annotater:60 - File uncomment/wi-list-redo-iod_20150722.txt is empty and should probably be removed: Skipping it for now.. 
15:56:39,615  WARN LoadContexts:132 - fail-safe cleanup (collections) : org.hibernate.engine.loading.CollectionLoadContext@5b79b2c6<rs=com.mchange.v2.c3p0.impl.NewProxyResultSet@7acfb741>
15:56:39,615  WARN LoadContexts:132 - fail-safe cleanup (collections) : org.hibernate.engine.loading.CollectionLoadContext@5b79b2c6<rs=com.mchange.v2.c3p0.impl.NewProxyResultSet@7acfb741>
Exception in thread "main" java.lang.NullPointerException
        at java.lang.String.compareTo(String.java:1139)
        at nl.ru.cmbi.whynot.model.Databank.compareTo(Databank.java:102)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:606)
        at org.hibernate.proxy.pojo.javassist.JavassistLazyInitializer.invoke(JavassistLazyInitializer.java:197)
        at nl.ru.cmbi.whynot.model.Databank_$$_javassist_1.compareTo(Databank_$$_javassist_1.java)
        at nl.ru.cmbi.whynot.model.Entry.compareTo(Entry.java:65)
        at nl.ru.cmbi.whynot.model.Annotation.compareTo(Annotation.java:54)
        at nl.ru.cmbi.whynot.model.Annotation.compareTo(Annotation.java:17)
        at java.util.TreeMap.compare(TreeMap.java:1188)
        at java.util.TreeMap.put(TreeMap.java:531)
        at java.util.TreeSet.add(TreeSet.java:255)
        at java.util.AbstractCollection.addAll(AbstractCollection.java:342)
        at java.util.TreeSet.addAll(TreeSet.java:312)
        at org.hibernate.collection.PersistentSet.endRead(PersistentSet.java:352)
        at org.hibernate.engine.loading.CollectionLoadContext.endLoadingCollection(CollectionLoadContext.java:261)
        at org.hibernate.engine.loading.CollectionLoadContext.endLoadingCollections(CollectionLoadContext.java:246)
        at org.hibernate.engine.loading.CollectionLoadContext.endLoadingCollections(CollectionLoadContext.java:219)
        at org.hibernate.loader.Loader.endCollectionLoad(Loader.java:1005)
        at org.hibernate.loader.Loader.initializeEntitiesAndCollections(Loader.java:993)
        at org.hibernate.loader.Loader.doQuery(Loader.java:857)
        at org.hibernate.loader.Loader.doQueryAndInitializeNonLazyCollections(Loader.java:274)
        at org.hibernate.loader.Loader.loadCollection(Loader.java:2166)
        at org.hibernate.loader.collection.CollectionLoader.initialize(CollectionLoader.java:62)
        at org.hibernate.persister.collection.AbstractCollectionPersister.initialize(AbstractCollectionPersister.java:627)
        at org.hibernate.event.def.DefaultInitializeCollectionEventListener.onInitializeCollection(DefaultInitializeCollectionEventListener.java:83)
        at org.hibernate.impl.SessionImpl.initializeCollection(SessionImpl.java:1863)
        at org.hibernate.collection.AbstractPersistentCollection.initialize(AbstractPersistentCollection.java:369)
        at org.hibernate.collection.AbstractPersistentCollection.read(AbstractPersistentCollection.java:111)
        at org.hibernate.collection.AbstractPersistentCollection.readSize(AbstractPersistentCollection.java:134)
        at org.hibernate.collection.PersistentSet.isEmpty(PersistentSet.java:169)
        at nl.ru.cmbi.whynot.annotate.Annotater.removeUnusedComments(Annotater.java:228)
        at nl.ru.cmbi.whynot.annotate.Annotater$$FastClassByCGLIB$$f5d09685.invoke(<generated>)
        at net.sf.cglib.proxy.MethodProxy.invoke(MethodProxy.java:204)
        at org.springframework.aop.framework.Cglib2AopProxy$CglibMethodInvocation.invokeJoinpoint(Cglib2AopProxy.java:689)
        at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:150)
        at org.springframework.transaction.interceptor.TransactionInterceptor.invoke(TransactionInterceptor.java:110)
        at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:172)
        at org.springframework.aop.framework.Cglib2AopProxy$DynamicAdvisedInterceptor.intercept(Cglib2AopProxy.java:622)
        at nl.ru.cmbi.whynot.annotate.Annotater$$EnhancerByCGLIB$$1e1627a1.removeUnusedComments(<generated>)
        at nl.ru.cmbi.whynot.annotate.Annotater.main(Annotater.java:64)
Command exited with non-zero status 1

My suspicion is that is occurs here:
https://github.com/cmbi/whynot/blob/master/whynot-core/src/main/java/nl/ru/cmbi/whynot/model/Databank.java#L102

It's odd, since the databank.name and entry.databank_id fields have been configured NOT NULL in psql.

Replace top_highest with mongo query

The top_highest class is a list wrapper that keeps track of the top n entries. Not only is it poorly implemented (inefficient, not pythonic, doesn't implement the iterator interface), it's also unnecessary: the same functionality can be provided by sorting and limiting in the mongo query.

Circle diagrams make no sense

In the current implementation the circle diagram show two overlapping categories (unannotated and valid) as different sections. Just to give the hierarchy of categories:
Possible entries = missing + present
Missing = annotated + unannotated
Present = valid + obsolete

If you want to make all categories clickable, you can make two concentric circle diagrams with missing + present in the centre and with annotated + unannotated + valid + obsolete in the outer ring, aligned with the correct centre categories.

[EPIC] Clarify project boundaries

Features

There are two distinct features of this project:

Serve HTML over The Internet so users can see which files are missing, and their reason, from various databanks
Update the mongo database when databanks change. This is mostly controlled via the databanks scripts which we store in another project.

The problems with this are:

This project is not responsible for updating its own data.
The frontend, frontend and databanks script (indirectly) all access the same database. This increases coupling and could make data integrity issues difficult to find.

Ideas

Split this project into the following separate services:

frontend: only responsible for serving HTML to view data. It doesn't talk to the database, but instead to a backend REST server, to get data that it needs to display.
backend: provides a REST API allowing access to whynot data, and runs a separate background Process to update the database. The background process should run independently of the databanks script mentioned above.

When thinking about the backend I wondered if it should be two separate services itself, where one just serves a REST API and the other updates the database. The problem with this is that you then have two services that operate on the same database. Perhaps it's not so bad since one reads and the other writes. I think it's still something to consider.

Languages like go and rust (although its http library is not stable as far as I know) would be a great option for the backend, particularly the crawling and annotating part. Frameworks like EmberJS would be great for the frontend.

This issue is a big rewrite so don't run off with it blindly. It needs some serious though.

The same downloadable files have different filenames

In the 'pie chart view', the "unannotated" list of id's has filename WHATIF_PDB_acc_UNANNOTATED is, while in the 'unannotated view' the identical file has filename PDBACCWHATIFunannotated_pdbids.txt

Don't change sys.path

Changing sys.path is a code smell. Instead of changing sys.path, the code should be better organised into packages.

structure factors missing

Whenever a structure_factors file is not deposited, it gets the status "unannotated" in whynot. It should actually say: "not deposited"

Send email when exception not caught

When an exception occurs in both the website and the api, it should be caught and logged, resulting in an email being sent.

Use package namespaces in imports

The project doesn't use a root package from which all imports are made.

crawler re-adds all BDB entries

When running, the crawler first removes and then adds all the present bdb entries. A time consuming process. The timestamp check at https://github.com/cmbi/whynot/blob/master/whynot-apps/src/main/java/nl/ru/cmbi/whynot/crawl/Crawler.java#L73 is possibly the cause.

Move updating to celery

Updating the whynot database should run as a periodic task in celery. This means the following scripts:

annotate.py
crawl.py
install.py

Not all databanks are indexed (correctly)

A number of databanks do not have any statistics (e.g. HSSP and PDB_REDO). The counts for other databanks (e.g. DSSP and STRUCTURFACTORS) are incorrect: missing entries are not listed.

PDB_REDO comments not indexed

On behalf of Robbie Joosten:

Some PDB_REDO WHY_NOT files are not indexed (properly) nor stored in the database anymore (this used to work fine).
Example: Calpha-only structures, e.g. 3tb8.

PDB_REDO now tries to do these files over and over again.

annotations: gets the list of annotations for the given pdb id
entries: gets all entries in a given databank and state (PRESENT, VALID, etc)

There are two generic use cases for the API:

Allow other services to update whynot when an entry is missing by providing the annotation. This would primarily be used by the databank update scripts. It has the knock on effect that whynot wouldn't need to crawl and annotate itself anymore, which performs poorly and often lags behind the real state of the databanks.
Allow humans to manually update entries. Currently this is done by placing files in a comment and uncomment folder on the server, which is not ideal.

I also think this is a good opportunity to standardise the API using a hypermedia format like JSON-API and consolidate the API endpoints.

To Do

Consider security implications. How exposed is the API?
Propose endpoints