wtsi-npg / npg_qc Goto Github PK

• very occasionally qc-ers would want to fail a library
• would allow for correct placement of uqc flag

the upstream_tags check (npg_qc/autoqc/checks/upstream_tags.pm) should use bambi instead of BamIndexDecoder for its decoding (see BamIndexDecoder_cmd attribute)

samtools_stats & sequence_summary

Create a light weight controller to provide data for what is qc-able in a page. It should work using the same/similar queries to those used for reporting current manual QC outcomes. Query the controller to take decisions about when to generate manual qc widgets in client side.

npg_qc::utils::bam_genotype uses samtools mpileup which apparently will soon be deprecated. Best practice is to use bcftools mpileup (http://www.htslib.org/workflow/).

typo on line 125 was not picked up in the tests

in returned hash in get_outcomes_as_hash(), lib/npg_qc/Schema/ResultSet/MqcLibraryOutcomeEnt.pm

Hash on either full rpt key or pt key

The Mqc controller have both get and post operations. Consider splitting into two. Consider consolidating all get operations in one controller (MqcRun) or create a family of controllers reporting the mqc state as json.

Is there any benefit in using these feeds as API by bam meta updater?

an option (runforder path glob, like /\gs\d+ or /\sf\d+), which the cron will use so only runs appropriate for the farm are considered

ability to run from interop files archiva - alternative runfolder root/path?

and lauches a job anyway. The lane is a one-sample pool

The upstream qc check uses illumina2bam BamIndexDecoder to be consistent with the main pipeline it should use bambi decode

to distinguish between oucomes created by

• the users via the web interface,
• one-off scripts as a part of some RT ticket,
• automatic evaluators.

Research task. There might be some scope to optimise queries.

example: run 15831 lane 1

This path doesn't need to be set - it's handled by Conda activation and will break if Conda is moved

run 17831 - qc in progress
qc-er is logged in
all lanes are passed according to the ml warehouse data
http://sf2-farm-srv1.internal.sanger.ac.uk:1969/checks/runs/17831 - widgets are not displayed, table cells with lane number have blue background (pass)
http://sf2-farm-srv1.internal.sanger.ac.uk:1969/checks/runs?db_lookup=1&show=all&run=17831&lane=2 - widgets are displayed (showing undecided), table cells with lane number have blue background (pass)

Error displayed: No LIMs data for run 17831 position 1 at /nfs/users/nfs_m/mg8/working/npg_qc/npg_qc_viewer/bin/../lib/npg_qc_viewer/Controller/MqcRun.pm line 94.

raised by fetch_tag_index_array_for_run_position() in lib/npg_qc_viewer/Model/MLWarehouseDB.pm

Could be implemented as in pbcal, ie the makefile creates version.h file that is then compiled into the code. Or copy the source file, change the version there as for perl modules and compile the copy.

This path doesn't need to be set - it's handled by Conda activation and will break if Conda is moved

Using $.is(':visible') has proven to be considerably slow when applied to a significant number of objects. Re-factor where possible as css rules so is not executed inside loops. We suspect it traverses the DOM each time it is executed to find if the element and its parents are part of subset of elements in the DOM (visible ones). An example of this situation is the code which generates d3 plots. The problem seems to be significantly worse with firefox.

should install locally test dependencies
should run javascript tests
shoudl be possible to run grunt tests as a part of 'tets' action for Build.PL
add running grunt tests to Travis build and to prod deployment scripts (probably)

Having data with multiple references concatenated produces a big horizontal html table. The whole set of references is considered as a single very long word making the table fill the screen. Because plot sizes are based on the container size, very big plots are being generated.

It may be affecting the functionality of right menu too.

See run 18804, lane 1 sequence mismatch.

rather than from a cache of XML feeds. Grep for NPG_WEBSERVICE_CACHE_DIR to identify tests, which use this cache.

... coming from a file system, from a database with no id_run, position, tag_index columns, having one component or more...

Create a role defining get_X methods, where X is id_run, position, tag_index, subset, inherit from this role in both parent chack and result object.

See handling of this problem in #493 in npg_qc_viewer/root/src/ui_checks/rna_seqc.tt2

Mqc state update should be delegated to a model. Too much awareness in the controller about low-level model behaviour.

as if cannot link to the flowcell table. Check that correct id is used in a search.

Users report application randomly stops generating controls for MQC when using firefox. No error is reported. The page generates properly after reloading.

One might be looking at reanalysis, but the details will come from the prod run

short_desc column shoudl be unique

Mqc outcome rows have boolean methods assessing the outcome as final, accepted, etc. These methods are straight delegations to similar methods on the dictionary objects. This is OK as long as the relationship exists. One might think that the relationship always exist since it's based on a non nullable foreign key. However, some of the objects in the code do not have this relationship because they are new (created with new()) in-memory objects.

Coding around this, for example, returning false if the relationship is not defined, would be logically wrong.

One of the solutions might be extending the new() method to return objects with the relationship already defined. If successful, this will remove lots of details from the parts of the code where these details do not belong and make the code more robust.

See discussion in #375

should have 'undecided final' entry

from Paul:" I have just QC’ed run 36525. However I have noticed there are two entries for Archival Pending for this run.
I know Neil had something similar a while ago and his run went through okay, however I thought it wise to inform NPG just in case run 36525 causes issues."

If the error message does not start with 'Error', the full message is displayed, example

No LIMs data for run 17877 position 1 at /nfs/users/nfs_m/mg8/working/npg_qc/npg_qc_viewer/bin/../lib/npg_qc_viewer/Controller/Mqc.pm line 85.

but

Error: No LIMs data for run 17877 position 1

Prefixing error messages in Perl with 'Error' for the sake of display in GUI is not a robust strategy; looks strange in perl.

extend clean action:
rm -rf root/static/bower_components

All past errors are displayed till the page i sreloaded from the server. Not possible to know whether the latest error is relevant. Modal dialog for errors might be better.

there are two problems with qc_db_delete_autoqc

firstly checks_list function has moved from npg_qc::autoqc::autoqc to npg_qc::autoqc::results::collection

secondly the code assume all the database table have a id_run column, this isn't true for qc checks which use composition i.e.

rna_seqqc, samtools_stats and sequence_summary

Problem with field names freeLK0 and freeLK1 in npg_qc::Schema::Result::VerifyBamId prevent data to be accessed from template and cases problems when showing verify bam

I wondered if pulling back the review and artic QC separately, and dealing with them separately (if possible and didn't make for code repetition), would simplify the code? (I'm not digging in enough to judge)

Historic tables for lane and plex level library qc have problems when deflating. Both historic tables were designed without a unique key for id_run/position and id_run/position/tag_index. They only work with a numeric primary key to identify each row. Because deflate works with the table definition (unique key), it can not properly deflate null value in tag_index. This can cause problems if using deflate before a search. As the null value will no be changed to the default value (-1) so it will not find the rows specified in the condition.