lyonslab / coge Goto Github PK

When a new request is run SynMap sets its job status to running. When the analysis is completed the get_results function is called. The status continues to say running even after it has completed.

Results currently involve blocking until the program has finished execution.

Need to add detailed monitoring of workflows. This should include job execution time. Start and Stop times.

Possible metrics

• Average number of jobs
• The number of running jobs
• The number of completed jobs
• Max/Current load

The cancel button on the job page must be able to cancel the job status. This should report to the user that job has been cancel.

If the ajax request fails the dialog box won't update the dialog.

The job status page needs to be updated to use the job engine.

• Add job monitoring (report detailed information from the job engine)
• Add workflow cancellation for users and administrators
• Add job support for job cancellation
• Add job restarting

The new SlickGrid implementation broke the cancellation button. This feature needs to be ported to use SlickGrid.

The restart button on the job page should restart workflows. This should notify the user that the job has restarted.

The toolbar should have a feature which allows the chromosomes to be sorted by the name or size of the chromosome.

When viewing fasta sequences the MAFFT button displays "MATTF" instead of "MAFFT".

After each analysis is completed a list of available files to download is rendered to the client. The list size can be very long when running analysis tools that can run on many organisms at once.

Additionally, each file has to be downloaded manually. There is no way to download all the files at once.

A possible solution is to add a single download link which opens a download dialog. The user can then select the files they wish to download. The user then will be able to download a compress version of the files selected.

Create a new tag for the new release of SynMap.

This log snippet shows multiple workers running the same task. Note that each worker is running on a separate port.

07/17/2014 06:29:50 yerba.workqueue:90[INFO] WORKQUEUE coge-main: The workflow 1868 is scheduling job /opt/apache/coge/scripts/load_genome.pl  -user_name lht -name "" -desc "only scaffolds above 20kbp" -version "1.1" -restricted 1 -source_name "University of Queensland" -organism_id "39853" -type_id "1" -staging_dir '/storage/coge/tmp/staging/lht/1868' -fasta_files 'SSPACE_trimmed_out.final.scaffolds.fasta.rename.abv20k' -irods_files '' -config coge.conf -result_dir '/storage/coge/tmp/results/lht/1868'
2014/07/17 18:29:52.04 [68341] work_queue_python: wq: coge.iplantcollaborative.org (127.0.0.1:51667) busy on '/opt/apache/coge/scripts/load_genome.pl  -user_name lht -name "" -desc "only scaffolds above 20kbp" -version "1.1" -restricted 1 -source_name "University of Queensland" -organism_id "39853" -type_id "1" -staging_dir '/storage/coge/tmp/staging/lht/1868' -fasta_files 'SSPACE_trimmed_out.final.scaffolds.fasta.rename.abv20k' -irods_files '' -config coge.conf -result_dir '/storage/coge/tmp/results/lht/1868''
2014/07/17 19:09:57.13 [68341] work_queue_python: wq: coge.iplantcollaborative.org (127.0.0.1:37615) busy on '/opt/apache/coge/scripts/load_genome.pl  -user_name lht -name "" -desc "only scaffolds above 20kbp" -version "1.1" -restricted 1 -source_name "University of Queensland" -organism_id "39853" -type_id "1" -staging_dir '/storage/coge/tmp/staging/lht/1868' -fasta_files 'SSPACE_trimmed_out.final.scaffolds.fasta.rename.abv20k' -irods_files '' -config coge.conf -result_dir '/storage/coge/tmp/results/lht/1868''
2014/07/17 19:58:37.56 [68341] work_queue_python: wq: coge.iplantcollaborative.org (127.0.0.1:34189) busy on '/opt/apache/coge/scripts/load_genome.pl  -user_name lht -name "" -desc "only scaffolds above 20kbp" -version "1.1" -restricted 1 -source_name "University of Queensland" -organism_id "39853" -type_id "1" -staging_dir '/storage/coge/tmp/staging/lht/1868' -fasta_files 'SSPACE_trimmed_out.final.scaffolds.fasta.rename.abv20k' -irods_files '' -config coge.conf -result_dir '/storage/coge/tmp/results/lht/1868''
2014/07/17 21:02:12.21 [68341] work_queue_python: wq: coge.iplantcollaborative.org (127.0.0.1:33819) busy on '/opt/apache/coge/scripts/load_genome.pl  -user_name lht -name "" -desc "only scaffolds above 20kbp" -version "1.1" -restricted 1 -source_name "University of Queensland" -organism_id "39853" -type_id "1" -staging_dir '/storage/coge/tmp/staging/lht/1868' -fasta_files 'SSPACE_trimmed_out.final.scaffolds.fasta.rename.abv20k' -irods_files '' -config coge.conf -result_dir '/storage/coge/tmp/results/lht/1868''
2014/07/17 22:10:28.04 [68341] work_queue_python: wq: coge.iplantcollaborative.org (127.0.0.1:35633) busy on '/opt/apache/coge/scripts/load_genome.pl  -user_name lht -name "" -desc "only scaffolds above 20kbp" -version "1.1" -restricted 1 -source_name "University of Queensland" -organism_id "39853" -type_id "1" -staging_dir '/storage/coge/tmp/staging/lht/1868' -fasta_files 'SSPACE_trimmed_out.final.scaffolds.fasta.rename.abv20k' -irods_files '' -config coge.conf -result_dir '/storage/coge/tmp/results/lht/1868''
This entry below show multiple workers working on the same task. Each worker connects on a seperate port.

2014/07/17 23:29:28.33 [68341] work_queue_python: wq: coge.iplantcollaborative.org (127.0.0.1:38539) busy on '/opt/apache/coge/scripts/load_genome.pl  -user_name lht -name "" -desc "only scaffolds above 20kbp" -version "1.1" -restricted 1 -source_name "University of Queensland" -organism_id "39853" -type_id "1" -staging_dir '/storage/coge/tmp/staging/lht/1868' -fasta_files 'SSPACE_trimmed_out.final.scaffolds.fasta.rename.abv20k' -irods_files '' -config coge.conf -result_dir '/storage/coge/tmp/results/lht/1868''
2014/07/18 00:30:13.76 [68341] work_queue_python: wq: coge.iplantcollaborative.org (127.0.0.1:59706) busy on '/opt/apache/coge/scripts/load_genome.pl  -user_name lht -name "" -desc "only scaffolds above 20kbp" -version "1.1" -restricted 1 -source_name "University of Queensland" -organism_id "39853" -type_id "1" -staging_dir '/storage/coge/tmp/staging/lht/1868' -fasta_files 'SSPACE_trimmed_out.final.scaffolds.fasta.rename.abv20k' -irods_files '' -config coge.conf -result_dir '/storage/coge/tmp/results/lht/1868''
2014/07/18 01:17:00.17 [68341] work_queue_python: wq: coge.iplantcollaborative.org (127.0.0.1:51691) busy on '/opt/apache/coge/scripts/load_genome.pl  -user_name lht -name "" -desc "only scaffolds above 20kbp" -version "1.1" -restricted 1 -source_name "University of Queensland" -organism_id "39853" -type_id "1" -staging_dir '/storage/coge/tmp/staging/lht/1868' -fasta_files 'SSPACE_trimmed_out.final.scaffolds.fasta.rename.abv20k' -irods_files '' -config coge.conf -result_dir '/storage/coge/tmp/results/lht/1868''
2014/07/18 01:54:22.86 [68341] work_queue_python: wq: coge.iplantcollaborative.org (127.0.0.1:51691) sending back file-8b6583d2337d354fd783542ab701e0d1-1868 to /storage/coge/tmp/staging/lht/1868
2014/07/18 01:54:22.87 [68341] work_queue_python: wq: coge.iplantcollaborative.org (127.0.0.1:51691) sending back file-5205ac7aadc00cfaa101ddf54244c354-log.done to /storage/coge/tmp/staging/lht/1868/log.done
07/18/2014 01:54:22 yerba.workflow:279[INFO] WORKFLOW Create Genome: Added info to job /opt/apache/coge/scripts/load_genome.pl  -user_name lht -name "" -desc "only scaffolds above 20kbp" -version "1.1" -restricted 1 -source_name "University of Queensland" -organism_id "39853" -type_id "1" -staging_dir '/storage/coge/tmp/staging/lht/1868' -fasta_files 'SSPACE_trimmed_out.final.scaffolds.fasta.rename.abv20k' -irods_files '' -config coge.conf -result_dir '/storage/coge/tmp/results/lht/1868'

A workflow should have job level details that can be queried and viewed from the job engine.

• Add job level status
• Add job level cancellation
• Add job level restarting for administrators

When a synmap analysis fails, it is often due to a problem generating one of the intermediate files required for the workflow. E.g., fasta file of CDS sequences, blast databases, etc. We have been manually going to server to find and delete these files when there is a bad analysis reported. Want an admin button to do this:

rm /scratch/coge/cache/fasta/$genomeID
rm -r /scratch/coge/cache/last/db/$genomeID
or
rm -r /scratch/coge/cache/blast/db/$genomeID
rm /opt/apache2/coge/web/data/fasta/$genomeID
rm -r /opt/apache2/coge/web/data/diags/*****/$genomeID

The current render loop iterates over all points sent to the client side. The rendering is triggered whenever the canvas is zoomed or panned. Additionally, the rendering is performed in a nonblocking manner. A suggest method to improve the performance would be implementing a Quad tree.

The dialog short-link should be visible so the user can always have access to it.

The dotplot.pl command cannot correctly determine the bounds for axis labels. The GD::Image is undefined and cannot proceed.

Hello @LyonsLab CoGe JBrowse maintainers,

Just a thought, would you consider decoupling the jbrowse/plugins/CoGe directory from the main coge repo into its own standalone repository, so that, anyone who wishes to use your CoGe/View/Track/Wiggle/XYPlot track type (or other neat extensions) can easily clone the repo into their own JBrowse forks?

Thank you for your consideration.

Hi,
Do you have any plans to provide a docker container?

Thank you in advance

Michal

The toolbar will toggle various display options, load additional data, and perform additional interactions with the plot.

If there are spaces in iRODS folder names, CoGe's workflow will fail:

Fetching /iplant/home/arabiflora/coge_data/Concatenated AFG data/A02concat_R1.fastq
Job: mkdir -p /scratch/coge/tmp/uploads/arabiflora/FCFBB44EEBD611EAA59889A743376945/irods/iplant/home/arabiflora/coge_data/Concatenated AFG data && export irodsEnvFile='/opt/apache2/coge/irodsEnv' && iget -fT '/iplant/home/arabiflora/coge_data/Concatenated AFG data/A02concat_R1.fastq' '/scratch/coge/tmp/uploads/arabiflora/FCFBB44EEBD611EAA59889A743376945/irods/iplant/home/arabiflora/coge_data/Concatenated AFG data' && touch /scratch/coge/tmp/uploads/arabiflora/FCFBB44EEBD611EAA59889A743376945/irods/iplant/home/arabiflora/coge_data/Concatenated AFG data/A02concat_R1.fastq.done
Submitted at: 31/08/20 at 03:13:44PM
Completed at: 31/08/20 at 03:13:58PM
Execution time: 12.550773 sec
Assigned to task: 14389
Return status: 0
Expected outputs: /scratch/coge/tmp/uploads/arabiflora/FCFBB44EEBD611EAA59889A743376945/irods/iplant/home/arabiflora/coge_data/Concatenated AFG data/A02concat_R1.fastq, /scratch/coge/tmp/uploads/arabiflora/FCFBB44EEBD611EAA59889A743376945/irods/iplant/home/arabiflora/coge_data/Concatenated AFG data/A02concat_R1.fastq.done
Command Output:

The current HSP table generates a n column table where n is the number of selected genomes. This causes the table to render off the screen. A better method could be to render an interactive graph which can update the HSP table and corresponding layout.

Update the user dialog when there job has been cancelled.

The generate_chromosome_images method renders the start, stop and chromosome name into the image. The text rendering is broken in cogeblast branch.

CoGe's reported error:

######################### Sorting FASTA file Job: /opt/apache2/coge/scripts/sort_fasta.pl Pa_scaffold 0_7.fa > Pa_scaffold 0_7.fa.sorted Submitted at: 02/09/20 at 01:19:13AM Completed at: 02/09/20 at 01:19:14AM Execution time: 0.026153 sec Assigned to task: 16089 Return status: 2 Expected outputs: /scratch/coge/tmp/staging/twoehner85/272865/Pa_scaffold 0_7.fa.sorted Command Output: "Could not build fai index Pa_scaffold.fai\nCouldn't create index file at /opt/apache2/coge/scripts/sort_fasta.pl line

Note: file name is "Pa_scaffold 0_7.fa"

I have cached this file in my data store for debugging purposes.

Data points are currently not being rendered within the plot. The rendering has been recently changed from rendering svg elements to drawing with a canvas see issue #29. The new canvas needs to be overlaid on the plot.

The user may wish to zoom back to the original size. The reset button will pan the user out restoring the view to the full view. This does not restore the user to any saved view or state.

Currently the job page is visible by public users. Public users should be notified to login when they arrive at the job page.

CoGeBlast writes hits to a sqlite database while generating the result page. The number of hits that appear in results window will vary depending on how many successful hits get inserted into the database.

Create a poll rate that is less aggressive and is piecewise as a job takes longer.

Good morning
I would like to upload my jbrowse job in coge genomeevolutions but I don't have enought options like metadata optionsa and developers mode, so I download CoGe and tried to install the whole programs, modules and components. I would like to know how can I upload my genome and annotation in jbrowse to genome evolution coge.
thanks

The dotplot should display the chromosome name below the region of sequence. Labels need to be able to compress when the number of labels to be displayed is large.

Job: /usr/bin/nice --adjustment=10 /usr/local/bin/lastz CoGe_ZUlJXcog.fasta W=8 C=0 K=3000 M=0 O=400 E=30 4.faa > CoGe_ZUlJXcog-31.lastz
Submitted at: 06/09/13 at 01:22:11PM
Completed at: 06/09/13 at 01:22:16PM
Execution time: 0.230413 sec
Assigned to task: 1957
Return status: 256
FAILURE: bad fasta character in 4.faa, >gi|1: Y

This will bring side by side deployment of the old and new version of SynMap.

Hi, I use coge genome Evolutions, but can I add Hierarchical and Faceted track selectors to jbrowse throught CoGe.
I would like to get something like this:

The current plot supports the ability to skip chromosomes labelled as random or unknown. Any points which reference these random or unknown chromosomes should not be rendered when this field is toggled. The ordering and offset will have to be recalculated.