tissuemaps / jtlibrary

Please use the https://github.com/TissueMAPS/TissueMAPS instead.

Jterator Modules and Library

This repository contains code for the Jterator image analysis pipeline (JtModules), and a collection of support and utility functions (JtLibrary).

For more information, please refer to main TissueMAPS repository.

jtlibrary's Issues

Retrieval of saved objects

The current approach for saving and retrieving objects is based on outline coordinates of objects.

1. Outlines of objects in the labeled image are determined for each object: skimage.measure.find_contours()
2. y, x coordinates are saved in the HDF5 file
3. Coordinates are retrieved from the file
4. The label image is reconstructed: cv2.fillPoly()

It needs to be tested whether this approach works reliably for different types of objects...

Wrong object count in identify_secondary_iw.m

For some images, the object count is wrong after applying the watershed segmentation. The reason is not clear to me yet. It is related to the shape of the primary objects, however, since smoothing the input mask solves the problem. It could be related to the labeling of objects, which is different in Matlab and Python.

Plotting is broken in current develop due to import error

Whenever plotting is set to True in a module in the current develop branch (14db3a8), the pipeline fails because of the following import error:
JOB FAILED: cannot import name plotting File "/home/tissuemaps/tmlibrary/tmlib/workflow/cli.py", line 283, in __main__ cli_instance(arguments) File "/home/tissuemaps/tmlibrary/tmlib/workflow/cli.py", line 342, in __call__ method(**method_args) File "/home/tissuemaps/tmlibrary/tmlib/workflow/jterator/cli.py", line 102, in debug api.run_job(batch, assume_clean_state=False) File "/home/tissuemaps/tmlibrary/tmlib/workflow/jterator/api.py", line 679, in run_job store = self._run_pipeline(store, site_id, batch['plot']) File "/home/tissuemaps/tmlibrary/tmlib/workflow/jterator/api.py", line 363, in _run_pipeline module.run(self._engines[module.language]) File "/home/tissuemaps/tmlibrary/tmlib/workflow/jterator/module.py", line 470, in run return self._exec_py_module() File "/home/tissuemaps/tmlibrary/tmlib/workflow/jterator/module.py", line 196, in _exec_py_module py_out = func(**kwargs) File "/home/tissuemaps/jtlibrary/python/jtmodules/src/jtmodules/smooth.py", line 85, in main from jtlib import plotting

The "from jtlib import plotting" doesn't work anymore because of the refactor that reorganized the modules & jtlibrary code. @riccardomurri How can one import the plotting file which is located at "JtLibrary/python/jtlibrary/src/jtlib/"? This import needs to be changed for all modules that use plotting.

iterator pipeline issue

ubuntu@frontend001:/data/storage/experiments/experiment_1/workflow/jterator_segment_2d_cells_40x$ jterator -v 1 -p segment_2d_cells_40x init

_ _                _

() | ___ _ _ __ | | ___ _ _ tmlib.workflow.jterator (0.0.7)
| | / -) '/ ` | / _ \ '| Image analysis pipeline engine
/ ||| **,|___/_| https://github.com/TissueMAPS/TmLibrary
|**/

2016-06-01 13:24:40 | INFO | tmlib.workflow.cli | delete previous job output
2016-06-01 13:24:40 | INFO | tmlib.workflow.jterator.api | delete existing mapobject types
2016-06-01 13:24:40 | INFO | tmlib.workflow.cli | create batches
The following error occurred while reading from file:
while scanning for the next token
found character '\t' that cannot start any token
in "", line 6, column 16:
correct: true
^ File "/home/ubuntu/TmLibrary/src/tmlib/workflow/jterator/project.py", line 165, in _create_pipe
'description': f.read()
File "/home/ubuntu/TmLibrary/src/tmlib/readers.py", line 182, in read
return load_yaml(self._stream.read())
File "/home/ubuntu/TmLibrary/src/tmlib/readers.py", line 162, in load_yaml
return ruamel.yaml.load(string, ruamel.yaml.RoundTripLoader)
File "/home/ubuntu/.local/lib/python2.7/site-packages/ruamel/yaml/main.py", line 75, in load
return loader.get_single_data()
File "/home/ubuntu/.local/lib/python2.7/site-packages/ruamel/yaml/constructor.py", line 59, in get_single_data
node = self.get_single_node()
File "/home/ubuntu/.local/lib/python2.7/site-packages/ruamel/yaml/composer.py", line 48, in get_single_node
document = self.compose_document()
File "/home/ubuntu/.local/lib/python2.7/site-packages/ruamel/yaml/composer.py", line 68, in compose_document
node = self.compose_node(None, None)
File "/home/ubuntu/.local/lib/python2.7/site-packages/ruamel/yaml/composer.py", line 99, in compose_node
node = self.compose_mapping_node(anchor)
File "/home/ubuntu/.local/lib/python2.7/site-packages/ruamel/yaml/composer.py", line 158, in compose_mapping_node
item_value = self.compose_node(node, item_key)
File "/home/ubuntu/.local/lib/python2.7/site-packages/ruamel/yaml/composer.py", line 97, in compose_node
node = self.compose_sequence_node(anchor)
File "/home/ubuntu/.local/lib/python2.7/site-packages/ruamel/yaml/composer.py", line 128, in compose_sequence_node
node.value.append(self.compose_node(node, index))
File "/home/ubuntu/.local/lib/python2.7/site-packages/ruamel/yaml/composer.py", line 99, in compose_node
node = self.compose_mapping_node(anchor)
File "/home/ubuntu/.local/lib/python2.7/site-packages/ruamel/yaml/composer.py", line 158, in compose_mapping_node
item_value = self.compose_node(node, item_key)
File "/home/ubuntu/.local/lib/python2.7/site-packages/ruamel/yaml/composer.py", line 77, in compose_node
if self.check_event(AliasEvent):
File "/home/ubuntu/.local/lib/python2.7/site-packages/ruamel/yaml/parser.py", line 116, in check_event
self.current_event = self.state()
File "/home/ubuntu/.local/lib/python2.7/site-packages/ruamel/yaml/parser.py", line 518, in parse_block_mapping_value
token.move_comment(self.peek_token())
File "/home/ubuntu/.local/lib/python2.7/site-packages/ruamel/yaml/scanner.py", line 1543, in peek_token
self._gather_comments()
File "/home/ubuntu/.local/lib/python2.7/site-packages/ruamel/yaml/scanner.py", line 1573, in _gather_comments
self.fetch_more_tokens()
File "/home/ubuntu/.local/lib/python2.7/site-packages/ruamel/yaml/scanner.py", line 276, in fetch_more_tokens
% utf8(ch), self.get_mark())

segment secondary module test

Error in jtlib.segmentSecondary (line 502)
FinalLabelMatrixImagePre = LabelsUsed(ActualObjectsLabelMatrixImage3+1);
Index exceeds matrix dimensions.

module for ilastik integration

Creation of a jterator module for pixel based classification à la CellProfiler classifypixels module. The idea is to train a classifier with the Ilastik GUI, save the output (either the classifier as ilastik project HDF5 file or a probability map as an image file) and then apply the output to images in a jterator pipeline. To this end, we need to create a Python module that implements functions to call ilastik independent of CP2.

Debugging of modules

The way modules are currently imported makes it difficult to debug them.

They should probably be included into the jtlib package as subpackage modules. Thereby one could import them "normally" and set break points in the code.

Maybe we should re-think the whole approach of providing the absolute path to modules. The idea behind that was that one could easily test code located somewhere else, but this may not pay off.

segment secondary module test bug

https://github.com/TissueMAPS/JtLibrary/blob/saadia/python/jtlib/modules/Jt_Modules_Saadia/segment_secondary.py

This code works fine in the first iteration, but in the next iteration- it gives the error:

Traceback (most recent call last):

File "", line 1, in
runfile('/Users/saadiaiftikhar/miniconda2/bin/jlib/test_20160531.py', wdir='/Users/saadiaiftikhar/miniconda2/bin/jlib')

File "/Users/saadiaiftikhar/miniconda2/lib/python2.7/site-packages/spyderlib/widgets/externalshell/sitecustomize.py", line 807, in runfile
execfile(filename, namespace)

File "/Users/saadiaiftikhar/miniconda2/lib/python2.7/site-packages/spyderlib/widgets/externalshell/sitecustomize.py", line 88, in execfile
builtins.execfile(filename, *where)

File "/Users/saadiaiftikhar/miniconda2/bin/jlib/test_20160531.py", line 99, in
[secondaryLabelMatrixImage,edited_primary_binary_image,th_array] = identify_secondary_objects_iw.identify_secondary_objects_iw(label_image,input_image,correction_factors,min_threshold)

File "identify_secondary_objects_iw.py", line 48, in identify_secondary_objects_iw
min_threshold, max_threshold)

File "segment_secondary.py", line 190, in segment_secondary
labels_used1[actual_objects_label_matrix_image3]

IndexError: index 4 is out of bounds for axis 1 with size 4

Implement ui mode for jterator

Set plot to True and don't write pipeline output into the database when running in UI mode.

pipeline

I tried various possibilities, but the error remained the same for the pipeline:

() | ___ _ _ __ | | ___ _ _ tmlib.workflow.jterator (0.0.7)
| | / -) '/ ` | / _ \ '| Image analysis pipeline engine
/ ||| **,|___/_| https://github.com/TissueMAPS/TmLibrary
|**/

2016-06-02 13:07:14 | INFO | tmlib.workflow.cli | delete previous job output
2016-06-02 13:07:14 | INFO | tmlib.workflow.jterator.api | delete existing mapobject types
2016-06-02 13:07:14 | INFO | tmlib.workflow.cli | create batches
The following error occurred while reading from file:
while parsing a block mapping
in "", line 2, column 1:
input:
^
expected , but found '-'
in "", line 8, column 1:
- name: filter_name
^ File "/home/ubuntu/TmLibrary/src/tmlib/workflow/jterator/project.py", line 212, in _create_handles
'description': f.read()
File "/home/ubuntu/TmLibrary/src/tmlib/readers.py", line 182, in read
return load_yaml(self._stream.read())
File "/home/ubuntu/TmLibrary/src/tmlib/readers.py", line 162, in load_yaml
return ruamel.yaml.load(string, ruamel.yaml.RoundTripLoader)
File "/home/ubuntu/.local/lib/python2.7/site-packages/ruamel/yaml/main.py", line 75, in load
return loader.get_single_data()
File "/home/ubuntu/.local/lib/python2.7/site-packages/ruamel/yaml/constructor.py", line 59, in get_single_data
node = self.get_single_node()
File "/home/ubuntu/.local/lib/python2.7/site-packages/ruamel/yaml/composer.py", line 48, in get_single_node
document = self.compose_document()
File "/home/ubuntu/.local/lib/python2.7/site-packages/ruamel/yaml/composer.py", line 68, in compose_document
node = self.compose_node(None, None)
File "/home/ubuntu/.local/lib/python2.7/site-packages/ruamel/yaml/composer.py", line 99, in compose_node
node = self.compose_mapping_node(anchor)
File "/home/ubuntu/.local/lib/python2.7/site-packages/ruamel/yaml/composer.py", line 151, in compose_mapping_node
while not self.check_event(MappingEndEvent):
File "/home/ubuntu/.local/lib/python2.7/site-packages/ruamel/yaml/parser.py", line 116, in check_event
self.current_event = self.state()
File "/home/ubuntu/.local/lib/python2.7/site-packages/ruamel/yaml/parser.py", line 505, in parse_block_mapping_key
token.start_mark)