sciai-lab / covid-if-annotations Goto Github PK

The following situations currently crash the app:

• a new point is added to the points layer (we should just deactivate this)
• a new label is added to the segmentation or a label is removed from the segmentation (we need to adjust the update function to handle these cases)

Integrating the I/O function was straight-forward via the @napari_hook_implementation decorator.
(Great job!).
However, I am not quite sure what's the best way to integrate additional GUI functionality within the plugin, e.g. hiding a segment that was proofread, see #2 or going from step 1 (correcting the segmentation) to step 2 (classifying cells), see #1.

Hide annotate cells should probably be a toggle that goes "hide" -> "show" annotated cells and changes it's label based on button state, as it was a little unclear to me how it functioned. If you are in "hide mode" I'd probably at most show the last edited cell (so you can keep interacting with it), but then as soon as you label a new one that last one disappears. I imagine that auto disappearing will be very satisfying!

We got feedback from a windows test user that the tool does not function properly on her two windows laptops. On both machines, the points layer is not displayed and the transparency slider on the segmentation layer doesn't work.

Maybe an old OpenGL or QT version?

Feedback from @akreshuk:
add a shortcut and button that selects the next available label and goes to the paint mode.

Can be added as a shortcut to the cell-segmentation layer.

Is there an option to fix the color for some of the labels in the labels_layer, @sofroniewn?
Ideally something like this:

fixed_colors = {1: `red`, 2: `green`, 3: `blue`}
viewer.add_labels(labels, fixed_colors=fixed_colors)

I have two different an application where this would come in handy:
using the same colors for the edge and point representation of semantic classes and hiding segments with annotations from the segmentation.

Feedback from @akreshuk

Add a tooltip or similar that explains the semantic label colors:
white = unlabeled, red = infected, cyan = control, yellow = uncertain.

@constantinpape first off this tool is amazing!! So awesome to see it come together so fast too.

I'm going through and trying out the dev version (installed from repo) and I encountered the following bug.

Load in unlabelled data, select cell-segmentation labels layer, use color picker to select label of a cell. Switch to fill bucket click on neighbouring cell to "merge" that cell with current one (i.e. in the event of over segmentation), click u to update layers, then make infected-vs-control points layer visible and I get the following index error

Traceback (most recent call last):
  File "/Users/nsofroniew/GitHub/covid-if-annotations/napari_covid_if_annotations/gui.py", line 19, in <lambda>
    update_gui_btn.clicked.connect(lambda: update_layers(viewer))
  File "/Users/nsofroniew/GitHub/covid-if-annotations/napari_covid_if_annotations/_key_bindings.py", line 103, in update_layers
    viewer.layers['infected-vs-control'].refresh_colors()
  File "/Users/nsofroniew/GitHub/napari/napari/layers/points/points.py", line 991, in refresh_colors
    self._refresh_color('face', update_color_mapping)
  File "/Users/nsofroniew/GitHub/napari/napari/layers/points/points.py", line 1089, in _refresh_color
    color_event()
  File "/Users/nsofroniew/GitHub/napari/napari/utils/event.py", line 508, in __call__
    self._invoke_callback(cb, event)
  File "/Users/nsofroniew/GitHub/napari/napari/utils/event.py", line 529, in _invoke_callback
    cb_event=(cb, event),
  File "/Users/nsofroniew/GitHub/napari/napari/utils/event.py", line 523, in _invoke_callback
    cb(event)
  File "/Users/nsofroniew/GitHub/napari/napari/_vispy/vispy_points_layer.py", line 54, in _on_data_change
    edge_color = self.layer._view_edge_color
  File "/Users/nsofroniew/GitHub/napari/napari/layers/points/points.py", line 1334, in _view_edge_color
    return self.edge_color[self._indices_view]
IndexError: index 728 is out of bounds for axis 0 with size 728

Not sure if this is on napari end/ covid-if-annotator end, but wanted to document

The layer names are hard-coded in may places, so it would be good to freeze them, i.e. don't allow users to change them. See also #8 (comment)

cc @sofroniewn

The function get_edge_segmentation is fairly slow, because it erodes all segments individually:
https://github.com/hci-unihd/covid-if-annotations/blob/master/napari_covid_if_annotations/image_utils.py#L20-L30

Initially, I thought this would only be called once, so I didn't think about performance.
But now it is called every time the layers are updated and makes this step a bit unresponsive.
(I checked and it's much smoother if this is not activated).

Some options to speed this up:

• Vectorize / go to numba
• Use grayscale diation (would this work in this case?)

@jni any suggestiosn on this from the skimage side?

I am deactivating the select, add and remove button of the points layer, because they are not necessary or could lead to bugs in the case of add / remove.

However, after toggling the visibility of the points layer, these buttons are enabled again.

I tried to put in this check in on_layer_change, but it does not catch this, because it seems like toggling the visibility is not an event that triggers viewer.layers.events.changed.

If label switching via the left mouse is activated and a new point is being selected, the label of the point that has been selected switches.

I guess this check needs to be adapted to recognize that a new point is being selected and not switch labels in that case.

Another option would be to switch labels with right click.

The label hiding functionality can be improved:

• Actually hide the labels in the segmentation, pending #5.
• Draw the edges for unlabeled nodes and DON'T hide them.
• Add some indicator to the gui element that shows if hiding is on / off.

Installing on Linux with conda:

> git clone https://github.com/hci-unihd/covid-if-annotations.git
> cd covid-if-annotations
> conda create -c conda-forge -n covid-if-annotations napari scikit-image h5py pandas
> conda activate covid-if-annotations
(covid-if-annotations) > pip install -e .
(covid-if-annotations) > covid_if_annotations

results in:

Traceback (most recent call last):
  File "/home/heriche/miniconda3/envs/covid-if-annotations/bin/covid_if_annotations", line 33, in <module>
    sys.exit(load_entry_point('napari-covid-if-annotations', 'console_scripts', 'covid_if_annotations')())
  File "/home/heriche/Applications/covid-if-annotations/napari_covid_if_annotations/launcher/covid_if_annotations.py", line 150, in main
    launch_covid_if_annotation_tool(args.path, args.saturation_factor, args.edge_width)
  File "/home/heriche/Applications/covid-if-annotations/napari_covid_if_annotations/launcher/covid_if_annotations.py", line 134, in launch_covid_if_annotation_tool
    initialize_annotations(viewer, annotation_path)
  File "/home/heriche/Applications/covid-if-annotations/napari_covid_if_annotations/launcher/covid_if_annotations.py", line 23, in initialize_annotations
    with h5py.File(path, 'r') as f:
  File "/home/heriche/miniconda3/envs/covid-if-annotations/lib/python3.8/site-packages/h5py/_hl/files.py", line 395, in __init__
    name = filename_encode(name)
  File "/home/heriche/miniconda3/envs/covid-if-annotations/lib/python3.8/site-packages/h5py/_hl/compat.py", line 111, in filename_encode
    filename = fspath(filename)
TypeError: expected str, bytes or os.PathLike object, not int

Providing a file with
(covid-if-annotations) > covid_if_annotations --path ~/Downloads/gt_image_001.h5
produces the same error.

@sofroniewn: I am using the image layer to display the segmentation edges showing the semantic classification labels after seeing this post.

This makes more sense, because I actually don't need any of the labels layer functionality instead of having a transparent background label.
However, I am not quite sure how to create the colormap I need with vispy.Colormap.
I need the following color map {1: red, 2: cyan, 3: yellow} (and {0: transparent}, but that works already). See https://github.com/hci-unihd/covid-if-annotations/blob/master/napari_covid_if_annotations/layers.py#L87-L98 for details.

(I could also dig into the vispy documentation more to figure this out, but I hope you know the answer immediately ;)).

To make correcting the cell segmentation efficient in step 1 (see #1), we would need to be able to
hide cells that have already been corrected, or have been inspected and deemed as correct.
Otherwise users would likely visit the same cells all over again and it would be difficult to tell when to stop.

I have a slight issue with the reader IO hook:

I have a function modify_layers, that needs to be called AFTER the layers have been added to the viewer.
This is fine if the viewer is initialized with data, but if I add data via the IO hook, I can't do this in the launch function.

Right now, I have a dirty fix by calling this in the keybindings, e.g. here, but this means when adding data through drag'n'drop the viewer is only fully initialized after the first keybinding is used.

This came up in #3 (comment) already, but I don't really see how the solution @tlambert03 suggests would help here, because while I am in the napari_reader, the layers have not been added to the viewer yet.

The only clean solution I can see here is to add a callback on the viewer that is called after the layers returned by the reader have been added.
But I am very happy to hear other suggestions, @sofroniewn, @tlambert03 :).

In order to continue from saved results!

List of potential test users

• @metavibor or Severina
• Cuylen Lab

The goal of our proofreading efforts is to generate ground-truth for cell segmentation and infected / control cell classification.
@akreshuk, @wolny and me discussed how to design the tool for this today:

1. In the first step, users correct the existing segmentation using the painting functionality of the labels layer.
2. In the second step, users label the cells into "infected", "control" or "uncertain".

For the first step, having functionality to hide the cells that have already been inspected / proofread would be very helpful. I will open a separate issue to discuss this in more detail.

For the second step users would start from scratch (or from previous partial annotations, see below) and would need to label all cells as belonging to one of the three categories. In my prototype, I have done this by adding an additional labels layer that was initially empty and then propagating the annotations from this layer to the cell labels, and displaying the results in a separate layer that outlines the edges. See this image:

Once both steps are completed users upload the exported results via the EMBL annotation app (which is also used to obtain the initial files).

Also, we would like to allow users to submit "partially" annotated images. An image could be marked as partial if significant segmentation errors were noticed after step 1 was already finished (we don't want to allow going back and forth between step 1 and 2 in a single session to keep things simple) or if not all cells were classified in step 2.
Partial images can be uploaded in the same way, but are marked differently and would go back into the pool of available images for annotation; but initialized with the partially corrected segmentation / partially classified cells.

When clicking fast, not all mouse clicks are registered when clicking on points to change the "infected" / "control" / "uncertain" label.

Everything runs fine until

 > pip install -e .
ERROR: File "setup.py" not found. Directory cannot be installed in editable mode

Collecting some of the user feedback and my first impression after going through the initial 22 uploads.

Feedback:

Viewer / Tool:

• Overall tool seems to be easy enough to use.
• Fast mouse clicks are not registered, see #29.
• Points are too large and hide part of the cell. (Comment from me: yes, but making them smaller makes it hard to click them).
• Cell outlines are too thick for the small cells (Comment from me: ideally we would use inter-pixel contours and even resize them dynamically depending on the zoom level)
• Correcting segmentation errors can be very time consuming
• Yellow color for "uncertain" is hard to see in display night mode

Segmentation:

• Cell segmentation overall looks good.
• Some systematic issues with the cell segmentation:
  • Cells in mitosis are over segmented into 2 or even more parts (this is caused by the nucleus segmentation used for seeds; getting rid of this is part of the reason for the whole exercise.)
  • Cells sometimes "bleed out" into the background

Observations:

• Overall annotations look good
• Instead of correcting the segmentations, most annotators flag objects with segmentation errors as uncertain (given that large segmentation errors are not so common and take some time to correct, this should be fine)
• Some annotators label a lot of cells as uncertain, including cells that are clearly not infected. In these cases, I have proofread the annotations, because the training data is already skewed towards infected cells and we might have insufficient numbers of labeled non-infected cells if half of them are labeled as uncertain.

When we make the standalone apps for win/linux, the following things should be checked

• Check that the tool can be started and that you can load the example data via drag'n'drop.
• Check that the points are displayed and that you can change their color by clicking.
• Make the cell-outlines layer visible
• Check the keybindings and corresponding buttons:
  • u: the cell outlines should be colored the same as the corresponding points
  • h: only cell outlines for white points should remain
  • n: the segmentation layer is activated and you can paint with the next available id; do some painting !
  • shift + s the current annotations are stored as <FILE_NAME>_annotations.h5 in the same directory as the input data
• Check that reloading the labels work:
  • Close the app
  • Open the image again
  • Now, drag'n'drop the annotation file you have previously saved.
  • the segmentation should be updated to show your painted labels

Implementing the tool is moving along great and I think I will have a working version on Monday.
So we can start to think about how to make a standalone app in order to ship this to users.

@akreshuk
I think we can just package this ilastik style. Dependencies are minimal (napari, scikit-image, h5py) and all on conda-forge.

@sofroniewn
What are your thoughts on this?

See how we can use the writer io hook to export the annotation results.