ysbecca / py-wsi Goto Github PK

Hi. When I try: "import py_wsi" in a script window in the Python IDE I get the following error "WinError 126 The specified module could not be found".
I have installed the other required packages I think.
According to the intellisense in the IDE the py_wsi is installed since member objects are displayed. What could be the problem with my installation? Do I have to install openslide in addition to openslide-python?
Kind regards
Kai-Inge

import py_wsi
Traceback (most recent call last):
File "<pyshell#0>", line 1, in
import py_wsi
File "C:\Program Files (x86)\Microsoft Visual Studio\Shared\Python36_64\lib\site-packages\py_wsi_init_.py", line 6, in
from .patch_reader import *
File "C:\Program Files (x86)\Microsoft Visual Studio\Shared\Python36_64\lib\site-packages\py_wsi\patch_reader.py", line 10, in
from openslide import open_slide
File "C:\Program Files (x86)\Microsoft Visual Studio\Shared\Python36_64\lib\site-packages\openslide_init_.py", line 29, in
from openslide import lowlevel
File "C:\Program Files (x86)\Microsoft Visual Studio\Shared\Python36_64\lib\site-packages\openslide\lowlevel.py", line 41, in
lib = cdll.LoadLibrary('libopenslide-0.dll')
File "C:\Program Files (x86)\Microsoft Visual Studio\Shared\Python36_64\lib\ctypes_init.py", line 426, in LoadLibrary
return self.dlltype(name)
File "C:\Program Files (x86)\Microsoft Visual Studio\Shared\Python36_64\lib\ctypes_init.py", line 348, in init
self._handle = _dlopen(self._name, mode)
OSError: [WinError 126] The specified module could not be found

Hi, @ysbecca
Thanks a lot for such great work!
I am beginner in py_wsi...generated xml for my image from aperio software annotation option...
but when i executed turtle.sample_and_store_patches(....) it gives me error i copied below...pls suggest some solution..i am strucked at this point...
i defined exact labels in label_map
i am using py_wsi 1.0...because i unbale to install 2.0..
Waiting..

Setting label as -1 for UNRECOGNISED LABEL.
{'Normal': 1, 'Benign': 2, 'Carcinoma in situ': 3, 'In situ carcinoma': 4, 'Invasive carcinoma': 5, 'Carcinoma invasive': 6}
py_wsi error: provided label not present in label map.
Setting label as -1 for UNRECOGNISED LABEL.
{'Normal': 1, 'Benign': 2, 'Carcinoma in situ': 3, 'In situ carcinoma': 4, 'Invasive carcinoma': 5, 'Carcinoma invasive': 6}
py_wsi error: provided label not present in label map.
Setting label as -1 for UNRECOGNISED LABEL.
{'Normal': 1, 'Benign': 2, 'Carcinoma in situ': 3, 'In situ carcinoma': 4, 'Invasive carcinoma': 5, 'Carcinoma invasive': 6}
py_wsi error: provided label not present in label map.

Could you please explain the purpose of this notebook and give a proper documentation in the file. Specifically, from cell 9 onwards.

Thanks

..

runfile('E:/Project_Sample/datbase/dread.py', wdir='E:/Project_Sample/datbase')
Traceback (most recent call last):

File "", line 1, in
runfile('E:/Project_Sample/datbase/dread.py', wdir='E:/Project_Sample/datbase')

File "C:\Users\Devidas.conda\envs\Devidas\lib\site-packages\spyder_kernels\customize\spydercustomize.py", line 827, in runfile
execfile(filename, namespace)

File "C:\Users\Devidas.conda\envs\Devidas\lib\site-packages\spyder_kernels\customize\spydercustomize.py", line 110, in execfile
exec(compile(f.read(), filename, 'exec'), namespace)

File "E:/Project_Sample/datbase/dread.py", line 7, in
import py_wsi

File "C:\Users\Devidas.conda\envs\Devidas\lib\site-packages\py_wsi_init_.py", line 6, in
from .patch_reader import *

File "C:\Users\Devidas.conda\envs\Devidas\lib\site-packages\py_wsi\patch_reader.py", line 10, in
from openslide import open_slide

File "C:\Users\Devidas.conda\envs\Devidas\lib\site-packages\openslide_init_.py", line 29, in
from openslide import lowlevel

File "C:\Users\Devidas.conda\envs\Devidas\lib\site-packages\openslide\lowlevel.py", line 41, in
_lib = cdll.LoadLibrary('libopenslide-0.dll')

File "C:\Users\Devidas.conda\envs\Devidas\lib\ctypes_init_.py", line 426, in LoadLibrary
return self._dlltype(name)

File "C:\Users\Devidas.conda\envs\Devidas\lib\ctypes_init_.py", line 348, in init
self._handle = _dlopen(self._name, mode)

OSError: [WinError 126] The specified module could not be found

Hi,

I don't know if it was intentional or not, but retrieve_tile_dimensions can return two or three parameters:

...
if not self.__check_file_found(file_name):
	return (0, 0)
...

making impossibile to unpack them (and generating a ValueError) in code such:

level_count, level_tiles, level_dims = turtle.retrieve_tile_dimensions('A01.svs', patch_size=128)
print("Level count:         " + str(level_count))
print("Level tiles:         " + str(level_tiles))
print("Level dimensions:    " + str(level_dims))

Maybe, in that specific case, the function could Raise a custom message RuntimeError if the file is not found.

Hi,

I'm trying to use py-wsi to write patches to my disk following your jupyter notebook example. My image (40x) has 18 levels, however when I try to use the line of code for the 18th level, I get an error that says the requested level does not exist. I wondering if there is some type of indexing difference? Below is the code I ran:

level=18
patch_size=32
overlap=0
turtle.sample_and_store_patches(patch_size, level ,overlap, limit_bounds=True)
TCGA-LL-A7T0-01Z-00-DX1.B03BBA63-ACF4-4BCA-9F2B-F631F0C6A25C.svs [py-wsi error]: requested level does not exist. Number of slide levels: 18
[py-wsi error]: no patches sampled from TCGA-LL-A7T0-01Z-00-DX1.B03BBA63-ACF4-4BCA-9F2B-F631F0C6A25C.svs . Continuing.

The svs file can be found from the genomic data commons if you wanted to recreate the error. Let me know if you have a fix.

Thanks,
Kevin

Is there any way to convert Ome.tiff format to SVS format ?

Hi Rebecca,
python version is 3.5.6
Your wsi library is really awesome and useful I have started working on whole slide images.
However, I am trying it on an Anaconda environment, but it seems like I cannot install it.

In dataset I have XML file which is only for the reference and is not the ground truth. Binary pixel masks is provided as TIF files are the ground truth. That is why there is a difference in label map. Can you suggest any solution to this problem? Thankyou

Hi, @ysbecca
Firstly, thanks a lot for your work!

I am a new to py_wsi, I met a problem when create the turtle object. Once I add the storage_type option, it will has the following error. I've read the previous issue, so I upgrade the the version of py-wsi, it still has the same problem!

----> 1 turtle = py_wsi.Turtle(file_dir, db_location, db_name, xml_dir=xml_dir, label_map=label_map, storage_type='lmdb')

TypeError: init() got an unexpected keyword argument 'storage_type'

I would like to draw N=50 tiles randomly from the pre-extracted tiles from each WSI, how to achieve this using py-wsi?

Hi Rebecca,

I am looking at the master branch of your code, py-wsi.

In file patch_reader.py lines ##156-162 are within the while-loop which starts at line #125:

    # Write to HDF5 files all in one go.
    if storage_option == 'hdf5':
        save_to_hdf5(db_location, patches, coords, file_name[:-4], labels)

    # Need to save tile dimensions if LMDB for retrieving patches by key.
    if storage_option == 'lmdb':
        save_meta_in_lmdb(meta_env, file_name[:-4], [x_tiles, y_tiles])

Which leads to redundant writing to HDF5 and LMDB files.

Is this done deliberately?

Hi,

When I increase the patch size to 128, I get this error

MapFullError: mdb_put: MDB_MAP_FULL: Environment mapsize limit reached

Hi,
I'm unable to change the storage type to disk.

The init signature doesn't show storage_type parameter

Please help me in getting access to this storage_type.

Thanks.

Vesion : py-wsi 2.0

While loading patches after storing them in a new lmdb database, I am getting "TypeError: a bytes-like object is required, not 'NoneType'" "

code snippet :-

patches, coords, classes, labels = turtle.get_patches_from_file("SB00045509.svs")

ERROR :-

TypeError Traceback (most recent call last)
in
1 #loading patches and associated metadata from the db using turtle manager
----> 2 patches, coords, classes, labels = turtle.get_patches_from_file("SB00045509.svs")

~/ICIAR/py-wsi-master/py_wsi/turtle.py in get_patches_from_file(self, file_name, verbose)
182 else:
183 # LMDB by default.
--> 184 items = self.__get_items_from_file(file_name[:-4])
185 return self.__items_to_patches_and_meta(items)
186

~/ICIAR/py-wsi-master/py_wsi/turtle.py in get_items_from_file(self, file_name)
426 for y in range(y - 1):
427 for x in range(x - 1):
--> 428 items.append(get_patch_from_lmdb(txn, x_, y_, file_name))
429 return items
430

~/ICIAR/py-wsi-master/py_wsi/store.py in get_patch_from_lmdb(txn, x, y, file_name)
47 str_id = file_name + '-' + str(x) + '-' + str(y)
48 raw_item = txn.get(str_id.encode('ascii'))
---> 49 item = pickle.loads(raw_item)
50 return item
51

TypeError: a bytes-like object is required, not 'NoneType'

Hi,

your code store.py has
def read_lmdb(location, name):
''' Read-only allows for multiple consecutive reads. '''
return lmdb.open(name, readonly=True)

Which prevents from opening the lmdb. To make this work I have changed this to

def read_lmdb(location, name):
''' Read-only allows for multiple consecutive reads. '''
return lmdb.open(location+'/'+name, readonly=True)

Is this expected or am I doing something wrong.

Hi Becca!

Fantastic project - thank you so much for putting this together.

I'm running through your tutorial, and everything works swimmingly except when I set "augment=True":

NameError                                 Traceback (most recent call last)

<ipython-input-19-d194f03ce242> in <module>()
      4                 total_sets=2,
      5                 shuffle_all=True,
----> 6                 augment=True)

~\AppData\Local\Continuum\miniconda3\envs\tensorflow_3.6\lib\site-packages\py_wsi\dataset.py in read_datasets(turtle, set_id, valid_id, total_sets, shuffle_all, augment, is_test)
    150         dataset.test = fetch_dataset(turtle, -1, -1, False)
    151     else:
--> 152         dataset.train = fetch_dataset(turtle, set_id, total_sets, augment)
    153         dataset.valid = fetch_dataset(turtle, valid_id, total_sets, augment)
    154         if shuffle_all:

~\AppData\Local\Continuum\miniconda3\envs\tensorflow_3.6\lib\site-packages\py_wsi\dataset.py in fetch_dataset(turtle, set_id, total_sets, augment)
    123     if augment:
    124       orig = np.copy(patches)
--> 125       if select_augment < 0:
    126         for j in range(1, 9):
    127           patches = np.concatenate((au.augment_patches(orig, j), patches), axis=0)

NameError: name 'select_augment' is not defined

I've looked through the code and cannot find select_augment anywhere else except for the one occurence in dataset.py.

Any thoughts?

Hi there!
Im using python 3.9.0 and working in virtualenv.
I just following for your instruction:

- brew install openslide
- pip install py_wsi

and here is my pip list output after that:

Package          Version
---------------- -------
h5py             3.1.0
lmdb             1.0.0
numpy            1.19.4
openslide-python 1.1.2
Pillow           8.0.1
pip              20.3.1
py-wsi           1.1
setuptools       49.2.1
Shapely          1.7.1
wheel            0.36.1

I see the same errors:

ModuleNotFoundError: No module named 'py_wsi'
ModuleNotFoundError: No module named 'openslide'

Plz help!

Hello, @ysbecca . First of all, thanks for your great works.
I have a question regard to calculating the number of total patches.
For understanding, I pasted my results as below.

======================================================
Total WSI images: 1
File names: ['01_01_0088.svs']
Setting patch size 256 and tile size 256
Level count: 17
Level tiles: ((1, 1), (1, 1), (1, 1), (1, 1), (1, 1), (1, 1), (1, 1), (1, 1), (1, 1), (2, 2), (4, 3), (7, 6), (14, 11), (27, 22), (53, 43), (105, 86), (210, 172))
Level dimensions: ((1, 1), (2, 2), (4, 3), (7, 6), (14, 11), (27, 22), (53, 43), (105, 86), (210, 172), (420, 344), (840, 687), (1680, 1374), (3360, 2747), (6720, 5493), (13440, 10986), (26879, 21971), (53757, 43941))
Patch size: 256
01_01_0088.svs
============ Patches Dataset Stats ===========
Total patches sampled: 130
Patches saved to: /data/JHS/paip_2019/Phase1/py_wsi/0088/
Patches saved with prefix: patch_db

Time usage: 0:00:15

I set the level as 12. As I understand, 'Total patches sampled' should be 42(=7*6), because the dimension of 12th level tiles is [7 by 6]. However, the number of total patches is 130.
I want to know the exact process to calculate the number of patches.

Thank you.

Dear @ysbecca I am looking to extract tiles from a .svs file, perform segmentation on the tile and then stitch all the tiles back so they reappear as the whole slide image in the correct order. Is it possible to do that with py_wsi? I am asking since I could not locate code for stitching/merging the code in the repo.

Hi, I want to extract only 'normal' label patches and form a lmdb database with just normal label patches.
I want to extract and save highest resolution (level 16) patches but my system is going out of memory at level 16 and since, I want to save only normal level, so it should not be a problem with memory if I could just do that.
Please, help me where should I make the changes in the code.

Also, I am confused about the set_id parameter. What does it signify? is it the image number from which the patches are extracted or something else. Suppose if i want to save patches image wise in separate folders, so if set_id is 0, all the patches in that set is from image 1?

There are 3 Images, 3 XML files and 3 .svs files. I am using Aperio Image Scope for processing. I have installed required libraries, set the paths correctly. These are my parameters, file directory of SVS images, patch size, level (magnification), and overlap.

import py_wsi
import lmdb
file_dir = "C:/Users/hits/filesxml/"
xml_dir = file_dir
patch_size = 200
level = 2
db_location = ""
db_name = "patch_db"
overlap = 0

I am encountering following error:
ValueError: not enough values to unpack (expected 4, got 0).
There is problem with this code line augment = False. I have gone through your code base, haven't been able to figure out the problem.

The stack trace is given below:

import py_wsi.dataset as ds
import py_wsi.imagepy_toolkit as imagepy_toolkit
dataset = ds.read_datasets(turtle,
set_id=1,
valid_id=0,
total_sets=2,
shuffle_all=True,
augment=False)
ValueError Traceback (most recent call last)
in ()
4 total_sets=2,
5 shuffle_all=True,
----> 6 augment=False)

C:\Users\hits\Anaconda3\lib\site-packages\py_wsi\dataset.py in read_datasets(turtle, set_id, valid_id, total_sets, shuffle_all, augment, is_test)
155 dataset.valid = fetch_dataset(turtle, valid_id, total_sets, augment)
156 if shuffle_all:
--> 157 dataset.train.shuffle_all()
158 dataset.valid.shuffle_all()
159

C:\Users\hits\Anaconda3\lib\site-packages\py_wsi\dataset.py in shuffle_all(self)
76 list_all = list(zip(self._images, self._labels, self._image_cls, self._coords))
77 random.shuffle(list_all)
---> 78 self._images, self._labels, self._image_cls, self._coords = zip(*list_all)
79
80 # [self._images, self._labels, self._image_cls, self._ids, self._coords] = \

ValueError: not enough values to unpack (expected 4, got 0)

                 ###  Issues with .scn WSI for Image patch generation using the xml.

I would like to know if .scn image which is also a Whole Slide Image can be processed by py_wsi apart from the .svs images. Since .scn images can also be viewed and annotated using Aperio Imagescope.
I would be extremely grateful to know if this functionality already exists in py_wsi or it could be added as a feature for processing the pathological Whole Slide Images?
My research work is on this domain and it would be really helpful if you could integrate such capabilities incase it doesn't exist already. I am actually unable to proceed with my research work due to this issue.

Awaiting your reply and thoughts about it. Please do let me know. @ysbecca

Hi, I am installing py-wsi using pip, but I am encountering following error
``(tensorflow) C:>pip install py-wsi
Collecting py-wsi
Using cached https://files.pythonhosted.org/packages/15/d2/70508f9fd4aa01e21c89f421f4c5d3a5f533af0eed6ea6bffae92c4a9678/py_wsi-0.21.tar.gz
Collecting shapely (from py-wsi)
Using cached https://files.pythonhosted.org/packages/7d/3c/0f09841db07aabf9cc387662be646f181d07ed196e6f60ce8be5f4a8f0bd/Shapely-1.6.4.post1.tar.gz
Complete output from command python setup.py egg_info:
Traceback (most recent call last):
File "", line 1, in
File "C:\Users\IIIT\AppData\Local\Temp\pip-build-wxoplbyb\shapely\setup.py", line 80, in
from shapely.buildcfg import geos_version_string, geos_version,
File "C:\Users\IIIT\AppData\Local\Temp\pip-build-wxoplbyb\shapely\shapely_buildcfg.py", line 200, in
lgeos = CDLL("geos_c.dll")
File "C:\Users\IIIT\Anaconda3\envs\tensorflow\lib\ctypes_init.py", line 348, in init
self._handle = _dlopen(self._name, mode)
OSError: [WinError 126] The specified module could not be found

----------------------------------------

Command "python setup.py egg_info" failed with error code 1 in C:\Users\IIIT\AppData\Local\Temp\pip-build-wxoplbyb\shapely`

I have all the dependencies already satisfied.

Hi Rebecca, thanks for the amazing code I've been using it for cytopathology slides to extract images for CNN, and it's almost perfect for liquid based cytology samples. I have a question, the highest level for most of the slides is 17. Except for on an odd occasion, the highest level is 18. Is there a way to to set the level to the highest level automatically ? I run a batch (about 10 files) of .svs files at a time, and this would be very helpful. Thanks so much in advance!
Simon

I want to show the whole slide image with pyside2, just like the function of asap software. Do you have methods or suggestions?

Hi Rebecca,

I have problems when I use the storage_type option but if I don't write storage_type='lmdb' or storage_type='disk' py_wsi.Turtle works perfectly. I would like to saves patches as PNG files, so I would very grateful to know what can be the problem. This is the error I get:

turtle = py_wsi.Turtle(file_dir, db_location, db_name, xml_dir=xml_dir, storage_type='lmdb')

TypeError Traceback (most recent call last)

in

----> 1 turtle = py_wsi.Turtle(file_dir, db_location, db_name, xml_dir=xml_dir, storage_type='lmdb')

TypeError: init() got an unexpected keyword argument 'storage_type'

Hi Rebecca! I wonder is there any Python API can save normal image to .svs image. After I processed .svs files with opencv, I have to reconvert the result for saving memory. Thanks a lot!

~\Desktop\py-wsi-master\py_wsi\store.py in new_lmdb(location, name, map_size_bytes)
58
59 def new_lmdb(location, name, map_size_bytes):
---> 60 return lmdb.open(location + name, map_size=map_size_bytes)
61
62 def print_lmdb_keys(env):

Error: /Users/shiva/Project1/py_wsi/patch_db_meta: The requested operation cannot be performed on a file with a user-mapped section open.

So this error occured while sampling and storing patches .
please look at it

I would like to retrieve patches from svs images at 20x magnification. From tutorial I have seen, patches have been retrieved only using levels, Is there a way to get patches using magnification and store them in png format?

please help me, I had it:

`effer@jeffer-sky:~/Descargas$ pip install py_wsi
Collecting py_wsi
Using cached https://files.pythonhosted.org/packages/32/f4/a4f692b8d7791df38874e678e28c60783284ca4a5fe4f083214f45e4ede2/py_wsi-1.1.tar.gz
Complete output from command python setup.py egg_info:
Traceback (most recent call last):
File "", line 1, in
File "/tmp/pip-install-Sc0b5n/py-wsi/setup.py", line 12, in
with open(path.join(here, 'README.rst'), encoding='utf-8') as f:
TypeError: 'encoding' is an invalid keyword argument for this function

----------------------------------------

Command "python setup.py egg_info" failed with error code 1 in /tmp/pip-install-Sc0b5n/py-wsi/`

I am using MAC OS and python 3.7. I am getting below error. Please suggest how can I solve this issue.

Using cached https://files.pythonhosted.org/packages/32/f4/a4f692b8d7791df38874e678e28c60783284ca4a5fe4f083214f45e4ede2/py_wsi-1.1.tar.gz
Complete output from command python setup.py egg_info:
Traceback (most recent call last):
File "", line 1, in
File "/private/var/folders/nh/yztkyf6x0jj10r0n4x25gjb00000gn/T/pip-install-2Meh31/py-wsi/setup.py", line 12, in
with open(path.join(here, 'README.rst'), encoding='utf-8') as f:
TypeError: 'encoding' is an invalid keyword argument for this function

----------------------------------------

Command "python setup.py egg_info" failed with error code 1 in /private/var/folders/nh/yztkyf6x0jj10r0n4x25gjb00000gn/T/pip-install-2Meh31/py-wsi/

Hi i ahve face this please tell how to fix . i run this code import py_wsi but error
I installed all dependencies but still this issue
FileNotFoundError: Could not find module 'libopenslide-0.dll' (or one of its dependencies). Try using the full path with constructor syntax.

It is possible to create multiple png file to svs file and add some meta data to it. meta data like patientname,age,like that

Hi Rebecca. Thank you for writing this repo. It's very helpful.

However, the annotation format is still unclear for me. If possible, could you send me a link to download the A01.svs dan A01.xml file?

Thank you.

ModuleNotFoundError Traceback (most recent call last)
in
----> 1 import py_wsi

ModuleNotFoundError: No module named 'py_wsi'
i dont know why py_wsi cannot be imported in latest version...even i tried to switch kernel to python 3.6.1 but failed

Hi!
Thanks for this project, I find it very useful.

Just a minor issue with tab indentation in the python files, I think it's not too late to convert all indentation from tabs to spaces, as per PEP-0008 https://www.python.org/dev/peps/pep-0008/#tabs-or-spaces

           ##Patches with Different Labels do not get Processed Correctly.

I have tried to process images with different labels. The image patches that are generated belong to single default label of type 0. I have tried customizing the code but to no avail. Also I have tried running the program at various levels from Level 1 to Level 16 that is the maximum number of levels which exist for the image I am processing. However I get image patches or tiles with only default label 0.

Please could you suggest what can be done so that all the patches belonging to different labels are extracted correctly rather just going into the default Normal or Label 0. I have tried my level best to resolve it, however I wasn't able to extract image patches with valid labels correctly.

@ysbecca Please could let me know I am stuck up in the middle of my research.

Is there a way to add functionality for other files types beyond SVI? I am working with TIFF files and see that I can change file type in one of the modules, but curious about other functionality. I am not very experienced with WSI so I am not sure if this is a big ask. Thanks!

def __get_files_from_dir(self, file_dir, file_type='.svs'):

Hi Rebecca!

Thanks again for the great package.

I've installed py-wsi with deep learning environments on a few Windows computers now. All the dependencies w/ Openslide, etc., can be a pain, definitely a "works on my machine" situation.

Do you use Docker with py-wsi? If so, it would be great to see how you have the Dockerfile set up.

If not, I'm working on one now and would be happy to share it when I get it working.

How and from where to get XML file? I am downloading .svs files from TCGA but there are no xml files there and neither there is any mention of them.

Hi Rebecca,
Your package is very help.
Could you provide a pointer for sample data?
I used a downloaded .svs. What xml file should I provide? Could you provide an example?

Unet was used for patch segmentation. After got the result for each patch, I would like to generate a mask image with the size as the same as whole side image.
Is there a function for that?

Hello, I know that this is not the best place to ask, but I got some doubts.
My project dissertation is Deep Learning for oncocytic cells detection and I'm using py_wsi to load images. However, I don't understand how I can use the .xml file to label my patches.
You have:

label_map = {'Normal': 0,
             'Benign': 1,
             'Carcinoma in situ': 2,
             'In situ carcinoma': 2,
             'Carcinoma invasive': 3,
             'Invasive carcinoma': 3,
            }

• Are those labels discriminated on the xml file?

Also, I think that cropping the image without a preprocessing can jeopardize some of the cell information since you may be slicing a cell in half.

I really think this part should be treated carefully because it will have serious repercussions in the segmentation phase and every feature I can extract is crucial.

Thank you in advance, your usage example is guiding me in these first steps and I really appreciate it.

Regards