alleninstitute / aics-segmentation Goto Github PK

In the segmenter BioArxiv paper there is no citation of the underlying libraries, such as NumPy, SciPy, and scikit-image. Would it be possible to rectify the situation by releasing a new version of the paper?

The authors of those libraries depend on citations to 1. know when their libraries are used and 2. request funding because those libraries play an important role in the ecosystem. Thank you for understanding!

Edge preserving smoothing in 3D depends on voxel spacing. Below is the code you currently have for edge_preserving_smoothing_3d under aicssegmentation/core/pre_processing_utils.

Rather, this is what I think should be incorporated:

What are your thoughts?

Hi,

I have been trying to segment some of my data using batch_processing.py in classic code. I have been doing segmentation over two sets of data which have exactly similar file structure (as in data.shape, data file extension being tiff). However, one set of files run perfectly fine but the run_toolkit.sh script shows the following error for the other set of files:

AICSImage module does not support this image file type: CellImage can only accept OME-TIFF, TIFF, and CZI file formats!

Surprisingly, when I read in the same data file in an ipython console, I can very well read the data and segment it.

Not sure what's going wrong here. Could someone please throw some light on this?

NOTE: I don't think it should matter but I would also mention that I am trying to run batch_processing.py for both the datasets simultaneously using their respective run_toolkit.sh bash scripts.

Here's my run_toolkit.sh bash script:

WFNAME=supercell
INPUTCH=2
OUTPUTDIR="/Users/sharm261/Desktop/stack_images_zdr/segmented_stacks/"
INPUTFILE_TYPE='.tif'

# script for processing a whole folder
INPUTDIR="/Users/sharm261/Desktop/original_img_stack/"

python batch_processing.py \
        --d \
        --workflow_name $WFNAME \
        --struct_ch $INPUTCH \
        --output_dir $OUTPUTDIR \
        per_dir \
        --input_dir $INPUTDIR \
        --data_type $INPUTFILE_TYPE```

Hi,

I have been using batch_processing.py (run with run_toolkit.sh bash script) to perform segmentation for my data. Once I execute the bash script, I start getting the output as per expectation.

However, when I lock the screen so as to wait for the segmentation overnight, I see that the processing had stalled at the very moment where I had left before locking the screen. I am NOT letting the computer to go to sleep (just locking screen).

Anyone has any clues why this might be happening?

Note: I am using an iMac which has macOD High Sierra version 10.13.6

I am trying to run a few aics-segmentation functions on a dask array so I can process a number of stacks in parallel.

For example aicssegmentation.core.vessel.filament_3d_wrapper ...
1) If I run it on a dask array of length 1, it completes 1x stack in ~20 seconds with minimal CPU usage. This is about the same as running without a wrapping dask array ... good.
2) If I run it on a dask array of length 4, it completes each 1x stack in ~600 seconds with CPU looking like the 1x case. The 4x stacks are run in parallel but are not increasing CPU usage and are ~30 times slower than a 1x stack? [update], ran it again with a np.float and each call to filament_3d_wrapper when run across 4x stacks took ~1240 seconds, yikes!

I started looking at the source and after some tracing came up with no obvious reason. All I see is normal Python/NumPy/SciPy code? Seem to remember that aics-segmentation has a set of batch functions? Should I use that instead? Any links to example code?

Here is some sample code. In particular, scipy.ndimage.median_filter seems to work fine (runs in parallel and maxes out CPU) but filament_3d_wrapper runs >30x slower and does not max out the CPU (looks like usage at 1x stack).

import time
import numpy as np
import scipy

import dask
import dask.array as da

from aicssegmentation.core.vessel import filament_3d_wrapper

def myRun(path, commonShape, common_dtype):

	# create fake data
	stackData = np.random.normal(loc=100, scale=10, size=commonShape)
	#stackData = stackData.astype(common_dtype)
	
	# takes about 9 seconds if we have 1x in dask array
        # and still 9 seconds if we have 4x in dask array
	medianKernelSize = (3,4,4)
	print('  median filter', path)
	startTime = time.time()
	#
	smoothData = scipy.ndimage.median_filter(stackData, size=medianKernelSize)
	#
	stopTime = time.time()
	print('    median filter done in', round(stopTime-startTime,2), 'seconds', path)
	
	# takes about 19 seconds if we have 1x in dask array
        # but 500+ seconds if we have 4x in dask array
	print('  filament_3d_wrapper', path)
	startTime = time.time()
	#
	f3_param=[[1, 0.01]]
	filamentData = filament_3d_wrapper(smoothData, f3_param)
	filamentData = filamentData.astype(np.uint8)
	#
	stopTime = time.time()
	print('    filament_3d_wrapper done in', round(stopTime-startTime,2), 'seconds', path)
	
if __name__ == '__main__':

	# if I feed dask 1x stacks
	# filament_3d_wrapper returns in about 19 seconds (per stack)
	filenames = ['1']

	# if I feed dask 4x stacks
	# filament_3d_wrapper will run all 4 in parallel but CPU usage does not increase by 4x,
        # looks like I am running just 1x
	# filament_3d_wrapper returns in about 550-650 seconds (per stack)
	filenames = ['1', '2', '3', '4']
	
	# da.from_delayed() needs to know the shape and dtype it will work with?
	commonShape = (64, 512, 512)
	common_dtype = np.float #np.uint8

	# wrap myRun() function as a dask.delayed()
	myRun_Dask = dask.delayed(myRun)
	
	lazy_arrays = [dask.delayed(myRun_Dask)(filename, commonShape, common_dtype) for filename in filenames]

	lazy_arrays = [da.from_delayed(x, shape=commonShape, dtype=common_dtype) for x in lazy_arrays]

	x = da.block(lazy_arrays)
	
	x.compute()