funkelab / daisy Goto Github PK

When i submit a task with a bug on it, daisy try to solve it by submitting ~infinite number of workers.
Is it possible to set worker resubmission as option?

In dev-0.3, the block_id is assigned directly the cantor number, which counts from 0. In a personal communication with Logan, he said that @funkey prefers to have the blocks counted from 1 instead. This would be an easy change, and personally I have been using the 1 offset for some time now without any problem.

The problem is that making the change now would break compatibility with the previous numbering system, and can be a problem for usages where block_id is stored in a DB and used (it is used extensively in my proofreading pipeline).

So I propose that we fix the block ID enumeration by adding 1 to the cantor number returned value, and bump the daisy versioning to dev-0.4. Thought?

When submitting a cluster task tqdm save a new line for twice for each block which can become a gigabytes file

Is there a way to make an option that set update frequency ? or use an other option that saves only the last line

In the current implementation, blocks are only checked for completion once they are requested by workers. This checking should instead be run in parallel, such that the master process is eliminating already completed blocks from the queue independently of the worker operations.

Need to add funlib.persistence to dependencies.
Ideally, make any dependencies automatically installed during setup - so it "comes with batteries included" ;)

In line 182 of datasets.py, 'chunk_size' is obtained by 'get_chunk_size ' which controls the 'chunk_size' to be smaller than 256. 256 is not an adjustable parameter. I suggest changing it to an adjustable parameter so that the user can obtain the size of chunks they like.

Since "valid" fit would be missing at most 1 write roi block whose read roi might go out of the total roi, I would expect that an "overhang" fit would add at most 1 block on each axis. This is not the case though since a valid fit checks:
total_roi.contains(b.read_roi), and the "overhang" checks total_roi.contains(b.write_roi.get_begin()).

I've added an example to the documentation:

        "valid": Skip blocks that would lie outside of ``total_roi``. This
        is the default::

            |---------------------------|     total ROI

            |rrrr|wwwwww|rrrr|                block 1
                   |rrrr|wwwwww|rrrr|         block 2
                                            no further block

        "overhang": Add all blocks that overlap with ``total_roi``, even if
        they leave it. Client code has to take care of save access beyond
        ``total_roi`` in this case.::

            |---------------------------|     total ROI

            |rrrr|wwwwww|rrrr|                block 1
                   |rrrr|wwwwww|rrrr|         block 2
                          |rrrr|wwwwww|rrrr|  block 3 (overhanging)

            |---------------------------|     total ROI

            |rrrrrr|www|rrrrrr|                      block 1
                |rrrrrr|www|rrrrrr|                  block 2
                    |rrrrrr|www|rrrrrr|              block 3
                        |rrrrrr|www|rrrrrr|          block 4 (overhanging)
                            |rrrrrr|www|rrrrrr|      block 5 (overhanging)
                                |rrrrrr|www|rrrrrr|  block 6 (overhanging)

        "shrink": Like "overhang", but shrink the boundary blocks' read and
        write ROIs such that they are guaranteed to lie within
        ``total_roi``. The shrinking will preserve the context, i.e., the
        difference between the read ROI and write ROI stays the same.::

            |---------------------------|     total ROI

            |rrrr|wwwwww|rrrr|                block 1
                   |rrrr|wwwwww|rrrr|         block 2
                          |rrrr|www|rrrr|     block 3 (shrunk)

Documentation accurately describes the current functionality.

Is this meant to be or should we change the "overhang" check to something like total_write_roi.contains(b.write_roi.get_begin())

Hey,

I'm trying to update an old script for daisy 0.2.1. to 1.1.1. I (probably naively) attempted to just use my old process function as the new process function in a daisy Task, but the error log gives me the error in the title.

Would love any suggestions, happy to give more as needed, full traceback here:

tornado.application - ERROR - Exception in callback functools.partial(<function wrap.<locals>.null_wrapper at 0x7f9567cc44d0>, <Task finished coro=<TCPStream._send_message() done, defined at /home/griffin/anaconda3/envs/synsev/lib/python3.7/site-packages/daisy/tcp/tcp_stream.py:64> exception=TypeError('Pickling an AuthenticationString object is disallowed for security reasons')>)
Traceback (most recent call last):
  File "/home/griffin/anaconda3/envs/synsev/lib/python3.7/site-packages/tornado/ioloop.py", line 758, in _run_callback
    ret = callback()
  File "/home/griffin/anaconda3/envs/synsev/lib/python3.7/site-packages/tornado/stack_context.py", line 300, in null_wrapper
    return fn(*args, **kwargs)
  File "/home/griffin/anaconda3/envs/synsev/lib/python3.7/site-packages/tornado/ioloop.py", line 779, in _discard_future_result
    future.result()
  File "/home/griffin/anaconda3/envs/synsev/lib/python3.7/site-packages/daisy/tcp/tcp_stream.py", line 69, in _send_message
    pickled_data = pickle.dumps(message)
  File "/home/griffin/anaconda3/envs/synsev/lib/python3.7/multiprocessing/process.py", line 330, in __reduce__
    'Pickling an AuthenticationString object is '
TypeError: Pickling an AuthenticationString object is disallowed for security reasons

Running Ubuntu 20.04, python 3.6, pip 21.3.1 in conda venv

Trying w/out cache directory:

Trying using specific version:

The local scheduler uses asyncio functionality that is not available in Python 3.5:

Traceback (most recent call last):
  File "/usr/local/bin/predict-affinities", line 11, in <module>
    load_entry_point('eqip==0.3.1.dev0', 'console_scripts', 'predict-affinities')()
  File "/usr/local/lib/python3.5/dist-packages/eqip/inference/backend_daisy.py", line 328, in predict_affinities_daisy
    read_write_conflict=False)
  File "/usr/local/lib/python3.5/dist-packages/daisy/scheduler.py", line 574, in run_blockwise
    return distribute([{'task': BlockwiseTask()}])
  File "/usr/local/lib/python3.5/dist-packages/daisy/scheduler.py", line 602, in distribute
    return Scheduler().distribute(dependency_graph)
  File "/usr/local/lib/python3.5/dist-packages/daisy/scheduler.py", line 69, in distribute
    self._start_tcp_server()
  File "/usr/local/lib/python3.5/dist-packages/daisy/scheduler.py", line 162, in _start_tcp_server
    asyncio._set_running_loop(new_event_loop)
AttributeError: module 'asyncio' has no attribute '_set_running_loop'

It probably does not make sense to make daisy.run_blockwise compatible with Python 3.5 because

the asyncio module has received new features, significant usability and performance improvements, and a fair amount of bug fixes. Starting with Python 3.6 the asyncio module is no longer provisional and its API is considered stable.

Unfortunately, tensorflow docker images only come with Ubuntu 16.04 and Python3.5. The solution for users is to use conda (or pip) and install tensorflow-gpu into a virtual environment. Unfortunately, I do not see a good solution for daisy to ensure that users have an appropriate version of Python on their system:

• adding python_requires='>=3.6' to setup.py would prohibit the use of the tensorflow docker containers (come only with Python3.5), which is probably required for lsf scenarios

I suggest to use this issue as a note to users who would like to rundaisy.run_blockwise in a non-lsf setting, maybe also linking to it in README.md.

When trying to use tensorflow in the workers of daisy.run_blockwise, tensorflow may only be imported inside the process_function. This includes implicit imports, e.g. through gunpowder. For some reason, tensorflow does not play well with multiprocessing. The solution is to make sure that tensorflow is imported exclusively in the relevant subprocesses (and not in the main process). This is not an issue that arises from within daisy but it would be helpful for users to know how to resolve this issue (maybe even add a link to this issue to README.md).

A simple example of what does not work:

import daisy
import gunpowder

def process_function():
    scheduler = ClientScheduler()
    while True:
        # do something with gunpowder, e.g. build pipeline and run prediction on requested blocks

daisy.run_blockwise(
    ...
    process_function=process_function,
    num_workers=2) # no issues if only one worker

and what works:

import daisy

def process_function():
    import gunpowder
    scheduler = ClientScheduler()
    while True:
        # do something with gunpowder, e.g. build pipeline and run prediction on requested blocks

daisy.run_blockwise(
    ...
    process_function=process_function,
    num_workers=2)

It would be good to include a license file in the project. This is needed for other people that would like to use the project. It also is often required when creating packages for users to install.

Additionally it would be good to add a MANIFEST.in, which includes the license file and add a setup.cfg that also includes a license file. This ensures that other package types produce include and ship the license file for easier downstream consumption.

A different , seemingly unmaintained, daisy package is already on pypi. This will be automatically installed if daisy is listed in install_requires in another package.

Add tests for MacOS and Windows. Currently there is a pickling error for both.