funkelab / daisy Goto Github PK
View Code? Open in Web Editor NEWBlock-wise task scheduling for large nD volumes.
License: MIT License
Block-wise task scheduling for large nD volumes.
License: MIT License
When i submit a task with a bug on it, daisy try to solve it by submitting ~infinite number of workers.
Is it possible to set worker resubmission as option?
In dev-0.3, the block_id
is assigned directly the cantor number, which counts from 0
. In a personal communication with Logan, he said that @funkey prefers to have the blocks counted from 1
instead. This would be an easy change, and personally I have been using the 1
offset for some time now without any problem.
The problem is that making the change now would break compatibility with the previous numbering system, and can be a problem for usages where block_id
is stored in a DB and used (it is used extensively in my proofreading pipeline).
So I propose that we fix the block ID enumeration by adding 1
to the cantor number returned value, and bump the daisy versioning to dev-0.4. Thought?
When submitting a cluster task tqdm save a new line for twice for each block which can become a gigabytes file
Is there a way to make an option that set update frequency ? or use an other option that saves only the last line
In the current implementation, blocks are only checked for completion once they are requested by workers. This checking should instead be run in parallel, such that the master process is eliminating already completed blocks from the queue independently of the worker operations.
Line 92 in d60ef45
Need to add funlib.persistence to dependencies.
Ideally, make any dependencies automatically installed during setup - so it "comes with batteries included" ;)
In line 182 of datasets.py, 'chunk_size' is obtained by 'get_chunk_size ' which controls the 'chunk_size' to be smaller than 256. 256 is not an adjustable parameter. I suggest changing it to an adjustable parameter so that the user can obtain the size of chunks they like.
Since "valid" fit would be missing at most 1 write roi block whose read roi might go out of the total roi, I would expect that an "overhang" fit would add at most 1 block on each axis. This is not the case though since a valid fit checks:
total_roi.contains(b.read_roi)
, and the "overhang" checks total_roi.contains(b.write_roi.get_begin())
.
I've added an example to the documentation:
"valid": Skip blocks that would lie outside of ``total_roi``. This
is the default::
|---------------------------| total ROI
|rrrr|wwwwww|rrrr| block 1
|rrrr|wwwwww|rrrr| block 2
no further block
"overhang": Add all blocks that overlap with ``total_roi``, even if
they leave it. Client code has to take care of save access beyond
``total_roi`` in this case.::
|---------------------------| total ROI
|rrrr|wwwwww|rrrr| block 1
|rrrr|wwwwww|rrrr| block 2
|rrrr|wwwwww|rrrr| block 3 (overhanging)
|---------------------------| total ROI
|rrrrrr|www|rrrrrr| block 1
|rrrrrr|www|rrrrrr| block 2
|rrrrrr|www|rrrrrr| block 3
|rrrrrr|www|rrrrrr| block 4 (overhanging)
|rrrrrr|www|rrrrrr| block 5 (overhanging)
|rrrrrr|www|rrrrrr| block 6 (overhanging)
"shrink": Like "overhang", but shrink the boundary blocks' read and
write ROIs such that they are guaranteed to lie within
``total_roi``. The shrinking will preserve the context, i.e., the
difference between the read ROI and write ROI stays the same.::
|---------------------------| total ROI
|rrrr|wwwwww|rrrr| block 1
|rrrr|wwwwww|rrrr| block 2
|rrrr|www|rrrr| block 3 (shrunk)
Documentation accurately describes the current functionality.
Is this meant to be or should we change the "overhang" check to something like total_write_roi.contains(b.write_roi.get_begin())
Hey,
I'm trying to update an old script for daisy 0.2.1. to 1.1.1. I (probably naively) attempted to just use my old process function as the new process function in a daisy Task, but the error log gives me the error in the title.
Would love any suggestions, happy to give more as needed, full traceback here:
tornado.application - ERROR - Exception in callback functools.partial(<function wrap.<locals>.null_wrapper at 0x7f9567cc44d0>, <Task finished coro=<TCPStream._send_message() done, defined at /home/griffin/anaconda3/envs/synsev/lib/python3.7/site-packages/daisy/tcp/tcp_stream.py:64> exception=TypeError('Pickling an AuthenticationString object is disallowed for security reasons')>)
Traceback (most recent call last):
File "/home/griffin/anaconda3/envs/synsev/lib/python3.7/site-packages/tornado/ioloop.py", line 758, in _run_callback
ret = callback()
File "/home/griffin/anaconda3/envs/synsev/lib/python3.7/site-packages/tornado/stack_context.py", line 300, in null_wrapper
return fn(*args, **kwargs)
File "/home/griffin/anaconda3/envs/synsev/lib/python3.7/site-packages/tornado/ioloop.py", line 779, in _discard_future_result
future.result()
File "/home/griffin/anaconda3/envs/synsev/lib/python3.7/site-packages/daisy/tcp/tcp_stream.py", line 69, in _send_message
pickled_data = pickle.dumps(message)
File "/home/griffin/anaconda3/envs/synsev/lib/python3.7/multiprocessing/process.py", line 330, in __reduce__
'Pickling an AuthenticationString object is '
TypeError: Pickling an AuthenticationString object is disallowed for security reasons
The local scheduler uses asyncio functionality that is not available in Python 3.5:
Traceback (most recent call last):
File "/usr/local/bin/predict-affinities", line 11, in <module>
load_entry_point('eqip==0.3.1.dev0', 'console_scripts', 'predict-affinities')()
File "/usr/local/lib/python3.5/dist-packages/eqip/inference/backend_daisy.py", line 328, in predict_affinities_daisy
read_write_conflict=False)
File "/usr/local/lib/python3.5/dist-packages/daisy/scheduler.py", line 574, in run_blockwise
return distribute([{'task': BlockwiseTask()}])
File "/usr/local/lib/python3.5/dist-packages/daisy/scheduler.py", line 602, in distribute
return Scheduler().distribute(dependency_graph)
File "/usr/local/lib/python3.5/dist-packages/daisy/scheduler.py", line 69, in distribute
self._start_tcp_server()
File "/usr/local/lib/python3.5/dist-packages/daisy/scheduler.py", line 162, in _start_tcp_server
asyncio._set_running_loop(new_event_loop)
AttributeError: module 'asyncio' has no attribute '_set_running_loop'
It probably does not make sense to make daisy.run_blockwise
compatible with Python 3.5 because
Unfortunately, tensorflow docker images only come with Ubuntu 16.04 and Python3.5. The solution for users is to use conda (or pip) and install tensorflow-gpu into a virtual environment. Unfortunately, I do not see a good solution for daisy to ensure that users have an appropriate version of Python on their system:
python_requires='>=3.6'
to setup.py
would prohibit the use of the tensorflow docker containers (come only with Python3.5), which is probably required for lsf scenariosI suggest to use this issue as a note to users who would like to rundaisy.run_blockwise
in a non-lsf setting, maybe also linking to it in README.md
.
When trying to use tensorflow
in the workers of daisy.run_blockwise
, tensorflow
may only be imported inside the process_function
. This includes implicit imports, e.g. through gunpowder
. For some reason, tensorflow
does not play well with multiprocessing
. The solution is to make sure that tensorflow
is imported exclusively in the relevant subprocesses (and not in the main process). This is not an issue that arises from within daisy but it would be helpful for users to know how to resolve this issue (maybe even add a link to this issue to README.md
).
A simple example of what does not work:
import daisy
import gunpowder
def process_function():
scheduler = ClientScheduler()
while True:
# do something with gunpowder, e.g. build pipeline and run prediction on requested blocks
daisy.run_blockwise(
...
process_function=process_function,
num_workers=2) # no issues if only one worker
and what works:
import daisy
def process_function():
import gunpowder
scheduler = ClientScheduler()
while True:
# do something with gunpowder, e.g. build pipeline and run prediction on requested blocks
daisy.run_blockwise(
...
process_function=process_function,
num_workers=2)
It would be good to include a license file in the project. This is needed for other people that would like to use the project. It also is often required when creating packages for users to install.
Additionally it would be good to add a MANIFEST.in
, which includes the license file and add a setup.cfg
that also includes a license file. This ensures that other package types produce include and ship the license file for easier downstream consumption.
A different , seemingly unmaintained, daisy package is already on pypi. This will be automatically installed if daisy is listed in install_requires
in another package.
Add tests for MacOS and Windows. Currently there is a pickling error for both.
A declarative, efficient, and flexible JavaScript library for building user interfaces.
๐ Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. ๐๐๐
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google โค๏ธ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.