bparbhu / dispatcher.jl

Build, distribute, and execute task graphs

Dispatcher

Dispatcher is a tool for building and executing a computation graph given a series of dependent operations.

Documentation:

Overview

Using Dispatcher, run! builds and runs a computation graph of DispatchNodes. DispatchNodes represent units of computation that can be run. The most common DispatchNode is Op, which represents a function call on some arguments. Some of those arguments may exist when building the graph, and others may represent the results of other DispatchNodes. An Executor executes a whole DispatchGraph. Two Executors are provided. AsyncExecutor executes computations asynchronously using Julia Tasks. ParallelExecutor executes computations in parallel using all available Julia processes (by calling @spawn).

Frequently Asked Questions

How is Dispatcher different from ComputeFramework/Dagger?

Dagger is built around distributing vectorized computations across large arrays. Dispatcher is built to deal with discrete, heterogeneous data using any Julia functions.

How is Dispatcher different from Arbiter?

Arbiter requires manually adding tasks and their dependencies and handling data passing. Dispatcher automatically identifies dependencies from user code and passes data efficiently between dependencies.

Isn't this just Dask?

Pretty much. The plan is to implement another Executor and integrate with the dask.distributed scheduler service to piggyback off of their great work.

How does Dispatcher handle passing data?

Dispatcher uses Julia RemoteChannels to pass data between dispatched DispatchNodes. For more information on how data transfer works with Julia's parallel tools see their documentation.