Parallel processing about navis

[clbarnes]

Pathos looks good to me! Agree that it could be an optional extra - the boilerplate for throwing the error when the function is called instead of on import would only need to be written in the decorator definition.

I'm aware of dill but am not very familiar with it - with any luck it allows out-of-band data like pickle 5 does. That significantly speeds up the ser/deser of numpy arrays. The default pickle has been 4 since python 3.8, so I suspect that's what the multiprocessing queues use 👎 .

An alternative way of dispatching operations over neuronlists would be something like this, implementing the parallelisation on the neuronlist:

def prune_neuron(neuron, param, more_params=None):
    ...
    return changed_neuron


class NeuronList:
    ...
    def map(self, fn, *args, parallel=None, **kwargs):
        if parallel:
            ...
        else:
            neurons = [fn(n, *args, **kwargs) for n in self]
        return type(self)(neurons)

You could still have a decorator as you suggest, it could just delegate its parallelism to NeuronList.map. I don't have strong feelings about which one to prefer!

In general I think it's a great step to factor out the dataframe/treeneuron/neuronlist into some shared implementation; same for handling parallelism setup (e.g. None for serial, 0 for max(1, (os.cpu_count() or 0) - 1)), although that might be a bit more tricky as sometimes you want more spare CPUs, or you prefer to use threading or whatever.

[schlegelp]

Actually your other suggestion already exists in NeuronList.apply :)

I was thinking of keeping that because it generalizes to any function - including non-navis functions. The advantage of the decorator is that we know that the output will be a NeuronList which makes some things easier.

[clbarnes]

Aha! My Ctrl+f was insufficient. Yes, that makes sense!

[schlegelp]

Implemented with 1c8ffe6