Parallelized batch processing about pynanoflann

[u1234x1234]

Hi Juho,

I get your point, but it seems like a custom solution, which will require additional efforts as there is no such option in the original nanoflann library, so I have no plans for this now. Contributions are welcome.

[u1234x1234]

Do you need kd_trees after queries? Or would such an interface work too?

import pynanoflann
import numpy as np

n_batches = 10
target = np.random.rand(n_batches, 10000, 3)
query = np.random.rand(n_batches, 2000, 3)

distances, nn_indexes = batched_search(target, query, n_neighbors=1, metric='L2', leaf_size=20)

[juldaani]

Hey Dmitrii,

Thanks for the very fast response.

In my current application I don't need the trees so they can be thrown away.

[u1234x1234]

I added Parallelized batch processing.

pip install git+https://github.com/u1234x1234/[email protected]

Full example:
https://github.com/u1234x1234/pynanoflann/blob/master/tests/test_batched_kneighbors.py

Internally it works for the list of numpy arrays of arbitrary shapes ie:

target = [
  np.random.rand(10000, 3),
  np.random.rand(5000, 3),
  np.random.rand(40000, 3),
]

I've implemented a more general interface at the cost of small overhead.

[juldaani]

Hey,

I tested the implementation and it works very well. No bugs found. And have to say the library is now very fast with these multicore capabilities. As far as I know this is the fastest kd-tree implementation for python.

This helped me a lot and made my algorithm many times faster. Thank you very much.

[u1234x1234]

I'm glad it helped you.