Cannot query an index constructed from sparse matrix about pynndescent

[lmcinnes]

Nope, but that should be fixed now. Sorry was away from dealing with github for a while and lost track of these.

[lmcinnes]

Found it -- it was the added checks of near zero distances that were unsupported by numba. Resolved easily enough. It should work now, and there should be a 0.4.6 release with it fixed available on PyPi (and conda-forge) shortly.

[ivirshup]

Running into this over on scanpy as well. Here's a minimal reproducer:

from scipy import sparse
from pynndescent import NNDescent

NNDescent(sparse.random(100, 100), metric="cosine").neighbor_graph

Traceback and warnings

/usr/local/lib/python3.7/site-packages/pynndescent/sparse_nndescent.py:502: NumbaTypeSafetyWarning: unsafe cast from uint32 to int32. Precision may be lost.
  init_rp_tree(inds, indptr, data, dist, current_graph, leaf_array)
/usr/local/lib/python3.7/site-packages/pynndescent/pynndescent_.py:1012: UserWarning: Failed to correctly find n_neighbors for some samples.Results may be less than ideal. Try re-running withdifferent parameters.
  "Failed to correctly find n_neighbors for some samples."
---------------------------------------------------------------------------
TypeError                                 Traceback (most recent call last)
<ipython-input-2-c1ba59f25019> in <module>
----> 1 NNDescent(sparse.random(100, 100), metric="cosine").neighbor_graph

/usr/local/lib/python3.7/site-packages/pynndescent/pynndescent_.py in neighbor_graph(self)
   1153             result = (
   1154                 self._neighbor_graph[0].copy(),
-> 1155                 self._distance_correction(self._neighbor_graph[1]),
   1156             )
   1157         else:

/usr/local/lib/python3.7/site-packages/numba/npyufunc/dufunc.py in _compile_for_args(self, *args, **kws)
    189                     argty = argty.dtype
    190                 argtys.append(argty)
--> 191         return self._compile_for_argtys(tuple(argtys))
    192 
    193     def _compile_for_argtys(self, argtys, return_type=None):

/usr/local/lib/python3.7/site-packages/numba/npyufunc/dufunc.py in _compile_for_argtys(self, argtys, return_type)
    209             self._dispatcher, self.targetoptions, sig)
    210         actual_sig = ufuncbuilder._finalize_ufunc_signature(
--> 211             cres, argtys, return_type)
    212         dtypenums, ptr, env = ufuncbuilder._build_element_wise_ufunc_wrapper(
    213             cres, actual_sig)

/usr/local/lib/python3.7/site-packages/numba/npyufunc/ufuncbuilder.py in _finalize_ufunc_signature(cres, args, return_type)
    157         if cres.objectmode:
    158             # Object mode is used and return type is not specified
--> 159             raise TypeError("return type must be specified for object mode")
    160         else:
    161             return_type = cres.signature.return_type

TypeError: return type must be specified for object mode

This does not replicate in umap with small sparse arrays, but does for large ones:

u = UMAP(metric="cosine")

u.fit(sparse.random(100, 100))  # Runs fine
u.fit(sparse.random(5000, 100))  # Errors

Traceback

/usr/local/lib/python3.7/site-packages/pynndescent/pynndescent_.py:1012: UserWarning: Failed to correctly find n_neighbors for some samples.Results may be less than ideal. Try re-running withdifferent parameters.
  "Failed to correctly find n_neighbors for some samples."
---------------------------------------------------------------------------
TypeError                                 Traceback (most recent call last)
<ipython-input-7-c507b2e75faf> in <module>
----> 1 u.fit(sparse.random(5000, 100))

/usr/local/lib/python3.7/site-packages/umap/umap_.py in fit(self, X, y)
   1664                 self.low_memory,
   1665                 use_pynndescent=True,
-> 1666                 verbose=self.verbose,
   1667             )
   1668 

/usr/local/lib/python3.7/site-packages/umap/umap_.py in nearest_neighbors(X, n_neighbors, metric, metric_kwds, angular, random_state, low_memory, use_pynndescent, verbose)
    297                 verbose=verbose,
    298             )
--> 299             knn_indices, knn_dists = nnd.neighbor_graph
    300             rp_forest = nnd
    301         else:

/usr/local/lib/python3.7/site-packages/pynndescent/pynndescent_.py in neighbor_graph(self)
   1153             result = (
   1154                 self._neighbor_graph[0].copy(),
-> 1155                 self._distance_correction(self._neighbor_graph[1]),
   1156             )
   1157         else:

/usr/local/lib/python3.7/site-packages/numba/npyufunc/dufunc.py in _compile_for_args(self, *args, **kws)
    189                     argty = argty.dtype
    190                 argtys.append(argty)
--> 191         return self._compile_for_argtys(tuple(argtys))
    192 
    193     def _compile_for_argtys(self, argtys, return_type=None):

/usr/local/lib/python3.7/site-packages/numba/npyufunc/dufunc.py in _compile_for_argtys(self, argtys, return_type)
    209             self._dispatcher, self.targetoptions, sig)
    210         actual_sig = ufuncbuilder._finalize_ufunc_signature(
--> 211             cres, argtys, return_type)
    212         dtypenums, ptr, env = ufuncbuilder._build_element_wise_ufunc_wrapper(
    213             cres, actual_sig)

/usr/local/lib/python3.7/site-packages/numba/npyufunc/ufuncbuilder.py in _finalize_ufunc_signature(cres, args, return_type)
    157         if cres.objectmode:
    158             # Object mode is used and return type is not specified
--> 159             raise TypeError("return type must be specified for object mode")
    160         else:
    161             return_type = cres.signature.return_type

TypeError: return type must be specified for object mode

Versions:

pynndescent 0.4.5
umap 0.4.0rc1
scipy 1.4.1
numba 0.48.0

[lmcinnes]

It seems like the distance correction -- the last step of recalculating true, rather than proxy, distances is failing for some reason. This has worked for me in the past, so it is some combination of things at work here. I'll see if I can make a reproducer, or see an obvious solution in the code.

[lmcinnes]

I'm going to close this for now. Feel free to re-open it if there are lingering issues.

[dkobak]

Sounds great! Does this also resolve #94 by any chance?

[dkobak]

Great, thanks!