FATAL ERROR: Error::New napi_get_last_error_info about hnswlib-node

[yoshoku]

This error seems to occur if a variable inside the filter function was deleted. I reproduced this error with the code below:

const hnswlib = require('hnswlib-node');

const index = new hnswlib.HierarchicalNSW('l2', 3);
index.initIndex({ maxElements: 4 });

index.addPoint([1, 2, 3], 1)
index.addPoint([4, 5, 6], 2)
index.addPoint([1, 2, 3], 3)
index.addPoint([7, 8, 9], 4)

arr = [1, 2]

const filterFnc = (label) => arr.includes(label);

delete arr;

console.table(index.searchKnn([1, 2, 3], 2, filterFnc));

$ node issue_132.js
FATAL ERROR: Error::New napi_get_last_error_info
 1: 0x104dd69f8 node::Abort() [/Users/foo/.anyenv/envs/nodenv/versions/20.3.1/bin/node]
 2: 0x104dd6ae4 node::OOMErrorHandler(char const*, v8::OOMDetails const&) [/Users/foo/.anyenv/envs/nodenv/versions/20.3.1/bin/node]
 3: 0x104da5af4 napi_open_callback_scope [/Users/foo/.anyenv/envs/nodenv/versions/20.3.1/bin/node]
 4: 0x11a71dd98 Napi::Error::Error(napi_env__*, napi_value__*) [/Users/foo/test_hnswlib-node/node_modules/hnswlib-node/build/Release/addon.node]
 5: 0x11a71d5ec Napi::Error::New(napi_env__*) [/Users/foo/test_hnswlib-node/node_modules/hnswlib-node/build/Release/addon.node]
 6: 0x11a72d140 Napi::Value::ToBoolean() const [/Users/foo/test_hnswlib-node/node_modules/hnswlib-node/build/Release/addon.node]
 7: 0x11a72d0cc CustomFilterFunctor::operator()(unsigned long) [/Users/foo/test_hnswlib-node/node_modules/hnswlib-node/build/Release/addon.node]
...

It is recommended to check if any variables in the filter function have been freed.

[nrathi]

Thanks for that repro!

I definitely don't delete any variables directly, I let everything get garbage collected. Furthermore, I experience the issue as soon as I restart my development server (after hours and hours of previous starts/stops). The issue persists for some time (maybe on the order of minutes?), and then resolves itself.

[nrathi]

Update: I figured out the problem, and it was exactly along the lines of what you said.

I was creating a Dictionary of Sets

const availableIndices = mapValues(groupBy(array, item => item.group), item => item.index)

Then, I was iterating through some other items and trying to use this in the filter function:

for (const item of items) {
  const searchResult = hsnwIndex.searchKnn(
    embedding,
    NUM_NEIGHBORS,
    (index) => availableIndices[item.group].has(index),
  );
}

But if the group hadn't been seen before, then this would throw the error because availableIndices[item.group] is undefined.

Thanks for your help.