Exception when taking intersection where both sets are empty. about hyperminhash

[kportertx]

Sorry for jerking this back and forth. Works as expected when only one set is empty, fails when both are empty.

test2.py

from hyperminhash import HyperMinHash

hmh = HyperMinHash(8, 6, 10, collision_correction="false")
hmh1 = HyperMinHash(8, 6, 10, collision_correction="false")

n_keys = 1000000

hmh.update(("{}".format(v) for v in range(n_keys)))

print("hmh intersect", hmh.intersection(hmh1))
print("hmh1 intersect", hmh1.intersection(hmh))

python test2.py
hmh intersect (0.0, 0.0, 0, 1015900.7572943214)
hmh1 intersect (0.0, 0.0, 0, 1015900.7572943214)

[kportertx]

Here is a potential way of solving this issue:
https://github.com/kportertx/hyperminhash/pull/1/files

The "_is_empty" flag could be useful in union and count as well.

[yunwilliamyu]

Ah, thanks for finding this edge case!

Instead of adding a new _is_empty flag like you suggest in the pull request, I'm personally leaning towards just checking if filled_buckets = 0, which is equivalent to _is_empty, and doesn't require any extra propagation code in things like add.

Thanks again for bringing it to my attention; I'll implement a fix this weekend.

[kportertx]

No problem, and thank you for providing a reference implementation for your paper.

[yunwilliamyu]

I just implemented a check in the Jaccard index code to return 0 when both sets are empty.

Of course, Jaccard index is undefined when both sets are empty, because it's 0/0. However, we simply do not want to return a NaN. This definition makes the intersection code operate as normal without any extra exception checking.

[kportertx]

After more thought on this, I think NaN may be the most appropriate response.

It may be true that two empty sets the same, but whether or or not it should return 0 or 1 is use case dependent, and in most cases this probably should be an error. NaN provides the most information and likely this scenario wasn't intended. I have opted to return float('NaN').