Indexing - performance warning - full index can result in a large number of pairs about recordlinkage

[rohitgarud]

@gajghaten I think the package uses logging to display warnings. Try the following code

import logging
logging.getLogger("recordlinkage").setLevel(logging.ERROR)

[gajghaten]

@rohitgarud Thanks! That worked!

The following helps too!

logger = logging.getLogger('recordlinkage')
logger.disabled = True

# Your code

logger.disabled = False

[rohitgarud]

@gajghaten Glad to help. But I think you should absolutely never use full index and always use blocking or sorted index. Full index will give (n choose 2 i.e. n*(n-1)/2) pairs, which increases quadratically with the number of records and slows down the record linkage or deduplication process significantly