Possible bug when len(labels) > 1 about recordlinkage

[J535D165]

Thanks for reporting this and your work on the PR.

I think the proposed solution is valid.

One thing to think about is: how should a compare function return 2D (or ND) outputs? I see that it was designed to return a ndarray (or frame). But maybe a tuple of numpy.array's or pandas.Series' is better. This is more consistent with the 2d inputs. Like here: https://github.com/J535D165/recordlinkage/blob/master/recordlinkage/base.py#L275-L276. Nevertheless, your code remains necessary to fix the issue.

Just curious, can you describe a situation where multiple columns are returned?

[gwerbin]

When you want to compute multiple scores off the same pair.

Here's an example from the code I was using pretty much verbatim:

import cytoolz as tz
import jellyfish as jf
import numpy as np
import recordlinkage as rl
import usaddress

## Make a sample data set

from recordlinkage.datasets import load_febrl1
febrl = load_febrl1()
febrl['street_address'] = febrl['street_number'] + ' ' + febrl['address_1']


## Set up deduplication

def _compare_addrs(addr1, addr2):
    """ Compare address components

    Align on address components and return:
      1) fraction of address components that matched
      2) average Levenshtein distance of matched components

    Not recommended for general use, but it works in this application.
    """
    if not addr1 or not addr2:
        return (np.nan, np.nan)
        
    d = {}
    keys1 = set(addr1)
    keys2 = set(addr2)

    for k in k1 | k2:
        k1 = addr1.get(k)
        k2 = addr2.get(k)
        if not k1 or not k2:
            d[k] = 0
        else:
            d[k] = jf.levenshtein_distance(a, b)
    
    return (len(keys1 & keys2) / min(len(keys1), len(keys2)),
            sum(d.values()) / len(d))

# Hack the return value into a 2D Numpy array
# (avoid using decorators so that Joblib pickling works)
compare_addrs = tz.compose(np.transpose,
                           np.array,
                           np.vectorize(_compare_addrs, otypes=(float, float)))

comp = rl.Compare(n_jobs=4)
comp.compare_vectorized(compare_addrs,
                        'addr_components',
                        'addr_components',
                        label=['addr_overlap', 'addr_dist'])


## Run deduplication

febrl['addr_components'] = febrl['street_address'].map(usaddress.parse, na_action='ignore')
comp.compute(febrl)

Obviously there's a lot of boilerplate code here, but the point is that it would be very inefficient to calculate each value separately.