Function of simply calculating pairwise similarity without matching about string_grouper

[ParticularMiner]

Hi @KiraJYQiu,

This requires much more computation than what I actually need.

You are right. In computational terms, what you want are only the diagonal elements of the product of two sparse matrices of the same dimensions/shape. See PR #40 for a possible solution.

I'm not sure exactly how you do it, but I feel I should warn you in advance (so ignore this paragraph if it is not relevant) — modifying the string variables, in general, changes the corpus used to vectorize the strings whose pair-similarities are to be computed. So the new similarity-scores of the modified strings would be based on a corpus different from the one on which the original string-similarities were based and therefore the old and new similarity-scores should not be directly compared. Instead, all the original strings should also be included together with the modified strings for the pairwise similarity-computation. I hope this helps.

[KiraJYQiu]

Thank you for your reply
First, for your solution in #40, I just want to make sure that compute_pairwise_similarities uses the same options as other functions in string_grouper. By default it should also exclude "[,-./]|\s" before calculating similarity so that their algorithms are consistent.

Second, I know what I do. I generate new variables for modifying the original strings. For some databases that I need to link via fuzzy match, the entity names can be very noisy (e.g. FEC data’s employer name includes occupation/job titles). Excluding this noise by a few sentences of codes can’t be exactly accurate: it can exclude what should be excluded but also exclude what should be kept. Thus I may need two similarity scores (for nosier and cleaner strings) for comparison. Again thanks for such detailed concerns.

[ParticularMiner]

I just want to make sure that compute_pairwise_similarities uses the same options as other functions in string_grouper. By default it should also exclude "[,-./]|\s" before calculating similarity so that their algorithms are consistent.

Yes, it does, as you can confirm by inspecting the source code.

All the best as you cleanup your databases!