vintasoftware / deduplication-slides

"1 + 1 = 1 or Record Deduplication with Python" Jupyter Notebook

⚠️ Looking for PyCon US 2020 slides? They're on pycon-2020 branch. Please click here!

1 + 1 = 1 or Record Deduplication with Python

Jupyter Notebook from the talk "1 + 1 = 1 or Record Deduplication with Python", presented at PyBay 2018 and PyGotham 2018. The slides.ipynb version was presented at PyBay, while the slides-reduced.ipynb version was presented at PyGotham.

Running (Binder)

It's possible to run the slides-reduced.ipynb version online! Click here:

Errors on Binder?

In case you face errors on Binder, try again later or use the non-interactive rendered version.

Running (Local)

Install libpostal (instructions here) and pip install -r requirements.txt. Run jupyter notebook

---

🇧🇷 1 + 1 = 1 ou Pareamento de Registros com Python

Jupyter Notebook da talk "1 + 1 = 1 ou Pareamento de Registros com Python", apresentada na Python Brasil 2019.

Rodando (Binder)

É possível rodar online, sem precisar instalar nada no seu computador! Clique aqui:

Erros no Binder?

Caso o Binder não funcione, tente novamente depois ou use a versão renderizada sem interação.

Rodando (Local)

Instalar libpostal (instruções aqui) e pip install -r requirements.txt. Rodar jupyter notebook e abrir slides-pt-br.ipynb

deduplication-slides's Issues

Question on Random Forest classifier as learner (rf_learner.py)

Hi @fjsj, I bumped into your repo on the PyCon2020 slides and am trying to use that on my work, where I am currently using dedupe to perform some entity resolution for my data. If you don't mind, I would like to ask your question regarding your custom implementation of Random Forest that inherits the RLRLearner class from dedupe (rf_learner.py):

class RFLearner(RLRLearner):

    def __init__(self, data_model, *args, **kwargs):
        self.svm_classifier = _build_model()
        super().__init__(data_model, *args, **kwargs)

    def fit(self, X, y):
        y = numpy.array(y)

        # This replicates Dedupe's behavior, adapting it to sklearn:
        # if there are only non-matching examples on y,
        #     grab a random record and consider it as a match with itself
        # if there are only matching examples on y,
        #     grab a random pair and consider it as a non-match
        # Also, if both X and y are empty, do both things above.
        # This happens on active learning when there's no existing training_pairs.
        if not y.any():
            random_pair = random.choice(self.candidates)
            exact_match = (random_pair[0], random_pair[0])
            X = numpy.vstack([X, self.transform([exact_match])])
            y = numpy.concatenate([y, [1]])
        if numpy.count_nonzero(y) == len(y):
            random_pair = random.choice(self.candidates)
            X = numpy.vstack([X, self.transform([random_pair])])
            y = numpy.concatenate([y, [0]])

        self.y = y
        self.X = X
        self.svm_classifier.fit(X, y)

In the fit() method, you mentioned that it is replicating Dedupe's behavior to initialize the X and y when there non-matching or matching examples on y. It makes perfect sense to me, but I am struggling to find where this is referenced to on Dedupe's codebase, could you please help me out with that?

Thank you so much for your time! :)