algoprog / quin

An easy to use framework for large-scale fact-checking and question answering

Hi, I'm Chris 👋

I'm a PhD student in Information Retrieval at the Center for Intelligent Information Retrieval at University of Massachusetts Amherst, supervised by Hamed Zamani. I’m currently doing research in conversational search.

👔 linkedin | 🐤 twitter | 📰 google scholar | 🏠 website

---

📑 Latest Research

• 📜 ProCIS: A Benchmark for Proactive Retrieval in Conversations in SIGIR 2024
• 📜 Simulating Task-Oriented Dialogues with State Transition Graphs and Large Language Models (preprint)
• 📜 Revisiting Open Domain Query Facet Extraction and Generation in ICTIR 2022
• 📜 Improving Evidence Retrieval for Automated Explainable Fact-Checking in NAACL 2021
• 📜 Latent Retrieval for Large-Scale Fact-Checking and Question Answering with NLI training in ICTAI 2020

---

💻 Latest Projects

• Faspect: A library with various model implementations for open domain query facet extraction and generation
• Quin: A framework for large-scale fact-checking and question answering

---

quin's Issues

Python version for installation

Hi, sorry to disturb haha. When I cloned the repo and tried to follow the instructions on ReadMe, the installation of the requirements.txt always fails. I tried using Python 2.7, Python 3.8 and Python 3.6, but Python 3.8 failed to install scipy==1.2.1, Python 2.7 and Python 3.6 failed to install faiss_cpu==1.6.3.

Thus, I wonder could u pls offer me some help on more details(like the python version) of installation? Thank you so much! : )

System: Windows 10
IDE: PyCharm Community 2020.01

Demo interface

Hi,
Nice project! I am working on a similar problem and I was wondering if your Web interface used in the demo is available anywhere. Cheers!

Re-Training models for custom database

Kindly provide the script to train the model on the custom database.

ModuleNotFoundError: No module named 'models.text_encoder'

I have downloaded NLI model weights and Densed Encoder FC in the folder model/weights folder.
After extracting the relevant folder in the model/weights directory, it looks like this:

I also had to modify the following line otherwise it would not detect my model:
self.text_embedding_model = SentenceTransformer('{}/encoder/qrbert-factcheck'.format(models_path), device=device)

Now, I am getting the following error:

Can you help resolve this error and show exactly how to configure your codebase?

Questions about datasets used to train dense encoder models

Hello,
I read both of your papers that are linked in this repo and went through the code, and I'm a little bit confused about what datasets were used to train each of the dense encoder models that are listed in the README.
From reading your papers, I'm assuming that (1) the dense encoder FC model was trained on NLI and Factual NLI datasets; (2) the dense encoder QA model was pretrained on NLI and Factual NLI and fine tuned on MSMARCO; and (3) the dense encoder M model was trained only on the Factual NLI+ dataset. Could you please confirm if this is correct? Thanks a lot!

Sparse Index not building properly

Hello,

I have found a small error in your building of the sparse index, There is a small mistake in your _calc_idf funtion for sparse indexes.

In this function you are using an eps value to replace the negative idf values. However your code is a little bit faulty.

1. Firstly you are using r_freq to determine negative idf value, why not just use the idf value itself as a check?
2. your statement:

if r_freq > 0.5:
       continue

Is not correct because what is happening here is that if the r_freq > 0.5 (or idf is negative) it skips the rest of the code under the if statement including:

if idf < 0:
   negative_idfs.append(word)

So those words whose idf's are negative never appear in the idf dictionary. You can confirm this by checking your idf dictionary for a word which exists in more than 50% of the corpus.

I would suggest changing the _calc_idf function to:

    def _calc_idf(self, nd):
        """
        Calculates frequencies of terms in documents and in corpus.
        This algorithm sets a floor on the idf values to eps * average_idf
        """
        # collect idf sum to calculate an average idf for epsilon value
        idf_sum = 0
        # collect words with negative idf to set them a special epsilon value.
        # idf can be negative if word is contained in more than half of documents
        negative_idfs = []
        for word, freq in nd.items():
            idf = math.log(self.corpus_size - freq + 0.5) - math.log(freq + 0.5)
            #r_freq = float(freq) / self.corpus_size
            if idf < 0:
                negative_idfs.append(word)
                continue
            self.idf[word] = idf
            idf_sum += idf      
        self.average_idf = idf_sum / len(self.idf)

        eps = self.epsilon * self.average_idf
        for word in negative_idfs:
            self.idf[word] = eps

This will ensure that the negative idf values are replaced with the value of epsilon and that they are present in the idf dictionary.