lengoanhcat / consistent_gqa

Train a consistent general question and answer machine

Logically Consistent Loss for Visual Question Answer

This code repository contains codes and scripts for running experiments, mentioned in the following paper.

Logically Consistent Loss for Visual Questions Answering, Anh-Cat Le-Ngo, Truyen Tran, Santu Rana, Sunil Gupta, Svetha Venkatesh

Key Logics

1. C-rule:

    if 'C2' in config.tflRules:
        constraints.append(
            tfl.constraint(
                "forall p: forall q: (isDiff(p,q) and isGoodTask(p,q)) -> (isGoodAns(p,q))")
            /tf.cast(tf.square(self.batchSize),tf.float32))
        numCW2 += 1

2. E-rule:

    constraints_str = [
        "queryObj(p) -> queryAttrObj(q)",
        "queryAttrObj(p) -> existAttrTrue(q)",
        "existAttrTrue(p) -> existAttrOrTrue(q)",
        "existAttrTrue(p) -> existNotTrue(q)",
        "existAttrTrue(p) -> existAttrNotTrue(q)",
        "existAttrOrTrue(p) -> existNotOrTrue(q)",
        "existNotOrTrue(p) -> existOrTrue(q)",
        "existOrTrue(p) -> existTrue(q)",
        "queryAttrObj(p) -> queryObj(q)",
        "queryNotObj(p) -> existNotTrue(q)",
        "existNotTrue(p) -> existTrue(q)",
        "existAttrNotTrue(p) -> existTrue(q)",
        "existAndTrue(p) -> existTrue(q)",
        "existRelTrue(p) -> existTrue(q)",
        "existRelTrue(p) <-> verifyRelTrue(q)",
        "verifyRelTrue(p) <-> queryRel(q)",
        "verifyRelTrue(p) <-> chooseRel(q)",

        "existOrFalse(p) -> existFalse(q)",

        "existFalse(p) -> existNotFalse(q)",
        "existFalse(p) -> existAttrNotFalse(q)",

        "existFalse(p) -> existRelFalse(q)",
        "existFalse(p) -> existAndFalse(q)",

        "existNotFalse(p) -> existAttrFalse(q)",
        "existAttrNotFalse(p) -> existAttrFalse(q)",

        "existNotOrFalse(p) -> existNotFalse(q)",
        "existNotOrFalse(p) -> existAttrNotFalse(q)",

        "existNotOrFalse(p) -> existAttrOrFalse(q)",
        "existAttrOrFalse(p) -> existAttrFalse(q)",

        "verifyAttrsTrue(p) -> verifyAttrTrue(q)",
        "verifyAttrAndTrue(p)-> verifyAttrTrue(q)",
        "verifyAttrTrue(p) -> queryAttr(q)",
        "queryAttr(p) -> verifyAttrFalse(q)",
        "queryAttr(p) -> chooseAttr(q)",
        "verifyAttrFalse(p) -> verifyAttrAndFalse(q)",
        "verifyAttrAndFalse(p) -> chooseAttr(q)",
        "chooseAttr(p) <-> chooseObj(q)",

        "verifyGlobalTrue(p) -> verifyGlobalFalse(q)",
        "verifyGlobalTrue(p) <-> queryGlobal(q)",
        "verifyGlobalFalse(p) -> chooseGlobal(q)",

        "compare(p) -> common(q)",
        "common(p) -> twoSameTrue(q)",
        "twoSameTrue(p) <-> twoDiffFalse(q)",

        "twoSameFalse(p) <-> twoDiffTrue(q)",

        "allSameTrue(p) <-> allDiffFalse(q)",
        "allSameFalse(p) <-> allDiffTrue(q)"
    ]

    if 'E4' in config.tflRules:
        entailment_str = '({})'.format(constraints_str[0]) + ''.join([
            ' or ({})'.format(task) for task in constraints_str[1:]])
        constraints.append(
            tfl.constraint(
                "forall p: forall q: (isDiff(p,q) and ({})) ->  isGoodAns(p,q)".format(entailment_str))\
            /tf.cast(tf.square(self.batchSize),tf.float32))
        numCW2 += 1

Prerequisites

1. Check out the MACnet architecture for VQA MAC
2. Follow the setup instructure for downloading and preprocessing original data

Hybrid Data Preprocess

1. Setup a mongodb server (>= 4.2.3) by following this instruction

2. Use the bash script import_gqa_into_mongodb.sh to import GQA questions and scenegraphs database from the following json files into MongoDB collections with their correspondent names

   • train_balanced_questions.json
   • train_all_questions/
   • train_sceneGraphs.json
   • val_balanced_questions.json
   • val_all_questions.json
   • val_sceneGraphs.json
   • testdev_balanced_questions.json
   • testdev_all_questions.json
   • test_balanced_questions.json
   • test_all_questions.json

3. Run the Python Script generate_multilabel_data_parallel.py to generate 16 partitions of family-batch data for train dataset

4. Run an experiment with run_experiment.sh

Analyze prediction after running experiment

1. Use extract_preds.py to extract predictions from valPredictions.json file into a simpler dictionary structure of {qid: answer} in txt files i.e. original.txt, hybrid.txt and logic.txt
2. Use Jupyter notebook postexp_analysis.ipynb to analyze answers in family-batch