yinboc / dgp

Rethinking Knowledge Graph Propagation for Zero-Shot Learning, in CVPR 2019

Dense Graph Propagation

The code for the paper Rethinking Knowledge Graph Propagation for Zero-Shot Learning.

Citation

@inproceedings{kampffmeyer2019rethinking,
  title={Rethinking knowledge graph propagation for zero-shot learning},
  author={Kampffmeyer, Michael and Chen, Yinbo and Liang, Xiaodan and Wang, Hao and Zhang, Yujia and Xing, Eric P},
  booktitle={Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition},
  pages={11487--11496},
  year={2019}
}

Requirements

• python 3
• pytorch 0.4.0
• nltk

Instructions

Materials Preparation

There is a folder materials/, which contains some meta data and programs already.

Glove Word Embedding

1. Download: http://nlp.stanford.edu/data/glove.6B.zip
2. Unzip it, find and put glove.6B.300d.txt to materials/.

Graphs

1. cd materials/
2. Run python make_induced_graph.py, get imagenet-induced-graph.json
3. Run python make_dense_graph.py, get imagenet-dense-graph.json
4. Run python make_dense_grouped_graph.py, get imagenet-dense-grouped-graph.json

Pretrained ResNet50

1. Download: https://download.pytorch.org/models/resnet50-19c8e357.pth
2. Rename and put it as materials/resnet50-raw.pth
3. cd materials/, run python process_resnet.py, get fc-weights.json and resnet50-base.pth

ImageNet and AwA2

Download ImageNet and AwA2, create the softlinks (command ln -s): materials/datasets/imagenet and materials/datasets/awa2, to the root directory of the dataset.

An ImageNet root directory should contain image folders, each folder with the wordnet id of the class.

An AwA2 root directory should contain the folder JPEGImages.

Training

Make a directory save/ for saving models.

In most programs, use --gpu to specify the devices to run the code (default: use gpu 0).

Train Graph Networks

• SGCN: Run python train_gcn_basic.py, get results in save/gcn-basic
• DGP: Run python train_gcn_dense_att.py, get results in save/gcn-dense-att

In the results folder:

• *.pth is the state dict of Graph Networks model
• *.pred is the prediction file, which can be loaded by torch.load(). It is a python dict, having two keys: wnids - the wordnet ids of the predicted classes, pred - the predicted fc weights

Finetune ResNet

Run python train_resnet_fit.py with the args:

• --pred: the .pred file for finetuning
• --train-dir: the directory contains 1K imagenet training classes, each class with a folder named by its wordnet id
• --save-path: the folder you want to save the result, e.g. save/resnet-fit-xxx

(In the paper's setting, --train-dir is the folder composed of 1K classes from fall2011.tar, with the missing class "teddy bear" from ILSVRC2012.)

Testing

ImageNet

Run python evaluate_imagenet.py with the args:

• --cnn: path to resnet50 weights, e.g. materials/resnet50-base.pth or save/resnet-fit-xxx/x.pth
• --pred: the .pred file for testing
• --test-set: load test set in materials/imagenet-testsets.json, choices: [2-hops, 3-hops, all]
• (optional) --keep-ratio for the ratio of testing data, --consider-trains to include training classes' classifiers, --test-train for testing with train classes images only.

AwA2

Run python evaluate_awa2.py with the args:

• --cnn: path to resnet50 weights, e.g. materials/resnet50-base.pth or save/resnet-fit-xxx/x.pth
• --pred: the .pred file for testing
• (optional) --consider-trains to include training classes' classifiers

dgp's Issues

Time of your train stage

in your paper, your model is trained for 3000 epochs, how much time do you spend in it

Questions about evaluation on awa2

Q1：If i want to evaluate on awa2，should i still train on imagenet？

About datasets download

Thanks for your awesome work! Can you provide the download link for ImageNet and AWA2 datasets?

How to deal with words without valid word embedding

Hi,

I noticed that some words in the test set don't have valid word embedding. How to deal with them? Are these categories ignored during evluation?

Thanks!

Hi! I saw in another issue that you explain the funtion 'induce_parent', saying its purpose is to attach imagenet21k and related node to the xml-node. I find this line confusing. Base on your explanation in the aforemetioned issue, I presume that this line should check whether one parent node belongs to xml-node but find it checking imagenet21k-node, and I'm therefore lost. Another thing is that there doesn't seem to be a line attaching the missing node itself to the xml-node. Could you enlighten me about this?

DGP/materials/make_induced_graph.py

Line 40 in b497ebb

if p not in vis:

fall2011.tar

How can I find the file fall2011.tar

What does imagenet-xml-wnids.json come from?

Hi,

I noticed that there is a "imagenet-xml-wnids.json" file used in make_induced_graph.py.
Would you mind to let me where does it come from and what is it used for?

Best Regards,

How to make graph on AWA2 dataset?

Did I make awa2_induced_graph by replace input and output arguments? And what the meaning of induce_parents function? Also I notice that you are construct graph with all (train + test) categories together and Would that obtain test information during training?

help

I am a newbie..............when I train graph networks,I doubt about these error and I don not know how to resolve them:
Traceback (most recent call last):
File "train_gcn_basic.py", line 61, in
gcn = GCN(n, edges, word_vectors.shape[1], fc_vectors.shape[1], hidden_layers).cuda()
File "E:\实验\DGP-master\models\gcn.py", line 51, in init
adj = spm_to_tensor(adj)
File "E:\实验\DGP-master\utils.py", line 68, in spm_to_tensor
(sparse_mx.row, sparse_mx.col))).long()
TypeError: can't convert np.ndarray of type numpy.int32. The only supported types are: double, float, float16, int64, int32, and uint8.

About Result

Thank you very much for your contribution!
I have tested your codes with default settings(with finetuning), and the best preformance on imageNet(2-hops) is 25.49%/25.82%(SGCN/DGP).
It seems there is a little degradation with your paper(26.2%/26.6% for SGCN/DGP).Do I miss something or you have some other tricks?
I find that one tenth of the pictures were randomly picked in 'evaluate_imagenet.py'. Maybe that's the reason?
By the way, I'll be very appriciate it if you can provide the 'best.pred'.
Thank you!

Dataset splits for awa2

Hi,

Thanks for the release of your code, which gives me a lot of references in zero-shot learning literature.

I found that the dataset split in the awa2-split.json file in this repo is different from the split shown on the awa2 dataset website: https://cvml.ist.ac.at/AwA2/.

Can I ask if you use different splits and if so why you use different splits?

Comparison to Graph Attention Networks

In 6.4 you made a comparison between 1-hidden-layer GCN with attention mechanism and 1-hidden-layer GCN. Why the results of GAT-1 are so close to GCN-1 ? The attention mechanism really works?

Details of SGCN

Thanks for your contributions on this work, but I have some questions about SGCN.

In the paper, it mentions that "As DGP, our SGCN model makes use of the proposed two-stage finetuning approach", but did it use the descendant and ancestor patterns and weighting schema, like DGP as well?

From the code, it seems that SGCN is very similar to GCNZ. Their differences mainly are the number of hidden layers, training epochs and implementation details, is it corret?

Thanks very much!

Expected finetuning time

How long does finetuning (20 epochs) take? I am at epoch 11 and it took about a day, so I expect the total tuning time will be roughly two days. Does that sound right?

ImportError: No module named nltk.corpus

root@aca4066e21cd:/tmp/pycharm_project_546/materials# python make_induced_graph.py
Traceback (most recent call last):
File "make_induced_graph.py", line 4, in
from nltk.corpus import wordnet as wn
ImportError: No module named nltk.corpus

About the GPU memory

How much GPU memory do you use when training the dense gcn?

I am using the GTX2080TI whose memory is 11G. But it always raises an error: RuntimeError: CUDA out of memory.

imagenet

Hi! Thanks for your code!
ImageNet dataset is unable to download all 22k images, could you please share your dataset to us?
Thanks!

how to know 'cat''s descendants and ancestors?

Where do you get this prior information?

sorry for my simple question, I'm a newbie for this topic.

How to get "2-hops", "3-hops" and "all-hops" test dataset

Hi, thanks for sharing codes. I tried to get "2-hops", "3-hops" and "all-hops" dataset (I followed this repo https://github.com/JudyYe/zero-shot-gcn/blob/master/DATASET.md), but most of URLs in ImageNet are not valid right now. I only can download ImageNet 2012 as whole (120 GB). Any tips for that? Thanks

Why val_loss is always 0?

hi,
I get val_loss=0.0000 when run "python train_gcn_*.py". Why is it ?
Thank you.

What‘s the difference between GPM and GCNZ with few layer

Hi,

It's a great work, but i have some confusions about GPM and GCNZ.

From the reported result in paper, GPM has already received a huge improvement compared to the SOTA method GCNZ, however the paper just said GPM only use single layer compared to GNCZ.

However in GNCZ, author's experiment indicate that addition layers bring the improvement results. So i'm very confused.

Can i ask more clear difference between GPM and GCNZ?

Thanks very much!