Scaling the search capability -PoC with Elastic search about clip-rsicd

[sujitpal]

We didn't use the MEGA data these appear to have been added after we were done with the project. Not sure if one is a subset of the other. If you are going to retrain with more data, then might make sense to download both these archives and check.

And as I mentioned earlier, we are using NMSLib for searching in the HF-spaces demo.

[INF800]

Hmm, found this link from somewhere https://mega.nz/folder/wCpSzSoS#RXzIlrv--TDt3ENZdKN8JA

[sujitpal]

Hi @INF800 hope this is enough information to start:

• you can download the images and captions directly from the RSICD repository (https://github.com/201528014227051/RSICD_optimal) -- the link is also available from the README.md file for this project.
• once done, you can download one of our models (again link and example code available on README.md) and generate the vectors from the image-caption pairs.
• at this point I suspect you would be ready to upload them to your Elasticsearch instance. We used NMSLib for the demo (code and demo links available in README.md) but in another variation we used the Vespa engine, which offers text, vector and hybrid (text+vector) searching.

[INF800]

I am stuck at the first step currently. (Because I am also looking at the training code which I will use in future)

These two files are actually downloadable

1. The new download source of RSICD-MEGA

dataset.json (2.8MB)
imgs.rar (252.9MB)

2. The new download source of Sydney-captions and UCM-catpions-MEGA.

annotations_rsicd.rar (836KB)
RSICD_images.rar (459.8MB)

• The first file ( 252MB) is subset of second file (459MB) right? Or are they both different?
• And are we using only second file (459MB) for training the best model(s) ?

[INF800]

And also, can I know what are we using for searching in HF spaces? The speed is appreciable.

[nikky4D]

@sujitpal for the data used, does the RSICD data also contain Sydney and UCM-Merced data? If not, did you use all three in your training or only the RSICD data?

[sujitpal]

For the training I believe we used RSICD plus the Sydney and UCM-Merced datasets, but I will defer to @arampacha for the authoritative answer.

[arampacha]

IIRC the data is not overlapping and we used all 3 datasets. This is also consistent with model cards we have. And don't see a reason not to trust those ;)