Include URL to smaller sized versions of images about dataset

[gkrasin]

Hi Olivier,

while, in general, I agree it's a good idea to provide a way to download smaller sizes, due to a way these images have been collected, there's no information about smaller thumbnails on our side. So, it's more of a call to the community.

The dataset viewer from @tzatter shows smaller-sized thumbnails from Flickr, which means there's a way to convert from the originals urls to thumbnails urls.

[olivierbouwman]

Thank you for the quick answer. I have so far only been able to get the smaller sized URLs using Flickr API calls which will take just over 100 days with the rate limits they have. Hopefully @tzatter has a solution.

[gkrasin]

Hold on, I might have an idea. It will likely take 2-3 days, but it should work for the majority of the images.

I will aim for "Medium 640" / _z thumbnails. In total, the thumbnails of this size should be below 1TB (so, ~20x smaller than the originals), so it would be much easier to download them. Any smaller thumbnails can be created by resizing these. For larger thumbnails crawling originals does not seem to be wrong.

[gkrasin]

Okay, the thumbnail URLs are coming:

https://c5.staticflickr.com/1/92/230781821_dff139d17b_z.jpg?zz=1
https://c5.staticflickr.com/8/7416/9940862744_4705662f07_z.jpg
https://c2.staticflickr.com/2/1282/4599122146_6e38260b2d_z.jpg
https://c3.staticflickr.com/8/7038/6947057537_dc8756fcda_z.jpg
https://c4.staticflickr.com/4/3873/14152766838_8030725dca_z.jpg
https://c6.staticflickr.com/1/174/421050398_5c63fab327_z.jpg?zz=1
https://c1.staticflickr.com/7/6210/6099515726_3a6a6cf44d_z.jpg
https://c2.staticflickr.com/8/7301/8746901660_2b2870ce20_z.jpg
https://c1.staticflickr.com/5/4052/4241923013_4e5b650ae2_z.jpg
https://c6.staticflickr.com/3/2672/3836490444_2e6b77e04c_z.jpg
https://c1.staticflickr.com/8/7343/13890442407_0f584a314f_z.jpg
https://c5.staticflickr.com/5/4134/4936718791_5e99d460aa_z.jpg
https://c6.staticflickr.com/2/1065/1358019429_a8c26ffacf_z.jpg
https://c3.staticflickr.com/6/5590/14864523078_63cd5c9941_z.jpg
https://c8.staticflickr.com/4/3254/3138117555_07fa474b2e_z.jpg
https://c6.staticflickr.com/8/7266/7630204884_e381669106_z.jpg
https://c3.staticflickr.com/4/3917/15159280745_e1821228ef_z.jpg
https://c6.staticflickr.com/7/6127/5940849494_f35dcf5fd8_z.jpg
https://c5.staticflickr.com/3/2663/13032564665_4cdd837a21_z.jpg

The batch job is expected to finish within 90 minutes, which means I should be able to publish images.csv with these URLs on Friday morning (Pacific time). Sadly, no sizes / md5 sums for the thumbnails, as no actual crawl of these thumbnails have been performed. May be later.

[olivierbouwman]

Awesome, thanks so much @gkrasin! Looking forward to this update. Out of curiosity, how did you get the secret for the other sizes?

[gkrasin]

Hi @olivierbouwman,

I didn't. In fact, I have not used Flickr API at all. I just realized that I forgot to delete crawled landing pages (which I crawled quite a long of time ago), and then extracted the thumbnail URLs by going through all of these HTMLs.

I have released the new version of images.csv. Can you please take a look, if that works for you?

[olivierbouwman]

Technically it seems to work great. Unfortunately, the download speed seems to be worse. Originals download at about 170 images a minute for me (one month download time which is doable), the new medium size downloads at about 17 images a minute (one year download time). I'm assuming flickr rate limits downloads, perhaps they rate limit original sizes differently than the smaller sizes? I tried to find some evidence of this by searching but no luck.

[gkrasin]

Interesting. I don't have any guesses about the speed at the moment.

[gkrasin]

I also don't observe that locally; so it could be something with the way you download.

[gkrasin]

I have spent some time trying to get sizes and md5 sums for these thumbnails. It has been discovered that the thumbnails are not really stored on the Flickr side and they are generated on the fly in a non-deterministic manner. What was the most surprising is that sometimes the image resolution might be different.

It means that publishing md5 / size numbers is impossible. It also means that whoever downloads these thumbnails from Flickr, get a lottery ticket with every image. I am actually seriously considering dropping the urls to these thumbnails, but will decide later next week.

[olivierbouwman]

I always assumed smaller sizes were premade by flickr because of some text in the API documentation (https://www.flickr.com/services/api/misc.urls.html mentions that certain sizes don't exist if the photo was taken before a certain date) but your test results seem to indicate otherwise. Perhaps different flickr servers have different settings for generating the thumbnails which sounds strange to say the least. I'm just sticking to downloading the original sizes for now since that seems fastest in our case. Thanks so much for looking into this!