Missing images about dataset

[gkrasin]

I think it's normal. Around 0.6% of the images have gone from Flickr since the time they have been discovered. I have a non-urgent plan to identify such urls and update the tarballs again, but for now, just ignore such cases as "expected".

Are your stats the same (0.6%) or you see a higher failure rate?

[spawnflagger]

It's < 1%, but don't know the exact count yet. I'll modify my download script to create "working" and "missing" CSVs. I can forward those on to you if that'll help.

[gkrasin]

@spawnflagger It will be useful and will save me some time. Thank you for the offer to help.

[spawnflagger]

Hi @gkrasin , turns out the error rate was ~ 0.75%. What is the best way to send you the "error" CSV files?

• train (9,011,219)
  • good = 8,943,585
  • failed = 67,634
    • MD5 Mismatch = 60,908
    • HTTP Error 502 = 5,991
    • HTTP Error 404 = 481
    • HTTP Error 504 = 232
    • HTTP Error 500 = 22
• validation (167,056)
  • good = 165,762
  • failed = 1,294
    • HTTP Error 500 = 984
    • MD5 Mismatch = 310

[gkrasin]

@spawnflagger thank you for collecting these stats! The easiest way would be to send the CSV files via email to [email protected] or share them on anywhere on the web and send me a link.