Maybe corrupt tar files about 3d_street_view

[amir32002]

You can check the md5 hash of the downloaded files to make sure they're not corrupt. Below are the hash codes for the file you are facing an issue with:

Hashes [base64] for dataset_aligned/0075.tar:
Hash (crc32c): 2MTpng==
Hash (md5): xcNixKPq8KxfiICqvg5JKA==

Hashes [base64] for dataset_unaligned/0075.tar:
Hash (crc32c): 54tfnw==
Hash (md5): TTApEJlRC6Vh+MegQZQu0w==

However, as you said individual files do not limit using the dataset. The images and meta files are stored in a per-image basis. Whichever files you cant access, just ignore them and the corresponding images.

[dantkz]

File dataset_unaligned/0069.tar is corrupt on Google Bucket Storage.

The crc32c hash I have is 8vBh2w== which is computed using this script:

import base64
import crcmod

def get_file_crc32c(filename):
    """ Compute crc32c hash for a file """
    file_bytes = open(filename, 'rb').read()
    crc32c = crcmod.predefined.Crc('crc-32c')
    crc32c.update(file_bytes)
    return base64.b64encode(crc32c.digest()).decode('utf-8')

if __name__ == '__main__':
    print(get_file_crc32c('dataset_unaligned/0069.tar'))

The downloaded file hash matches the hash from Google Bucket Storage:

$gsutil ls -L gs://streetview_image_pose_3d/dataset_unaligned/0069.tar
gs://streetview_image_pose_3d/dataset_unaligned/0069.tar:
    Creation time:          Sat, 07 Oct 2017 11:37:52 GMT
    Update time:            Sat, 07 Oct 2017 11:37:52 GMT
    Storage class:          MULTI_REGIONAL
    Content-Language:       en
    Content-Length:         18852484576
    Content-Type:           application/x-tar
    Hash (crc32c):          8vBh2w==
    Hash (md5):             CnwIZFSsZ7YAeD05DKcDJg==
    ETag:                   CLHjlpq23tYCEAE=
    Generation:             1507376272617905
    Metageneration:         1
    ACL:                    ACCESS DENIED
        Note:               You need OWNER permission on the object to read its ACL
TOTAL: 1 objects, 18852484576 bytes (17.56 GiB)

This file tar-bombs and generates a bunch of badly encoded files:

$ls -l
total 129427628
---------- 1 1028 users           0 Jan  1  1970 Ýô�$æ&�
drwxr-xr-x 1 1028 users    17640840 Sep 24  2016 0004
-rw-r--r-- 1 1028 users 10399212396 Oct  7  2017 0004.tar
drwxr-xr-x 1 1028 users    14693070 Sep 25  2016 0069
-rw-r--r-- 1 1028 users 18852484576 Oct  7  2017 0069.tar
-rw-r--r-- 1 1028 users 22280379533 Oct  7  2017 0070.tar
-rw-r--r-- 1 1028 users 17353088272 Oct  7  2017 0072.tar
-rw-r--r-- 1 1028 users 10234234075 Oct  7  2017 0075.tar
drwxr-xr-x 1 1028 users    43319920 Sep 25  2016 0080
-rw-r--r-- 1 1028 users 16946239492 Oct  7  2017 0080.tar
drwxr-xr-x 1 1028 users    44956100 Sep 25  2016 0081
-rw-r--r-- 1 1028 users 18961213955 Oct  7  2017 0081.tar
drwxr-xr-x 1 1028 users    34037640 Sep 24  2016 0084
-rw-r--r-- 1 1028 users 12923984178 Oct  7  2017 0084.tar
drwxr-xr-x 1 1028 users     9207520 Sep 24  2016 0097
-rw-r--r-- 1 1028 users  4583031254 Oct  7  2017 0097.tar
---------- 1 1028 users           0 Jan  1  1970 0²:_øK�üùÿ
---------- 1 1028 users           0 Jan  1  1970 Þ('�1]�¹ã��='s�.ñ�VÏñ�õ§.W'æçÖ�dÊñ�Bä�WA�ÏÿÙ
---------- 1 1028 users           0 Jan  1  1970 
�2¬¸4�)ÊêÇÿÙ
---------- 1 1028 users           0 Jan  1  1970 �¤�Äò`3C,düè¸æ\��èyé�ÍoÄskP��í;��k�.ÚHÃ`���
9>§° q@�ÿÙ
---------- 1 1028 users           0 Jan  1  1970 5Ä�m$a°GÈ����SØ�8¦�ÿÙ
---------- 1 1028 users           0 Jan  1  1970 �í92y®$»i#
�>D< äú�À�Å�¹ÿÙ
---------- 1 1028 users           0 Jan  1  1970 b¾©âYµ4Fo´äÉæ¸�é¤@pGÈ����_b���ÿÙ
-rwxrwxrwx 1 1028 users         986 Aug 29 08:48 download.sh
---------- 1 1028 users           0 Jan  1  1970 Dè£�ëì@ �ÿÙ
---------- 1 1028 users           0 Jan  1  1970 ô<ôÄ��¿&±l�þÑ��1�ܳ 8#äN�9>¿\Rm�$�ÿÙ
---------- 1 1028 users           0 Jan  1  1970 ÎfUi'y�ë¸æ¡�ã8­©£Ì.¾��¨��ÕhLÿÙ
drwxr-xr-x 1 1028 users           0 Aug 29 13:56 garb
-rwxr-xr-x 1 1028 users           0 Jan  1  1970 ��ýñI
-rwxr-xr-x 1 1028 users           0 Jan  1  1970 iØ��#­8`ô4ì�0E4§9^´�v8 u¤
Øðir;�@�ñ@4c=iq@®
---------- 1 1028 users           0 Jan  1  1970 ÓØí�=³I¦H#ÕU6¨ÎF{Ö°wDM[S>d�;Æþ`1É�ìüX>Ûàí�üýä�YÇ#¥bk°r$S�FÜûÖ��¥½Ç�'Ò¤�ô�Ï��£óT�»£ÿÙ
---------- 1 1028 users           0 Jan  1  1970 �Âòò*�J|¢¹ÿÙ
-rwxr-xr-x 1 1028 users           0 Jan  1  1970 È�ã±ÇJ¶­+�GXXã¥B²qÇ¥vO(Õ<=i|9��äKøt5ËL¡¡85³àë�,·z\�bâ<¦Oñ�E:ªêã¢ùecÃþ#xÒè�±�þ�+�®»âZ�ñÍê���?ý
---------- 1 1028 users           0 Jan  1  1970 ¬Þk���Í�¿�þeË:nÆ>Dè£�ëìq@�ÿÙ
---------- 1 1028 users           0 Jan  1  1970 ¦Í�èK�Én_3M,íé�-$qK) ��{�jÒF±ãh����ç®j��1\YÆ���=ɬMqBÞ ��Ë�Ì×I ä×9¯�ôÈÿ
---------- 1 1028 users           0 Jan  1  1970 ú�À�NÂ?ÿÙ
---------- 1 1028 users           0 Jan  1  1970 oÝø�ÂÞÎåà�O{}#Iqä¼�C43FOÎ��eÈQ��æé�½sÄ�kp��í[��k�.ÚHÃ`����9>§° qHgÿÙ
---------- 1 1028 users           0 Jan  1  1970 Þ�P��zÔ�G�ìvúTÒZH¼¨=ª¹��0 Ð�1�Î�éëV�g¶a±��õª sÍ4ç<Ó�_K�e��"�»)þ�4iµ����9��P[í]Îñ��0i��d��sÔRêUô?ÿÙ
---------- 1 1028 users           0 Jan  1  1970 ��Q2è¢�E�ÿÙ
---------- 1 1028 users           0 Jan  1  1970 q{}#É9�äÚ�C4düè1̹
3ÐóÓ�º��î5��f7@ùÂY���$`à�"����SØ�8ª�­a3ÿÙ
-rwxr-xr-x 1 1028 users           0 Jan  1  1970 �®÷�×qD�»�ûNâ�Dò� 6A!�^�úf¥Äºõ�K,÷·J¤íâë%N3��²§Øà֥γe�¥ÃÂ#�îöFyü¦���)c'æQ�eÈ�=�=1OUÕ.õ¸BÈ�*L�cù·Lè�
drwxr-xr-x 1 1028 users          14 Aug 29 13:35 ��«¼ú¥üQ,nžÖX��¼�H
�HCקó©s}â;H�Yõ;åRqÅáb§®��Ê�c�W.u«(mn�!�×W�Ï7�Ï�Z)P��F9� ô<ôÆ~¯¯K«B�O´n
-rwxr-xr-x 1 1028 users           0 Jan  1  1970 ;`ç¿|Qv�EèÛÅ�]çÕµ�£Tv-öÂÄ0�ä
@l�B�½?�K�CĶ�,³j�ê¤ã�Æb§®��Ê�c�W.u�8mn�!�×W�Ï7�Ï�Z)���F9� ÷ç¦3µ}v]Z�²
drwxr-xr-x 1 1028 users          12 Aug 29 13:35 |��*»²À�r@#�P\M®YǾ{«�BÁA3ä1Àn9ä`��Ç#ÔV¬þ
---------- 1 1028 users           0 Jan  1  1970 S��Ó°ö?ÿÙ
drwxr-xr-x 1 1028 users         130 Aug 29 13:35 ÙÅ¥H�±°À
---------- 1 1028 users           0 Jan  1  1970 <×�]´��Á�"tQÉõ>��(²��ÿÙ
---------- 1 1028 users           0 Jan  1  1970 uhBÉö�ÆO1Ä�,è��ù�¢�O¯±Å�?ÿÙ
---------- 1 1028 users           0 Jan  1  1970 ��VsþѪÄ�ëô§��É&¤�ÚYÛlQ3�a@Êÿ
-rwxr-xr-x 1 1028 users           0 Jan  1  1970 ��åÑõ¶V�æy¡�#v-ç�!�'�!PÙ��Ï^�úg:ê
^Ê��âIU�íâpÅN3�
drwxr-xr-x 1 1028 users         120 Aug 29 13:35 ³�ya©XÀ�\�Tc·�C�8Î��J�v85Òßëö�öW
---------- 1 1028 users           0 Jan  1  1970 �¿ê.YxmËÁ óÁcÉÏ���ÿÙ
---------- 1 1028 users           0 Jan  1  1970 �Zµ	7Ú��|×ó.�HÕ°GÈ����_@@¡è4�ÿÙ

[dantkz]

In case anyone else finds it useful, here is the list of files that have invalid compressed data--crc error. The hashes match the hash on Google Bucket Storage, so it is not download error, but rather an issue of the uploaded files.

file path        hashes match    file hash       source hash
dataset_unaligned_corrupt/0004.tar       True    QhZ58A==        QhZ58A==
dataset_unaligned_corrupt/0069.tar       True    8vBh2w==        8vBh2w==
dataset_unaligned_corrupt/0070.tar       True    j4FAJA==        j4FAJA==
dataset_unaligned_corrupt/0072.tar       True    UkV5Hg==        UkV5Hg==
dataset_unaligned_corrupt/0075.tar       True    54tfnw==        54tfnw==
dataset_unaligned_corrupt/0080.tar       True    MSC5BQ==        MSC5BQ==
dataset_unaligned_corrupt/0081.tar       True    62GXBA==        62GXBA==
dataset_unaligned_corrupt/0084.tar       True    x6FpWQ==        x6FpWQ==
dataset_unaligned_corrupt/0097.tar       True    4y2IBA==        4y2IBA==
dataset_unaligned_corrupt/0030.tar       True    sX8/gA==        sX8/gA==
dataset_aligned_corrupt/0012.tar         True    UOlgtg==        UOlgtg==
dataset_aligned_corrupt/0020.tar         True    wsKrTA==        wsKrTA==
dataset_aligned_corrupt/0032.tar         True    bfe+GA==        bfe+GA==
dataset_aligned_corrupt/0072.tar         True    t712Cg==        t712Cg==
dataset_aligned_corrupt/0074.tar         True    KkZW3Q==        KkZW3Q==
dataset_aligned_corrupt/0076.tar         True    whBODA==        whBODA==
dataset_aligned_corrupt/0079.tar         True    coZVuA==        coZVuA==

The dataset_unaligned/0069.tar file has crc error and it also generates bad files as mentioned above.