Comments (3)
You can check the md5 hash of the downloaded files to make sure they're not corrupt. Below are the hash codes for the file you are facing an issue with:
Hashes [base64] for dataset_aligned/0075.tar:
Hash (crc32c): 2MTpng==
Hash (md5): xcNixKPq8KxfiICqvg5JKA==
Hashes [base64] for dataset_unaligned/0075.tar:
Hash (crc32c): 54tfnw==
Hash (md5): TTApEJlRC6Vh+MegQZQu0w==
However, as you said individual files do not limit using the dataset. The images and meta files are stored in a per-image basis. Whichever files you cant access, just ignore them and the corresponding images.
from 3d_street_view.
File dataset_unaligned/0069.tar
is corrupt on Google Bucket Storage.
The crc32c hash I have is 8vBh2w==
which is computed using this script:
import base64
import crcmod
def get_file_crc32c(filename):
""" Compute crc32c hash for a file """
file_bytes = open(filename, 'rb').read()
crc32c = crcmod.predefined.Crc('crc-32c')
crc32c.update(file_bytes)
return base64.b64encode(crc32c.digest()).decode('utf-8')
if __name__ == '__main__':
print(get_file_crc32c('dataset_unaligned/0069.tar'))
The downloaded file hash matches the hash from Google Bucket Storage:
$gsutil ls -L gs://streetview_image_pose_3d/dataset_unaligned/0069.tar
gs://streetview_image_pose_3d/dataset_unaligned/0069.tar:
Creation time: Sat, 07 Oct 2017 11:37:52 GMT
Update time: Sat, 07 Oct 2017 11:37:52 GMT
Storage class: MULTI_REGIONAL
Content-Language: en
Content-Length: 18852484576
Content-Type: application/x-tar
Hash (crc32c): 8vBh2w==
Hash (md5): CnwIZFSsZ7YAeD05DKcDJg==
ETag: CLHjlpq23tYCEAE=
Generation: 1507376272617905
Metageneration: 1
ACL: ACCESS DENIED
Note: You need OWNER permission on the object to read its ACL
TOTAL: 1 objects, 18852484576 bytes (17.56 GiB)
This file tar-bombs and generates a bunch of badly encoded files:
$ls -l
total 129427628
---------- 1 1028 users 0 Jan 1 1970 Ýô�$æ&�
drwxr-xr-x 1 1028 users 17640840 Sep 24 2016 0004
-rw-r--r-- 1 1028 users 10399212396 Oct 7 2017 0004.tar
drwxr-xr-x 1 1028 users 14693070 Sep 25 2016 0069
-rw-r--r-- 1 1028 users 18852484576 Oct 7 2017 0069.tar
-rw-r--r-- 1 1028 users 22280379533 Oct 7 2017 0070.tar
-rw-r--r-- 1 1028 users 17353088272 Oct 7 2017 0072.tar
-rw-r--r-- 1 1028 users 10234234075 Oct 7 2017 0075.tar
drwxr-xr-x 1 1028 users 43319920 Sep 25 2016 0080
-rw-r--r-- 1 1028 users 16946239492 Oct 7 2017 0080.tar
drwxr-xr-x 1 1028 users 44956100 Sep 25 2016 0081
-rw-r--r-- 1 1028 users 18961213955 Oct 7 2017 0081.tar
drwxr-xr-x 1 1028 users 34037640 Sep 24 2016 0084
-rw-r--r-- 1 1028 users 12923984178 Oct 7 2017 0084.tar
drwxr-xr-x 1 1028 users 9207520 Sep 24 2016 0097
-rw-r--r-- 1 1028 users 4583031254 Oct 7 2017 0097.tar
---------- 1 1028 users 0 Jan 1 1970 0²:_øK�üùÿ
---------- 1 1028 users 0 Jan 1 1970 Þ('�1]�¹ã��='s�.ñ�VÏñ�õ§.W'æçÖ�dÊñ�Bä�WA�ÏÿÙ
---------- 1 1028 users 0 Jan 1 1970
�2¬¸4�)ÊêÇÿÙ
---------- 1 1028 users 0 Jan 1 1970 �¤�Äò`3C,düè¸æ\��èyé�ÍoÄskP��í;��k�.ÚHÃ`���
9>§° q@�ÿÙ
---------- 1 1028 users 0 Jan 1 1970 5Ä�m$a°GÈ����SØ�8¦�ÿÙ
---------- 1 1028 users 0 Jan 1 1970 �í92y®$»i#
�>D< äú�À�Å�¹ÿÙ
---------- 1 1028 users 0 Jan 1 1970 b¾©âYµ4Fo´äÉæ¸�é¤@pGÈ����_b���ÿÙ
-rwxrwxrwx 1 1028 users 986 Aug 29 08:48 download.sh
---------- 1 1028 users 0 Jan 1 1970 Dè£�ëì@ �ÿÙ
---------- 1 1028 users 0 Jan 1 1970 ô<ôÄ��¿&±l�þÑ��1�ܳ 8#äN�9>¿\Rm�$�ÿÙ
---------- 1 1028 users 0 Jan 1 1970 ÎfUi'y�ë¸æ¡�ã8©£Ì.¾��¨��ÕhLÿÙ
drwxr-xr-x 1 1028 users 0 Aug 29 13:56 garb
-rwxr-xr-x 1 1028 users 0 Jan 1 1970 ��ýñI
-rwxr-xr-x 1 1028 users 0 Jan 1 1970 iØ��#8`ô4ì�0E4§9^´�v8 u¤
Øðir;�@�ñ@4c=iq@®
---------- 1 1028 users 0 Jan 1 1970 ÓØí�=³I¦H#ÕU6¨ÎF{Ö°wDM[S>d�;Æþ`1É�ìüX>Ûàí�üýä�YÇ#¥bk°r$S�FÜûÖ��¥½Ç�'Ò¤�ô�Ï��£óT�»£ÿÙ
---------- 1 1028 users 0 Jan 1 1970 �Âòò*�J|¢¹ÿÙ
-rwxr-xr-x 1 1028 users 0 Jan 1 1970 È�ã±ÇJ¶+�GXXã¥B²qÇ¥vO(Õ<=i|9��äKøt5ËL¡¡85³àë�,·z\�bâ<¦Oñ�E:ªêã¢ùecÃþ#xÒè�±�þ�+�®»âZ�ñÍê���?ý
---------- 1 1028 users 0 Jan 1 1970 ¬Þk���Í�¿�þeË:nÆ>Dè£�ëìq@�ÿÙ
---------- 1 1028 users 0 Jan 1 1970 ¦Í�èK�Én_3M,íé�-$qK) ��{�jÒF±ãh����ç®j��1\YÆ���=ɬMqBÞ ��Ë�Ì×I ä×9¯�ôÈÿ
---------- 1 1028 users 0 Jan 1 1970 ú�À�NÂ?ÿÙ
---------- 1 1028 users 0 Jan 1 1970 oÝø�ÂÞÎåà�O{}#Iqä¼�C43FOÎ��eÈQ��æé�½sÄ�kp��í[��k�.ÚHÃ`����9>§° qHgÿÙ
---------- 1 1028 users 0 Jan 1 1970 Þ�P��zÔ�G�ìvúTÒZH¼¨=ª¹��0 Ð�1�Î�éëV�g¶a±��õª sÍ4ç<Ó�_K�e��"�»)þ�4iµ����9��P[í]Îñ��0i��d��sÔRêUô?ÿÙ
---------- 1 1028 users 0 Jan 1 1970 ��Q2è¢�E�ÿÙ
---------- 1 1028 users 0 Jan 1 1970 q{}#É9�äÚ�C4düè1̹
3ÐóÓ�º��î5��f7@ùÂY���$`à�"����SØ�8ª�a3ÿÙ
-rwxr-xr-x 1 1028 users 0 Jan 1 1970 �®÷�×qD�»�ûNâ�Dò� 6A!�^�úf¥Äºõ�K,÷·J¤íâë%N3��²§Øà֥γe�¥ÃÂ#�îöFyü¦���)c'æQ�eÈ�=�=1OUÕ.õ¸BÈ�*L�cù·Lè�
drwxr-xr-x 1 1028 users 14 Aug 29 13:35 ��«¼ú¥üQ,nžÖX��¼�H
�HCקó©s}â;H�Yõ;åRqÅáb§®��Ê�c�W.u«(mn�!�×W�Ï7�Ï�Z)P��F9� ô<ôÆ~¯¯K«B�O´n
-rwxr-xr-x 1 1028 users 0 Jan 1 1970 ;`ç¿|Qv�EèÛÅ�]çÕµ�£Tv-öÂÄ0�ä
@l�B�½?�K�CĶ�,³j�ê¤ã�Æb§®��Ê�c�W.u�8mn�!�×W�Ï7�Ï�Z)���F9� ÷ç¦3µ}v]Z�²
drwxr-xr-x 1 1028 users 12 Aug 29 13:35 |��*»²À�r@#�P\M®YǾ{«�BÁA3ä1Àn9ä`��Ç#ÔV¬þ
---------- 1 1028 users 0 Jan 1 1970 S��Ó°ö?ÿÙ
drwxr-xr-x 1 1028 users 130 Aug 29 13:35 ÙÅ¥H�±°À
---------- 1 1028 users 0 Jan 1 1970 <×�]´��Á�"tQÉõ>��(²��ÿÙ
---------- 1 1028 users 0 Jan 1 1970 uhBÉö�ÆO1Ä�,è��ù�¢�O¯±Å�?ÿÙ
---------- 1 1028 users 0 Jan 1 1970 ��VsþѪÄ�ëô§��É&¤�ÚYÛlQ3�a@Êÿ
-rwxr-xr-x 1 1028 users 0 Jan 1 1970 ��åÑõ¶V�æy¡�#v-ç�!�'�!PÙ��Ï^�úg:ê
^Ê��âIU�íâpÅN3�
drwxr-xr-x 1 1028 users 120 Aug 29 13:35 ³�ya©XÀ�\�Tc·�C�8Î��J�v85Òßëö�öW
---------- 1 1028 users 0 Jan 1 1970 �¿ê.YxmËÁ óÁcÉÏ���ÿÙ
---------- 1 1028 users 0 Jan 1 1970 �Zµ 7Ú��|×ó.�HÕ°GÈ����_@@¡è4�ÿÙ
from 3d_street_view.
In case anyone else finds it useful, here is the list of files that have invalid compressed data--crc error
. The hashes match the hash on Google Bucket Storage, so it is not download error, but rather an issue of the uploaded files.
file path hashes match file hash source hash
dataset_unaligned_corrupt/0004.tar True QhZ58A== QhZ58A==
dataset_unaligned_corrupt/0069.tar True 8vBh2w== 8vBh2w==
dataset_unaligned_corrupt/0070.tar True j4FAJA== j4FAJA==
dataset_unaligned_corrupt/0072.tar True UkV5Hg== UkV5Hg==
dataset_unaligned_corrupt/0075.tar True 54tfnw== 54tfnw==
dataset_unaligned_corrupt/0080.tar True MSC5BQ== MSC5BQ==
dataset_unaligned_corrupt/0081.tar True 62GXBA== 62GXBA==
dataset_unaligned_corrupt/0084.tar True x6FpWQ== x6FpWQ==
dataset_unaligned_corrupt/0097.tar True 4y2IBA== 4y2IBA==
dataset_unaligned_corrupt/0030.tar True sX8/gA== sX8/gA==
dataset_aligned_corrupt/0012.tar True UOlgtg== UOlgtg==
dataset_aligned_corrupt/0020.tar True wsKrTA== wsKrTA==
dataset_aligned_corrupt/0032.tar True bfe+GA== bfe+GA==
dataset_aligned_corrupt/0072.tar True t712Cg== t712Cg==
dataset_aligned_corrupt/0074.tar True KkZW3Q== KkZW3Q==
dataset_aligned_corrupt/0076.tar True whBODA== whBODA==
dataset_aligned_corrupt/0079.tar True coZVuA== coZVuA==
The dataset_unaligned/0069.tar
file has crc error and it also generates bad files as mentioned above.
from 3d_street_view.
Related Issues (12)
- df -lh
- permissions to download dataset.
- Permission to download dataset
- Hi, I have some questions about the aligned dataset and its label txt.
- Link to supplementary material is broken
- Camera intrinsics? HOT 6
- How can I access the dataset file list? HOT 1
- Specific area image crawler code available? HOT 1
- is there any trained model include in these code? HOT 1
- I want permissions to download dataset HOT 1
- I want permissions to download dataset
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from 3d_street_view.