mapillary / mapillary_tools

Command line tools for processing and uploading Mapillary imagery

Mapillary-plugin for JOSM

A plugin for showing Mapillary images inside the OpenStreetMap-Editor JOSM.

You can download images from mapillary.com.

For instructions on setting up your local copy of this repository, see INSTALL.md.

Contributing

Our contribution guidelines can be found in CONTRIBUTING.md.

• The source code locations
  • GitHub
  • JOSM GitLab Group
• Issues are managed in JOSM Trac
  • Report a New Ticket
• Translations are done on JOSM's Launchpad and were previously done on transifex.

• Maintainers:
  • Taylor Smock [email protected] (active, @smocktaylor)
  • nokutu (inactive)
  • floscher (inactive)
• License: GPL v2 or later
• Feel free to contact me for any bug or suggestion.

mapillary_tools's Issues

Support direction of movement

The photos from my garmin come with direction tagged, but not in the exif tag that is needed

[pnorman@pippin ~/osm/photos/20150913]$ exif 01-to_hume/VIRB0003-3.jpg | grep -i 'direct.*|'
Reference for direct|T
Direction of Movemen|321.234752
GPS Image Direction |T
GPS Image Direction |314.3 <- from interpolate script, not present originally

It would be nice to be able to use this information

command line argument to force "DONE" at end of upload_with_authentication.py

I would like upload_with_authentication.py to have an argument to force "DONE" after last file have been uploaded. I need this to be able to automate uploads (cron job style)

Uploads don't appear on Mapillary

I have uploaded a few sequences with upload_with_authentication.py, and after uploading each sequence it tells me that is has been successfully uploaded and that the sequence has been finished by uploading a DONE file after I confirm finishing the sequence by pressing y. Everything looks fine to me.

However, the pictures don't seem to arrive even after waiting for several hours. The image counters (uploaded / still processing) on my profile do not increase.

This is not the case if I use the manual upload via web browser. I uploaded the same sequences (each with around 160 images) and they appeared more or less immediately on the "still processing" counter in my profile. This works nicely for sequences of this size, but now I have some sequences with 1000-2000 images, where this manual upload would be a bit annoying.

remove_duplicates.py complains of None or corrupt exif data.

I'm running remove_duplicates on some images that I have geotagged with the geotag_from_gpx.py script.

remove_duplicates.py looks for the gps_speed EXIF tag and if it's not found it throws the error. That tag isn't set in the geotagging script though. I would think that the gps_speed tag should be set in the geotagging script or the check for the presence of that tag should be removed from the remove_duplicates.py script.

Cannot run script on different device

If the script and the files are not on the same device:

Exception in thread Thread-2:
Traceback (most recent call last):
  File "/usr/lib/python2.7/threading.py", line 810, in __bootstrap_inner
    self.run()
  File "/home/craig/dev/GoPro/mapillary_tools/python/upload.py", line 183, in run
    upload_file(filepath, **self.params)
  File "/home/craig/dev/GoPro/mapillary_tools/python/upload.py", line 123, in upload_file
    os.rename(filepath, "success/"+filename)
OSError: [Errno 18] Invalid cross-device link

upload.py: Invalid cross-device link

For uploading I copied the images to an external disk, the script is placed in the PATH (~/bin on an internal drive) and invoked it this way:
upload.py /path/to/imagefolder
or this way
python path/to/upload.py /mnt/media/ul/mapillary/

I tried it four times, it always ends up like this¹.

The system is quite old:
2.6.32-5-686 #1 SMP Mon Sep 23 23:00:18 UTC 2013 i686 GNU/Linux
Exifread is version 1.4.2

Trying it with python3 or 3.1 the scriptz exits this way:

Traceback (most recent call last):
  File "/home/malutka/bin/mapillary_upload.py", line 4, in <module>
    import urllib2, urllib
ImportError: No module named urllib2

¹

Uploading: 2014_07_27_13_36_55_714.jpg`
 Uploading: 2014_07_27_13_16_26_435.jpg
Uploading: 2014_07_27_13_16_20_453.jpg
Uploading: 2014_07_27_13_01_55_699.jpg
Exception in thread Thread-2:
Traceback (most recent call last):
  File "/usr/lib/python2.6/threading.py", line 532, in __bootstrap_inner
    self.run()
  File "/home/malutka/bin/mapillary_upload.py", line 167, in run
    upload_file(filepath, **self.params)
  File "/home/malutka/bin/mapillary_upload.py", line 118, in upload_file
    os.rename(filepath, "success/"+filename)
OSError: [Errno 18] Invalid cross-device link

Exception in thread Thread-3:
Traceback (most recent call last):
  File "/usr/lib/python2.6/threading.py", line 532, in __bootstrap_inner
    self.run()
  File "/home/malutka/bin/mapillary_upload.py", line 167, in run
    upload_file(filepath, **self.params)
  File "/home/malutka/bin/mapillary_upload.py", line 118, in upload_file
    os.rename(filepath, "success/"+filename)
OSError: [Errno 18] Invalid cross-device link

Exception in thread Thread-4:
Traceback (most recent call last):
  File "/usr/lib/python2.6/threading.py", line 532, in __bootstrap_inner
    self.run()
  File "/home/malutka/bin/mapillary_upload.py", line 167, in run
    upload_file(filepath, **self.params)
  File "/home/malutka/bin/mapillary_upload.py", line 118, in upload_file
    os.rename(filepath, "success/"+filename)
OSError: [Errno 18] Invalid cross-device link

Exception in thread Thread-1:
Traceback (most recent call last):
  File "/usr/lib/python2.6/threading.py", line 532, in __bootstrap_inner
    self.run()
  File "/home/malutka/bin/mapillary_upload.py", line 167, in run
    upload_file(filepath, **self.params)
  File "/home/malutka/bin/mapillary_upload.py", line 118, in upload_file
    os.rename(filepath, "success/"+filename)
OSError: [Errno 18] Invalid cross-device link

Standardize on one python exif library

Some scripts use pyexiv2 and some use exifread.

It would be better to only require one. Which is preferred?

do not descend in subdirectories by default

it would great if all tools by default would only operate on the contents of the current directory and subdirs would require a flag.

so many times i have uploaded a sequence, upload script has stalled and i resume it with the same sequence... for it to descend in the "success" directory. arrrgh.

or review images in josm, fix their location and compass direction, then spot that some have missing tags. move reviewed images in a 'tmp' subdirectory, run the script to assign location & direction to the remaining images... and it descends in my 'tmp' and nukes all the work.

even knowing about this, i have this problem so very often. it ends up in worse uploads from me, as in some cases i have been unwilling to redo all the manual cleanup.

upload_with_authentication.py just uploads all files in sub folders into the same sequence

I expected that I'd be able to run time_split.py, then run upload_with_authentication.py and it uploads each split folder as a separate sequence correctly, however it seem that I need to run it once for each sequence, which is horribly annoying, as e.g. I have 23 sequences to upload from the last 2 days accross several folders (otherwise end up with duplicate file names from the camera - Garmin VIRB).

/Volumes/Drobo/Pictures/Virb/2015-01-01/102_VIRB:
0   1   2   3

/Volumes/Drobo/Pictures/Virb/2015-01-01/103_VIRB:

/Volumes/Drobo/Pictures/Virb/2015-01-01/104_VIRB:
4   5   6   7   8

/Volumes/Drobo/Pictures/Virb/2015-01-01/105_VIRB:

/Volumes/Drobo/Pictures/Virb/2015-01-01/106_VIRB:
10  11  9

/Volumes/Drobo/Pictures/Virb/2015-01-01/107_VIRB:
12  13  14  15

/Volumes/Drobo/Pictures/Virb/2015-01-01/108_VIRB:
16

/Volumes/Drobo/Pictures/Virb/2015-01-01/109_VIRB:
17

/Volumes/Drobo/Pictures/Virb/2015-01-01/110_VIRB:
18  19

/Volumes/Drobo/Pictures/Virb/2015-01-01/111_VIRB:
20

/Volumes/Drobo/Pictures/Virb/2015-01-01/112_VIRB:
21  22

Exit on CTRL+C

The script doesn't exit on ctrl+c. It would be very nice to have that feature.

incorrect formatting in the upload_with_authentication.py output

minor, but... in the output of upload_with_authentication.py, sometimes a line can be prefixed with an excess space :

Uploading: IMG_3260.JPG
 Uploading: IMG_3308.JPG
Uploading: IMG_3305.JPG

sometimes it can concatenate two lines and then add an extra newline :

Uploading: IMG_3266.JPG
 Success: IMG_3171.JPGSuccess: IMG_3213.JPG
Success: IMG_3088.JPG
Uploading: IMG_3133.JPG

Uploading: IMG_3270.JPG

upload_with_authentication.py MOVE_FILES also moves "DONE" file

enable MOVE_FILES in upload_with_authentication.py, upload something, tell it to upload DONE. it uploads it and then moves into the "success", then tries to remove it. probably 'DONE' should not be moved to "success" directory.

Traceback (most recent call last):
  File "/mapillary_tools/python/upload_with_authentication.py", line 173, in <module>
    upload_done_file(params)
  File "/mapillary_tools/python/upload_with_authentication.py", line 49, in upload_done_file
    os.remove("DONE")
OSError: [Errno 2] No such file or directory: 'DONE'

upload_video_with_synced_gpx.py

Would it be possible to have video uploads where gpx file starts at first frame in the video? My Garmin VIRB Edit software allow me to trim videos so that I can make sure that GPX track starts at first frame, and last GPX track is at last frame, i.e. a synced video/gpx file. I would like it to work similar as upload_with_authentication.py

upload_with_authentication.py does not work with exifread 2.0.0

After upgrading exifread to version 2.0.0 upload_with_authentication.py does not work any more, for example:

Uploading sequence 6cfdfb62-2ff3-4817-842e-c217d1b84b23.
Missing required EXIF tag: GPS GPSLongitude
Skipping: /home/wh/virb/20150120/1/VIRB0200.JPG
===
Finalizing upload will submit all successful uploads and ignore all failed.

Possibly upload.py has the same error, but I did not test.

As a workaround, I downgraded exifread:

pip install exifread==1.4.2

That fixed the problem. I did not spend time yet on examining the real cause.

upload_with_authentication.py skips images on faill

Due to a wonky connection the upload fails from time to time. The upload script then skips the current image completely. Since the problem is not with that particular image but with the connection skipping these images isn't the best way to go.

URL error: <urlopen error [Errno 32] Broken pipe> on VIRB0738.JPG
URL error: <urlopen error [Errno 8] nodename nor servname provided, or not known> on VIRB0738.JPG

or

HTTP error: HTTP Error 400: Bad Request on VIRB0709.JPG

So it would be a good idea to either try again two or three times or move these images to the end of the stack to prevent loss.

add_mapillary_tag_from_exif.py problem with pyexiv2 in macports named py-exiv2

I'm trying the Python script add_mapillary_tag_from_exif.py in mac os 10.10.5 using python 2.7 through macports.
I've found the similar binding of pyexiv2 in macports but it's named py-exiv2, changing the library name in the script didn't had a good result, the script return:

File "add_mapillary_tag_from_exif.py", line 12
import py-exiv2
^
SyntaxError: invalid syntax

Is there a workaround for running the script using macports?
Thanks.

Upload each folder as sequence

Can't see here how it actually uploads, would like to put warning about upload.py in but do not see where it is actually used.

Furthermore, what is line 9: sys.cleansetdefaultencoding("utf-8")? I can't find any documentation, a Google search only gives me that page. Did you mean sys.setdefaultencoding("utf-8"), if so you will also need reload(sys) beforehand, unsure why but yeah.

EDIT - figured out why

This function is only available at Python start-up time, when Python scans the environment. It has to be called in a system-wide module, sitecustomize.py, After this module has been evaluated, the setdefaultencoding() function is removed from the sys module.
The only way to actually use it is with a reload hack that brings the attribute back.

TY StackOverflow 3828723

Erroneously uploaded thumbnails

I have created thumbnails for all the mapillary images, in the same directory, and then used upload.py mistakenly on the whole directory without deleting those thumbnails before. Apparently the thumbnails still had all the exif info.
The result is that some of the images are represented by those very low-res thumbnails instead of the original resolution images.
Re-uploading all (non-thumbnail) images did help to some extent, but some images are still thumbnails,
e.g. http://www.mapillary.com/map/im/DDmQdG7j3gQfOC_HkaO_cQ , but not the preceding or following picture, similarly http://www.mapillary.com/map/e/1NwogBL3zjruSd6o4nn1qg, or http://www.mapillary.com/map/e/u-sW2uJnYDl2PExXCWONOw
I do not have any idea to fix that from my side after the fact.

require explicit y/n for upload scripts

upload.py and upload_with_authentication.py require the user to "Finalize upload (y/n)" after upload. The current code will finalize after "y", "Y", "Yes", or "yes", which is nice. Any other input + newline will exit without uploading. This might surprise a user after waiting patiently for several hours worth of photos to upload*.

Could the finalize upload also check for an explicit "n", "N", "no", or "No", before exiting, and ask again if the input is neither a valid yes, nor a valid no?

• Don't ask me how I know this.

upload_each_folder_as_sequence.py

Hey.. i used mapillary_tools/python/upload_each_folder_as_sequence.py and it would be great, if
there could be made a second one, that upload's from several child folders into several sequences.

i used the split pythonfile and after that i want to upload this splitted files with a simple script and thought, that the mapillary_tools/python/upload_each_folder_as_sequence.py would do that, but it's only for a merged upload into a single sequence

hope i am in the correct folder, as i don't see any other way to suggest a change/new script for the mapillary_tools...

upload scripts create & reference "failed" directory but do not use it

as discussed in issue #50, upload scripts reference and create a directory "failed" which is supposed to hold failed uploads, but do not seem to actually move files in there (the files just stay in the original directory).

with more retries added, it is sort of expected that uploads are retried and never fail, so the sequence is complete - although i'm slightly confused about the default of 10 retries and do not know what happens when all 10 fail :)

also, looking at upload.py, line 122, it might move files to "failed" if it got some http response code (not exception) besides 204.

somebody with more understanding of both python and what can go wrong when uploading should carefully check this :)

upload script can hang

upload script (using upload_with_authentication.py, not sure where the hang happens) sometimes hangs. usually it seems to be one thread - all images but one get uploaded and then it just sits there.

stracing isn't that helpful - it's just a constant stream of :

select(0, NULL, NULL, NULL, {1, 0})     = 0 (Timeout)

using them with python 2.7.8

Who may use it?

I see a hash of permission on the code...

upload_with_authentication.py does not move failed files

if some files fail to upload with upload_with_authentication.py, they are not moved in the "failed" directory even with "MOVE_FILES = True". successfully uploaded files are moved.

although a much better approach would be to retry them either infinitely or allow user to initiate retry - somewhat covered in issue #23

Failed upload with upload_with_authentication.py conflicts with manual uploads

If an upload fails or is in progress and you go to http://www.mapillary.com/map/upload it'll suggest to push to Mapillary. There is no way to see which images will be pushed to Mapillary. Will the "Clear all images" button clear all the files uploaded and not pushed yet?

Something seems to go wrong when finalising the upload with upload_with_authentication.py

I get the following when finalising an upload with upload_with_authentication.py

Mac OS X 10.10.4, macports, python2.7, all dependencies updated with pip

Finalizing upload will submit all successful uploads and ignore all failed.
If all files were marked as successful, everything is fine, just press 'y'.
Finalize upload? [y/n]: y
Upload a DONE file to tell the backend that the sequence is all uploaded and ready to submit.
Uploading: DONE
Success: DONE
Traceback (most recent call last):
  File "upload_with_authentication.py", line 173, in <module>
    upload_done_file(params)
  File "upload_with_authentication.py", line 49, in upload_done_file
    os.remove("DONE")
OSError: [Errno 2] No such file or directory: 'DONE'

show upload statistics

currently the upload scripts just upload and do not show any stats. it would be useful to know :

• total images
• uploaded images
• % completed (or todo)
• time spent
• estimated time to complete

in the simplest case, this could just be printed to stdout repeatedly, or it could be some more fancy repainting.

i have shell scripts to show some of this information, but that's a bit hackish and would be much better to have built-in :)

upload_with_authentication.py does not show success/failed counts

upload.py seems to have some code to show how many files were uploaded successfully and how many failed, but upload_with_authentication.py does not - at the end it just prints :

Finalizing upload will submit all successful uploads and ignore all failed.
If all files were marked as successful, everything is fine, just press 'y'.

allow to resume a sequece with upload_with_authentication.py

upload_with_authentication.py currently does not allow to resume a sequence - attempting to do that just starts a new sequence.

this would be needed because it tends to skip images upon failure. even when that would be fixed, such a feature could be highly useful - let's say i'm uploading a sequence, and then i have to move to a different location with my laptop. currently that means restarting the upload of that sequence, in some cases resulting in gigabytes of data transferred again.

default to moving the files, or provide a cli option

when uploading files, the scripts default to leaving the files as-is, and there's an option MOVE_FILES that can be set to "True". moving the files would allow to see whether any have failed (hopefully much less after recent retry adding :) ) and - very important - to resume uploading the sequence.

as such it might be better to move the files by default, or at least have a cli option to enable that.

upload_with_authentication.py files not moved to success folder after successful upload

upload.py moves files ok to success folder after they are uploaded.

upload_with_authentication.py does not, it is missing

"move_files":True

time_split.py loses files with duplicate file names

The time_split.py file will loose files where there are 2 files in sub folders with the same file name. This is an issue with for example the Garmin VIRB elite which places 1000 photos in a folder, then restarts the numbering of the photo files in a new folder. Ideally it should start a new folder, or rename the files that it's moving if there is a name clash.

upload.py unable to handle connection closed

While uploading a huge batch of images, there is a problem with the connection to amazon and it closes(possible timeout on amazon's part?) but the script crashes and fails to resume the upload

also get this error:

upload.py stops with errors

Thank you for that tool. It's very helpful but it regularly stops with the following error:

Exception in thread Thread-4:
Traceback (most recent call last):
  File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/threading.py", line 808, in __bootstrap_inner
    self.run()
  File "upload.py", line 131, in run
    upload_file(filepath)
  File "upload.py", line 101, in upload_file
    response = urllib2.urlopen(request)
  File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/urllib2.py", line 127, in urlopen
    return _opener.open(url, data, timeout)
  File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/urllib2.py", line 404, in open
    response = self._open(req, data)
  File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/urllib2.py", line 422, in _open
    '_open', req)
  File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/urllib2.py", line 382, in _call_chain
    result = func(*args)
  File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/urllib2.py", line 1214, in http_open
    return self.do_open(httplib.HTTPConnection, req)
  File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/urllib2.py", line 1184, in do_open
    raise URLError(err)

URLError: <urlopen error [Errno 32] Broken pipe>

Try to build a GUI for upload

As it's little bit obsed, when u try to upload pictures, that are no with GPS values, the Webinterface get stuck.. so i try to build an GUI.. but .. get stuck and need help

Any Idea, why the E1 Field is not updating ???

[code]

!/usr/bin/python

-- coding: utf-8 --

from Tkinter import *
from tkMessageBox import *
from tkFileDialog import *
from ttk import *

global pathname
pathname ="pathnamevariabl"

import os

def answer():
showerror("Answer", "Sorry, no answer available")

def CloseButtonClass():
quit()

def BrowseButtonClass():
name = askopenfilename()

global pathname

pathname = os.path.split(name)[0]

print pathname

def HelpButtonClass():
showwarning('Help', 'Not yet implemented')

def GoButtonClass():
showwarning('Go', 'Not yet implemented')

class MapillaryGui(Frame):

def __init__(self, parent):
    Frame.__init__(self, parent)   

    self.parent = parent

    self.initUI()

def initUI(self):

    self.parent.title("Mapillary Upload GUI")
    self.style = Style()
    self.style.theme_use("default")
    self.pack(fill=BOTH, expand=1)

    self.columnconfigure(1, weight=1)
    self.columnconfigure(3, pad=7)
    self.rowconfigure(3, weight=1)
    self.rowconfigure(5, pad=7)

    lbl = Label(self, text="Folder")
    lbl.grid(sticky=W, pady=4, padx=5)


    #das weisse Feld  - old code
    #area = Text(self)

    area = Text(self)

    Label(area, text="Threat 1 : ").grid(row=0)
    Label(area, text="Threat 2 : ").grid(row=1)

    e1 = Entry(area)
    e2 = Entry(area)

    e1.grid(row=0, column=1)
    e2.grid(row=1, column=1)

    e1.insert(10,pathname)

    print pathname

    area.grid(row=1, column=0, columnspan=2, rowspan=4, 
        padx=5, sticky=E+W+S+N)


    # Button Browse
    abtn = Button(self, text="Browse", command=BrowseButtonClass)
    abtn.grid(row=1, column=3)

    #Button Close
    cbtn = Button(self, text="Close", command=CloseButtonClass)
    cbtn.grid(row=2, column=3, pady=4)

    #Button Help
    hbtn = Button(self, text="Help", command=HelpButtonClass)
    hbtn.grid(row=5, column=0, padx=5)

    #Button Go
    obtn = Button(self, text="GO", command=GoButtonClass)
    obtn.grid(row=5, column=3)

def main():

root = Tk()
root.geometry("300x200+300+300")
app = MapillaryGui(root)
root.mainloop()

if name == 'main':
main()

[/code]

urllib2 not installing, urllib3 and urllib5 fails to execute

For some reason urllib2 doesn't install (no longer available in pypi?), and when trying urllib5 from pypi I get a cannot find urllib2 error, so I tried with urllib3, which doesn't seem to have that problem. But than it started to complain on all the syntax (exception handling missing or with different names, Request function seem to have changed name, etc).

I would like to see this solved as I have almost 4000 images waiting to upload and need to do this with upload_with_authentication.py or through manual upload site, which takes too long. I really need this automated.

upload_each_folder_as_sequence.py seems to freeze between folders

upload_each_folder_as_sequence.py seems to freeze when completing a folder. There is no indication of what the script is doing, everything just hangs.

Mac OS X 10.10.4, python2.7, all dependencies up to date with pip

temporary solution is to run upload_with_authentication.py in loop

http error

When trying to upload i get this:

Uploading sequence 8994b835-6ff5-4f7a-ad76-d8cbef4d574f.
Uploading: tt_img_00280.jpg
Uploading: tt_img_00275.jpg
Uploading: tt_img_00263.jpg
Uploading: tt_img_00281.jpg
HTTP error: HTTP Error 403: Forbidden on tt_img_00281.jpg
HTTP error: HTTP Error 403: Forbidden on tt_img_00281.jpg
HTTP error: HTTP Error 403: Forbidden on tt_img_00275.jpg
HTTP error: HTTP Error 403: Forbidden on tt_img_00280.jpg
HTTP error: HTTP Error 403: Forbidden on tt_img_00281.jpg
HTTP error: HTTP Error 403: Forbidden on tt_img_00263.jpg
HTTP error: HTTP Error 403: Forbidden on tt_img_00281.jpg
Uploading: tt_img_00267.jpg
HTTP error: HTTP Error 403: Forbidden on tt_img_00275.jpg
HTTP error: HTTP Error 403: Forbidden on tt_img_00280.jpg
HTTP error: HTTP Error 403: Forbidden on tt_img_00263.jpg
HTTP error: HTTP Error 403: Forbidden on tt_img_00267.jpg
HTTP error: HTTP Error 403: Forbidden on tt_img_00275.jpg
HTTP error: HTTP Error 403: Forbidden on tt_img_00280.jpg
HTTP error: HTTP Error 403: Forbidden on tt_img_00263.jpg
HTTP error: HTTP Error 403: Forbidden on tt_img_00275.jpg
Uploading: tt_img_00259.jpg
HTTP error: HTTP Error 403: Forbidden on tt_img_00280.jpg
Uploading: tt_img_00266.jpg
HTTP error: HTTP Error 403: Forbidden on tt_img_00267.jpg
HTTP error: HTTP Error 403: Forbidden on tt_img_00259.jpg
HTTP error: HTTP Error 403: Forbidden on tt_img_00263.jpg
Uploading: tt_img_00268.jpg
HTTP error: HTTP Error 403: Forbidden on tt_img_00267.jpg
HTTP error: HTTP Error 403: Forbidden on tt_img_00266.jpg
HTTP error: HTTP Error 403: Forbidden on tt_img_00259.jpg
^C
BREAK: Stopping upload.

No such file or directory: 'DONE'

Whenever I run the script on my laptop (ubuntu 14.04) I get this error at the end, as if the last file is always specified to be the string 'DONE'.

Uploading: DONE
Success: DONE
Traceback (most recent call last):
  File "upload_with_authentication.py", line 154, in <module>
    upload_done_file(params)
  File "upload_with_authentication.py", line 47, in upload_done_file
    os.remove("DONE")
OSError: [Errno 2] No such file or directory: 'DONE'

Mapillary Tools on PyPi

@jesolem @peterneubauer

Instead of modifying the entire mapillary tools repo, I decided to create my own clean repo based from your Python tools, there is a lot of testing to be done before it's actually an official Mapillary API.

Once it's good enough as a release, you guys can fork it and pick it up from there.

http://deniscarriere.github.io/mapillary/

At the moment I only added the Upload feature, but I do plan on including @jesolem's GeoTag from GPX feature.

The end result would look something like this:

$ mapillary geotag --input "<File Path>" --gpx "path.gpx"
$ mapillary upload --input "<File Path>" --username "<Username>"

upload_with_authentication.py fails

$ python upload_with_authentication.py 100_VIRB/3/
Traceback (most recent call last):
File "upload_with_authentication.py", line 143, in
uploaders = [UploadThread(q, params) for i in range(NUMBER_THREADS)]
TypeError: init() takes exactly 2 arguments (3 given)

Geotag gets it wrong (for me)

Well, you asked for testers :) I'm trying to tag some photos from a cycle tour last weekend. The time on my camera looks correct (to within a minute), and my GPS (Garmin 600) shows the correct time. Yet the geotagging calculated by this tool is completely wrong.

With this set of photos, I've found with trial and error that an offset of 39500 looks about right for the last few photos in my collection (..14.JPG to ..22.JPG), but the computed lat/long for the last two (23 and 24) is too "early" (it should be maybe 30 minutes later).

Meanwhile, photos 0974 and 0975 both have a timestamp of 12:00pm on March 8th, yet are given dramatically locations. 0974 seems to have about the right location, but 0975 is from a place at least an hour earlier in the day.

Similarly, photo 0965 and 0967 both have the timestamp 17:36 on March 7th. 0967 is given a location which looks exactly right, but 0968 is from a place nearly 24 hours later!

I'm using Picasa to verify the locations geotagged in the photos. The command line I'm running looks like this:

python geotag_from_gpx.py /pictures/2015-03-07-SGZ-geotagged/ South\ Gippsland\ Jazz\ Tour\ merged.gpx 39500

I can't say for sure whether it's definitely the script that's not behaving right - maybe Picasa is showing the wrong data. (It's definitely doing some weird things, like not indicating that certain photos are geotagged, even though it shows the lat/long in the metadata).

-rwxrwxrwx 1 vagrant vagrant 3322880 Mar  7 02:28 P1030949.JPG
-rwxrwxrwx 1 vagrant vagrant 3310592 Mar  7 02:28 P1030950.JPG
-rwxrwxrwx 1 vagrant vagrant 3372544 Mar  7 02:28 P1030951.JPG
-rw-r--r-- 1 vagrant vagrant 3535619 Mar 14 10:08 P1030952.JPG
-rwxrwxrwx 1 vagrant vagrant 3307520 Mar  7 04:04 P1030953.JPG
-rw-r--r-- 1 vagrant vagrant 3252997 Mar 14 10:09 P1030954.JPG
-rw-r--r-- 1 vagrant vagrant 3402540 Mar 14 10:09 P1030956.JPG
-rwxrwxrwx 1 vagrant vagrant 3576320 Mar  7 05:16 P1030957.JPG
-rwxrwxrwx 1 vagrant vagrant 3346944 Mar  7 05:22 P1030958.JPG
-rw-r--r-- 1 vagrant vagrant 3250036 Mar 14 10:09 P1030959.JPG
-rw-r--r-- 1 vagrant vagrant 3452546 Mar 14 10:09 P1030960.JPG
-rw-r--r-- 1 vagrant vagrant 3335079 Mar 14 10:10 P1030961.JPG
-rwxrwxrwx 1 vagrant vagrant 3431936 Mar  7 06:17 P1030962.JPG
-rw-r--r-- 1 vagrant vagrant 3172363 Mar 14 10:10 P1030963.JPG
-rw-r--r-- 1 vagrant vagrant 3442613 Mar 14 10:10 P1030964.JPG
-rwxrwxrwx 1 vagrant vagrant 3470848 Mar  7 06:36 P1030965.JPG
-rwxrwxrwx 1 vagrant vagrant 3300352 Mar  7 06:36 P1030967.JPG
-rwxrwxrwx 1 vagrant vagrant 3585536 Mar  7 06:37 P1030968.JPG
-rwxrwxrwx 1 vagrant vagrant 3615232 Mar  7 06:56 P1030969.JPG
-rw-r--r-- 1 vagrant vagrant 3463210 Mar 14 10:11 P1030970.JPG
-rwxrwxrwx 1 vagrant vagrant 3552768 Mar  7 21:32 P1030971.JPG
-rwxrwxrwx 1 vagrant vagrant 3278336 Mar  7 21:33 P1030972.JPG
-rwxrwxrwx 1 vagrant vagrant 3338752 Mar  7 21:35 P1030973.JPG
-rwxrwxrwx 1 vagrant vagrant 3398656 Mar  8 01:00 P1030974.JPG
-rw-r--r-- 1 vagrant vagrant 3346223 Mar 14 10:12 P1030975.JPG
-rwxrwxrwx 1 vagrant vagrant 3305984 Mar  8 01:00 P1030976.JPG
-rwxrwxrwx 1 vagrant vagrant 3355136 Mar  8 01:01 P1030977.JPG
-rwxrwxrwx 1 vagrant vagrant 3387904 Mar  8 01:02 P1030978.JPG
-rwxrwxrwx 1 vagrant vagrant 3310592 Mar  8 01:08 P1030979.JPG
-rwxrwxrwx 1 vagrant vagrant 3298304 Mar  8 01:08 P1030980.JPG
-rwxrwxrwx 1 vagrant vagrant 3287552 Mar  8 01:08 P1030981.JPG
-rwxrwxrwx 1 vagrant vagrant 3369472 Mar  8 02:14 P1030982.JPG
-rw-r--r-- 1 vagrant vagrant 3297560 Mar 14 10:12 P1030983.JPG
-rw-r--r-- 1 vagrant vagrant 3412067 Mar 14 10:12 P1030984.JPG
-rw-r--r-- 1 vagrant vagrant 3356035 Mar 14 10:12 P1030985.JPG
-rw-r--r-- 1 vagrant vagrant 3441850 Mar 14 10:13 P1030986.JPG
-rw-r--r-- 1 vagrant vagrant 3326647 Mar 14 10:13 P1030987.JPG
-rwxrwxrwx 1 vagrant vagrant 3424256 Mar  8 05:05 P1030988.JPG
-rw-r--r-- 1 vagrant vagrant 3299158 Mar 14 10:13 P1030989.JPG
-rwxrwxrwx 1 vagrant vagrant 3280896 Mar  8 05:09 P1030990.JPG
-rwxrwxrwx 1 vagrant vagrant 3538944 Mar  8 05:11 P1030991.JPG
-rwxrwxrwx 1 vagrant vagrant 3293696 Mar  8 05:14 P1030992.JPG
-rwxrwxrwx 1 vagrant vagrant 3273728 Mar  8 05:15 P1030993.JPG
-rwxrwxrwx 1 vagrant vagrant 3545088 Mar  8 05:16 P1030994.JPG
-rwxrwxrwx 1 vagrant vagrant 3509248 Mar  8 05:18 P1030995.JPG
-rwxrwxrwx 1 vagrant vagrant 3593728 Mar  8 05:19 P1030996.JPG
-rwxrwxrwx 1 vagrant vagrant 3431424 Mar  9 01:58 P1030997.JPG
-rw-r--r-- 1 vagrant vagrant 3380586 Mar 14 10:13 P1030998.JPG
-rw-r--r-- 1 vagrant vagrant 3543991 Mar 14 10:14 P1030999.JPG
-rwxrwxrwx 1 vagrant vagrant 3460608 Mar  9 02:09 P1040001.JPG
-rw-r--r-- 1 vagrant vagrant 3577052 Mar 14 10:14 P1040002.JPG
-rw-r--r-- 1 vagrant vagrant 3390191 Mar 14 10:14 P1040004.JPG
-rw-r--r-- 1 vagrant vagrant 3466775 Mar 14 10:14 P1040005.JPG
-rw-r--r-- 1 vagrant vagrant 3377522 Mar 14 10:14 P1040006.JPG
-rwxrwxrwx 1 vagrant vagrant 3480576 Mar  9 04:06 P1040007.JPG
-rwxrwxrwx 1 vagrant vagrant 3316224 Mar  9 04:22 P1040008.JPG
-rwxrwxrwx 1 vagrant vagrant 3375104 Mar  9 05:06 P1040009.JPG
-rwxrwxrwx 1 vagrant vagrant 3575296 Mar  9 06:48 P1040010.JPG
-rw-r--r-- 1 vagrant vagrant 3337524 Mar 14 10:15 P1040011.JPG
-rwxrwxrwx 1 vagrant vagrant 3284992 Mar  9 07:13 P1040012.JPG
-rw-r--r-- 1 vagrant vagrant 3496411 Mar 14 10:15 P1040013.JPG
-rwxrwxrwx 1 vagrant vagrant 3307008 Mar  9 07:42 P1040014.JPG
-rw-r--r-- 1 vagrant vagrant 3427101 Mar 14 10:15 P1040015.JPG
-rwxrwxrwx 1 vagrant vagrant 3395584 Mar  9 07:43 P1040016.JPG
-rwxrwxrwx 1 vagrant vagrant 3441152 Mar  9 07:43 P1040017.JPG
-rwxrwxrwx 1 vagrant vagrant 3491328 Mar  9 07:52 P1040018.JPG
-rwxrwxrwx 1 vagrant vagrant 3387904 Mar  9 21:52 P1040019.JPG
-rwxrwxrwx 1 vagrant vagrant 3357184 Mar  9 21:57 P1040020.JPG
-rwxrwxrwx 1 vagrant vagrant 3439616 Mar  9 23:27 P1040021.JPG
-rw-r--r-- 1 vagrant vagrant 3414536 Mar 14 10:15 P1040022.JPG
-rw-r--r-- 1 vagrant vagrant 3283197 Mar 14 10:16 P1040023.JPG
-rwxrwxrwx 1 vagrant vagrant 3558912 Mar 10 06:45 P1040024.JPG

GPX file of day 2 (971 to 018.JPG):
https://www.dropbox.com/s/kgdapat6r316jrw/2015-03-08%2008.44.41%20Day.gpx?dl=0

These two photos are at the same time and place (about -38.56252, 145.95753):

• https://www.dropbox.com/s/5wg2n4cfgsijqkn/P1030974.JPG?dl=0
• https://www.dropbox.com/s/c3o9re0p5zstovq/P1030975.JPG?dl=0

Connection reset by peer

I am trying to upload a (large) sequence and getting intermittent 'connection reset by peer' errors. The full output is in this gist. After the last error, the script never returns and it looks like the sequence did not get uploaded.

Advise to run deduplication before interpolation

I'm not sure if this should be here, the guide, or both, but if both deduplicating and interpolating for direction, much better results are obtained by deduplicating first. Otherwise, wrong directions are obtained where the GPS drifts while stopped.

exception - BadStatusLine: ''

sometimes uploading fails with an exception like this:

Exception in thread Thread-3:
Traceback (most recent call last):
  File "/usr/lib64/python2.7/threading.py", line 810, in __bootstrap_inner
    self.run()
  File "/home/rich/files/mapillary/mapillary_tools/python/upload.py", line 187, in run
    upload_file(filepath, **self.params)
  File "/home/rich/files/mapillary/mapillary_tools/python/upload.py", line 120, in upload_file
    response = urllib2.urlopen(request)
  File "/usr/lib64/python2.7/urllib2.py", line 127, in urlopen
    return _opener.open(url, data, timeout)
  File "/usr/lib64/python2.7/urllib2.py", line 404, in open
    response = self._open(req, data)
  File "/usr/lib64/python2.7/urllib2.py", line 422, in _open
    '_open', req)
  File "/usr/lib64/python2.7/urllib2.py", line 382, in _call_chain
    result = func(*args)
  File "/usr/lib64/python2.7/urllib2.py", line 1222, in https_open
    return self.do_open(httplib.HTTPSConnection, req)
  File "/usr/lib64/python2.7/urllib2.py", line 1187, in do_open
    r = h.getresponse(buffering=True)
  File "/usr/lib64/python2.7/httplib.py", line 1067, in getresponse
    response.begin()
  File "/usr/lib64/python2.7/httplib.py", line 409, in begin
    version, status, reason = self._read_status()
  File "/usr/lib64/python2.7/httplib.py", line 373, in _read_status
    raise BadStatusLine(line)
BadStatusLine: ''

Cannot upload panoramas missing GPSImgDirection

The upload of a panorama created on Android with Samsung surround shot, with the tag GPano:PoseHeadingDegrees set, fails with error message "Missing required EXIF tag: GPS GPSImgDirection".

improvement of readme.md

i'm an ordinary mapper and using gopro hd 1080 in stills-mode. so i have to run some of the scripts of mapillary_tools. but some explanations of readme.md are unclear to me:

• add_project.py is only for mapillary business, isn't it? or how can i use it?
• geotag_from_gpx.py works fine, after install gpxpy-module
• interpolate_direction.py doesn't works, but have i to run it after geotag_from_gpx.py which already set bearing?
• add_mapillary_tag_from_exif.py what is it for and where get MAPILLARY_UPLOAD_TOKEN
• upload_with_authentication.py results in #34, because i also set export MAPILLARY_PERMISSION_HASH="hash" (with quotes) after reading http://blog.mapillary.com/technology/2014/07/21/upload-scripts.html

Do you may/want be responsible for this Greasemonkey script in this repository?

See GM_mapillary-osm-notes. Do we may move it to it here?

Example: http://www.openstreetmap.org/note/194601

Better error handling from dependencies

I am trying to send over a weak and unstable connection and frequently get the following error blocking the script. The scripts should handle errors better, and errors such as below should result in retry transmission of the same file

Exception in thread Thread-1:
Traceback (most recent call last):
  File "/opt/local/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/threading.py", line 810, in __bootstrap_inner
    self.run()
  File "/Users/skippern/Pictures/Mapillary/upload.py", line 187, in run
    upload_file(filepath, **self.params)
  File "/Users/skippern/Pictures/Mapillary/upload.py", line 120, in upload_file
    response = urllib2.urlopen(request)
  File "/opt/local/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/urllib2.py", line 154, in urlopen
    return opener.open(url, data, timeout)
  File "/opt/local/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/urllib2.py", line 431, in open
    response = self._open(req, data)
  File "/opt/local/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/urllib2.py", line 449, in _open
    '_open', req)
  File "/opt/local/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/urllib2.py", line 409, in _call_chain
    result = func(*args)
  File "/opt/local/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/urllib2.py", line 1227, in http_open
    return self.do_open(httplib.HTTPConnection, req)
  File "/opt/local/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/urllib2.py", line 1200, in do_open
    r = h.getresponse(buffering=True)
  File "/opt/local/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/httplib.py", line 1132, in getresponse
    response.begin()
  File "/opt/local/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/httplib.py", line 453, in begin
    version, status, reason = self._read_status()
  File "/opt/local/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/httplib.py", line 409, in _read_status
    line = self.fp.readline(_MAXLINE + 1)
  File "/opt/local/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/socket.py", line 480, in readline
    data = self._sock.recv(self._rbufsize)
error: [Errno 54] Connection reset by peer

upload_with_authentication.py error in thread means stall at the end of process

I found the following error

Success: VIRB0560.JPG
Uploading: VIRB0562.JPG
Exception in thread Thread-1:
Traceback (most recent call last):
  File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/threading.py", line 810, in __bootstrap_inner
    self.run()
  File "/Users/shaunmcdonald/code/mapillary_tools/python/upload.py", line 182, in run
    upload_file(filepath, **self.params)
  File "/Users/shaunmcdonald/code/mapillary_tools/python/upload.py", line 120, in upload_file
    response = urllib2.urlopen(request)
  File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/urllib2.py", line 127, in urlopen
    return _opener.open(url, data, timeout)
  File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/urllib2.py", line 404, in open
    response = self._open(req, data)
  File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/urllib2.py", line 422, in _open
    '_open', req)
  File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/urllib2.py", line 382, in _call_chain
    result = func(*args)
  File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/urllib2.py", line 1222, in https_open
    return self.do_open(httplib.HTTPSConnection, req)
  File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/urllib2.py", line 1187, in do_open
    r = h.getresponse(buffering=True)
  File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/httplib.py", line 1045, in getresponse
    response.begin()
  File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/httplib.py", line 409, in begin
    version, status, reason = self._read_status()
  File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/httplib.py", line 373, in _read_status
    raise BadStatusLine(line)
BadStatusLine: ''

Success: VIRB0558.JPG
Uploading: VIRB0563.JPG

whilst uploading has meant that after the last file uploaded the python script just stalled and didn't do anything more.

It was sequence "Uploading sequence b14339de-8a17-4eed-aeca-3df14110f068."

There was also the error

URL error: <urlopen error [Errno 32] Broken pipe> on VIRB0301.JPG

Which may affect it too.

Could the script be made to cope with the above error?