Faster way to copy files from camera about python-gphoto2

[jim-easterbrook]

I'm surprised it's that much slower. Your version of the script is probably as quick as it can be. I'd need to have a look at the source code of the gphoto2 command to see what it's doing differently.

It might be possible to do it quicker by having a separate thread to do the camera_file.save, but using threads usually creates more problems than it solves.

[jim-easterbrook]

The copy-files.py example does more than simply copying the files. It gets the timestamp of the file on the camera before copying the file to a directory created using the timestamp. (It's also a bit of a mess.)

If you write a Python script that simply copies all the files on the camera to one directory on the computer with no other messing about then I would expect it to take the same amount of time.

[RodrigoCatto]

Hi @jim-easterbrook

I tried to write a Python script that simply copies all the files on the camera to one directory, and removed the timestamp and checking if file is there already (since I'm just testing the time that it takes to download all images at once).

def main():

    locale.setlocale(locale.LC_ALL, '')
    logging.basicConfig(
        format='%(levelname)s: %(name)s: %(message)s', level=logging.WARNING)
    callback_obj = gp.check_result(gp.use_python_logging())
    camera = gp.check_result(gp.gp_camera_new())
    gp.check_result(gp.gp_camera_init(camera))
    print('Getting list of files from camera...')
    camera_files = list_camera_files(camera)
    if not camera_files:
        print('No files found')
        return 1
    print('Copying files...')
    t0 = time.time()
    for path in camera_files:
        folder, name = os.path.split(path)
        dest = os.path.join(PHOTO_DIR, name)
        camera_file = gp.check_result(gp.gp_camera_file_get(camera, folder, name, gp.GP_FILE_TYPE_NORMAL))
        camera_file.save(dest)

    gp.check_result(gp.gp_camera_exit(camera))
    t_eleapsed = time.time() - t0
    print("Eleapsed time: {:.2f}s".format(t_eleapsed))
    return 0

Even with this code, the time to save images comparing to the CLI gphoto2 --get-all-files is still around two times slower. I did some digging and found that this is the part taking more time to process (Almost 1s per JPEG photo on my Canon R):
gp.check_result(gp.gp_camera_file_get(camera, folder, name, gp.GP_FILE_TYPE_NORMAL))

Do you think that there is a way to optimize my script more?

[jim-easterbrook]

I've just been having a look at how the gphoto2 command saves files (https://github.com/gphoto/gphoto2/blob/master/gphoto2/main.c#L575). It's all a bit confusing but I think the main difference is that it creates a CameraFile from a C file descriptor (using gp_file_new_from_fd) and passes that to gp_camera_file_get.

This isn't currently possible with python-gphoto2 as it calls gp_file_new in the gp_camera_file_get wrapper to create the CameraFile.

[jim-easterbrook]

In commit 14055c7 I allow passing a CameraFile to gp_camera_file_get. I tried copying files using a CameraFile created from a file descriptor but didn't get any speed improvement.

[RodrigoCatto]

Thanks for digging into it @jim-easterbrook. I'll do some testing here and send you the results, but it was still a good find.

[RodrigoCatto]

Closing this issue as we ended up moving towards other camera hardware. Thanks again @jim-easterbrook for taking the time to look into it. If I stumble across this again, I'll make sure to test it.