dogsheep / dogsheep-photos Goto Github PK
View Code? Open in Web Editor NEWUpload your photos to S3 and import metadata about them into a SQLite database
License: Apache License 2.0
Upload your photos to S3 and import metadata about them into a SQLite database
License: Apache License 2.0
To spot duplicate images (where the file content differs such that the sha256 is no longer a match) it would be useful to calculate and store perceptual hashes of some sort.
Having installed in a venv I get:
(venv) (base) Robins-MacBook:datasette robin$ photo-to-sqlite apple-photos photos.db
-bash: photo-to-sqlite: command not found
dogsheep-photos allows to annotate photos using Apple Photo's db. It would be nice to have such ability on other OSes too. For example using trained local model or using Google Vision API (see #14).
Following the readme, on big sur, and having entered my auth creds via dogsheep-photos s3-auth
:
(venv) (base) Robins-MacBook:datasette robin$ dogsheep-photos upload photos.db ~/Pictures/Photos\ /Users/robin/Pictures/Library.photoslibrary --dry-run
Fetching existing keys from S3...
Traceback (most recent call last):
File "/Users/robin/datasette/venv/bin/dogsheep-photos", line 8, in <module>
sys.exit(cli())
File "/Users/robin/datasette/venv/lib/python3.8/site-packages/click/core.py", line 829, in __call__
return self.main(*args, **kwargs)
File "/Users/robin/datasette/venv/lib/python3.8/site-packages/click/core.py", line 782, in main
rv = self.invoke(ctx)
File "/Users/robin/datasette/venv/lib/python3.8/site-packages/click/core.py", line 1259, in invoke
return _process_result(sub_ctx.command.invoke(sub_ctx))
File "/Users/robin/datasette/venv/lib/python3.8/site-packages/click/core.py", line 1066, in invoke
return ctx.invoke(self.callback, **ctx.params)
File "/Users/robin/datasette/venv/lib/python3.8/site-packages/click/core.py", line 610, in invoke
return callback(*args, **kwargs)
File "/Users/robin/datasette/venv/lib/python3.8/site-packages/dogsheep_photos/cli.py", line 96, in upload
key.split(".")[0] for key in get_all_keys(client, creds["photos_s3_bucket"])
File "/Users/robin/datasette/venv/lib/python3.8/site-packages/dogsheep_photos/utils.py", line 46, in get_all_keys
for row in page["Contents"]:
KeyError: 'Contents'
Possibly since the bucket is in EU (London) eu-west-2
and this into is not requested?
http://127.0.0.1:8001/photos/photos_with_apple_metadata gives "Invalid SQL no such table: main.uploads"
I get the following error when running "pip3 install dogsheep-photos"
" from inspect import ismethod, isclass, formatargspec
ImportError: cannot import name 'formatargspec' from 'inspect' (/Library/Frameworks/Python.framework/Versions/3.12/lib/python3.12/inspect.py). Did you mean: 'formatargvalues'?"
Python 3.12.0rc1
sqlite 3.43.0
datasette, version 0.64.3
Modeled on github-to-sqlite auth
- prompts the user for their S3 credentials and saves them to auth.json
.
I want people to be able to query their Apple Photos metadata without having to first run upload
to upload all of their files to their own S3 bucket.
To do this I can have apple-photos
calculate SHA256 hashes of each photo if the uploads
table does not yet exist (or does not contain that photo).
Follow-on from #1. Apple Photos runs some very sophisticated machine learning on-device to figure out if photos are of dogs, llamas and so on. I really want to extract those labels out into my own database.
Just saw this:
Uploading 0.05 GB
https://docs.aws.amazon.com/AmazonS3/latest/API/RESTCommonResponseHeaders.html
Objects created by the PUT Object, POST Object, or Copy operation, or through the AWS Management Console, and are encrypted by SSE-S3 or plaintext, have ETags that are an MD5 digest of their object data.
This is hard-coded at the moment, which is bad:
https://github.com/dogsheep/photos-to-sqlite/blob/d5d69b9019703c47bc251444838578dd752801e2/photos_to_sqlite/cli.py#L269-L272
So I can show people what this does, using some of my photos.
I followed the instructions to setup credentials but I am getting a invalid bucket name. Can you put a sample auth.json file in the base that shows the correct format for this? Thanks
photos-to-sqlite
doesn't really capture the full scope of this project anymore.
Upload was added in #4
Refs #17.
The build is broken right now because you can't pip install osxphotos
on Ubuntu.
Follow on from #4.
Here's one way to add that:
select
rowid,
photo,
(
select
json_group_array(
json_object(
'label',
normalized_string,
'href',
'/photos/labelled?_hide_sql=1&label=' || normalized_string
)
)
from
labels
where
labels.uuid = photos_with_apple_metadata.uuid
) as labels,
date,
A custom Datasette plugin that can be run locally on a Mac laptop which knows how to serve photos such that they can be seen in the browser.
Originally posted by @simonw in #19 (comment)
Since #10 provided such a speedup, can the same thing be done for the actual uploads?
http://ls.pwd.io/2013/06/parallel-s3-uploads-using-boto-and-threads-in-python/ suggests it can really help performance.
https://pypi.org/project/ExifReader/
New fork that should be able to handle EXIF in HEIC files.
Forked here: ianare/exif-py#102 (comment)
Refs #3
% python -i $(which photos-to-sqlite) apple-photos photos.db
Traceback (most recent call last):
File "/Users/simon/.local/share/virtualenvs/photos-to-sqlite-0uGSHd6e/lib/python3.8/site-packages/osxphotos/photoinfo.py", line 611, in place
return self._place # pylint: disable=access-member-before-definition
AttributeError: 'PhotoInfo' object has no attribute '_place'
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/Users/simon/.local/share/virtualenvs/photos-to-sqlite-0uGSHd6e/bin/photos-to-sqlite", line 11, in <module>
load_entry_point('photos-to-sqlite', 'console_scripts', 'photos-to-sqlite')()
File "/Users/simon/.local/share/virtualenvs/photos-to-sqlite-0uGSHd6e/lib/python3.8/site-packages/click/core.py", line 829, in __call__
return self.main(*args, **kwargs)
File "/Users/simon/.local/share/virtualenvs/photos-to-sqlite-0uGSHd6e/lib/python3.8/site-packages/click/core.py", line 782, in main
rv = self.invoke(ctx)
File "/Users/simon/.local/share/virtualenvs/photos-to-sqlite-0uGSHd6e/lib/python3.8/site-packages/click/core.py", line 1259, in invoke
return _process_result(sub_ctx.command.invoke(sub_ctx))
File "/Users/simon/.local/share/virtualenvs/photos-to-sqlite-0uGSHd6e/lib/python3.8/site-packages/click/core.py", line 1066, in invoke
return ctx.invoke(self.callback, **ctx.params)
File "/Users/simon/.local/share/virtualenvs/photos-to-sqlite-0uGSHd6e/lib/python3.8/site-packages/click/core.py", line 610, in invoke
return callback(*args, **kwargs)
File "/Users/simon/Dropbox/Development/photos-to-sqlite/photos_to_sqlite/cli.py", line 249, in apple_photos
photo_row = osxphoto_to_row(sha256, photo)
File "/Users/simon/Dropbox/Development/photos-to-sqlite/photos_to_sqlite/utils.py", line 91, in osxphoto_to_row
place = photo.place
File "/Users/simon/.local/share/virtualenvs/photos-to-sqlite-0uGSHd6e/lib/python3.8/site-packages/osxphotos/photoinfo.py", line 614, in place
self._place = PlaceInfo5(self._info["reverse_geolocation"])
File "/Users/simon/.local/share/virtualenvs/photos-to-sqlite-0uGSHd6e/lib/python3.8/site-packages/osxphotos/placeinfo.py", line 505, in __init__
self._plrevgeoloc = archiver.unarchive(revgeoloc_bplist)
File "/Users/simon/.local/share/virtualenvs/photos-to-sqlite-0uGSHd6e/lib/python3.8/site-packages/bpylist/archiver.py", line 16, in unarchive
return Unarchive(plist).top_object()
File "/Users/simon/.local/share/virtualenvs/photos-to-sqlite-0uGSHd6e/lib/python3.8/site-packages/bpylist/archiver.py", line 256, in top_object
return self.decode_object(self.top_uid)
File "/Users/simon/.local/share/virtualenvs/photos-to-sqlite-0uGSHd6e/lib/python3.8/site-packages/bpylist/archiver.py", line 247, in decode_object
obj = klass.decode_archive(ArchivedObject(raw_obj, self))
File "/Users/simon/.local/share/virtualenvs/photos-to-sqlite-0uGSHd6e/lib/python3.8/site-packages/osxphotos/placeinfo.py", line 126, in decode_archive
mapItem = archive.decode("mapItem")
File "/Users/simon/.local/share/virtualenvs/photos-to-sqlite-0uGSHd6e/lib/python3.8/site-packages/bpylist/archiver.py", line 140, in decode
return self._unarchiver.decode_key(self._object, key)
File "/Users/simon/.local/share/virtualenvs/photos-to-sqlite-0uGSHd6e/lib/python3.8/site-packages/bpylist/archiver.py", line 216, in decode_key
return self.decode_object(val)
File "/Users/simon/.local/share/virtualenvs/photos-to-sqlite-0uGSHd6e/lib/python3.8/site-packages/bpylist/archiver.py", line 247, in decode_object
obj = klass.decode_archive(ArchivedObject(raw_obj, self))
File "/Users/simon/.local/share/virtualenvs/photos-to-sqlite-0uGSHd6e/lib/python3.8/site-packages/osxphotos/placeinfo.py", line 180, in decode_archive
sortedPlaceInfos = archive.decode("sortedPlaceInfos")
File "/Users/simon/.local/share/virtualenvs/photos-to-sqlite-0uGSHd6e/lib/python3.8/site-packages/bpylist/archiver.py", line 140, in decode
return self._unarchiver.decode_key(self._object, key)
File "/Users/simon/.local/share/virtualenvs/photos-to-sqlite-0uGSHd6e/lib/python3.8/site-packages/bpylist/archiver.py", line 216, in decode_key
return self.decode_object(val)
File "/Users/simon/.local/share/virtualenvs/photos-to-sqlite-0uGSHd6e/lib/python3.8/site-packages/bpylist/archiver.py", line 247, in decode_object
obj = klass.decode_archive(ArchivedObject(raw_obj, self))
File "/Users/simon/.local/share/virtualenvs/photos-to-sqlite-0uGSHd6e/lib/python3.8/site-packages/bpylist/archiver.py", line 112, in decode_archive
return [archive._decode_index(index) for index in uids]
File "/Users/simon/.local/share/virtualenvs/photos-to-sqlite-0uGSHd6e/lib/python3.8/site-packages/bpylist/archiver.py", line 112, in <listcomp>
return [archive._decode_index(index) for index in uids]
File "/Users/simon/.local/share/virtualenvs/photos-to-sqlite-0uGSHd6e/lib/python3.8/site-packages/bpylist/archiver.py", line 137, in _decode_index
return self._unarchiver.decode_object(index)
File "/Users/simon/.local/share/virtualenvs/photos-to-sqlite-0uGSHd6e/lib/python3.8/site-packages/bpylist/archiver.py", line 247, in decode_object
obj = klass.decode_archive(ArchivedObject(raw_obj, self))
File "/Users/simon/.local/share/virtualenvs/photos-to-sqlite-0uGSHd6e/lib/python3.8/site-packages/osxphotos/placeinfo.py", line 217, in decode_archive
placeType = archive.decode("placeType")
File "/Users/simon/.local/share/virtualenvs/photos-to-sqlite-0uGSHd6e/lib/python3.8/site-packages/bpylist/archiver.py", line 140, in decode
return self._unarchiver.decode_key(self._object, key)
File "/Users/simon/.local/share/virtualenvs/photos-to-sqlite-0uGSHd6e/lib/python3.8/site-packages/bpylist/archiver.py", line 216, in decode_key
return self.decode_object(val)
File "/Users/simon/.local/share/virtualenvs/photos-to-sqlite-0uGSHd6e/lib/python3.8/site-packages/bpylist/archiver.py", line 227, in decode_object
raise CircularReference(index)
bpylist.archiver.CircularReference: archive has a cycle with uid(13)
In the debugger I traced this back to:
178 @staticmethod
179 def decode_archive(archive):
180 -> sortedPlaceInfos = archive.decode("sortedPlaceInfos")
181 finalPlaceInfos = archive.decode("finalPlaceInfos")
182 return PLRevGeoMapItem(sortedPlaceInfos, finalPlaceInfos)
The upload
command currently only handles static images:
Need to cover movies taken by my phone and DSLR too.
I want to share a subset of my photos, without sharing everything. Idea:
$ photos-to-sqlite create-subset photos.db public.db "select sha256 from ... where ..."
So the command takes a SQL query that returns sha256 hashes, then creates a new file called public.db
containing just the data corresponding to those photos.
It can detect faces, run OCR, do image labeling (it knows what a lemur is!) and do object localization where it identifies objects and returns bounding polygons for them.
Faces, albums, locations, that kind of thing.
A declarative, efficient, and flexible JavaScript library for building user interfaces.
๐ Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. ๐๐๐
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google โค๏ธ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.