king-millez / snapmap-archiver Goto Github PK

I think git messed up the coordinates.py file with a capital C which VS Code didn't catch.

I'll fix this some time but it's time for bed rn

A better way to name the downloaded Snaps would be nice, rather than [SNAP_ID].mp4, which isn't really searchable. The data returned by utils.organise_data() could be used, which includes the Snap location and the timestamp from when it was posted.

The actual API also contains some extra data for certain snaps, like the raw text used for a video.

At the moment I'm thinking a good naming convention could be [MINOR_LOCATION] - [TIMESTAMP] - [SNAP_ID].mp4...

When querying the API, you can pass the parameter radiusMeters, which will give you more/less specific content the lower/higher the number you provide is. Starting from 95000 and working down to 0, we could iterate through the maximum amount of content in a specific area. I'm thinking iterating by -2500 for values > 10000 would be good, then -1000 until 1000, then -100 to 100, then -10 to 0, arbitrarily ending with 1.

I notice you're using Aria for downloading the videos. Is there a significant enough time difference using the external library over say a piece of code like

import requests
...
with open(filename + '.mp4', 'w+') as f:
   # req_headers from get_data.py
   f.write(requests.get(snap['media']['raw_url'], headers=req_headers))

now that above code most likely won't run out of the box but I feel removing the Aria requirement would make it more attractive to people as they could get it running straight from pip

And sorry for opening an issue to discuss this, I don't know where the best place to talk about the code is.

I have the impression that when you export the json with the command --write-json the file generates this kind of output:
"create_time": "1673031690121"

If we do :
date -d @1673031690121
Thu April 06, 54986 19:02:01 CEST

but if we remove the last three characters :
date -d @1673031690
Fri Jan 06 2023 20:01:30 CET

which seems to correspond more to the real date.

I assume the date is taken by snapmap explorer, but there seems to be a format problem.

There is a type of video that isn't handled by the tool, currently throws an error when queried.

I've tried this with a few different locales, and it seems not to be working:

➜ snapmap-archiver -o . -l='decimal.lat,decimal.long' -r 10460 --write-json -t 7d
2024-05-20 10:30:45.564 | INFO     | snapmap_archiver.SnapmapArchiver:__init__:52 - Skipping Snaps older than [1715610645].
Location: (decimal.lat,decimal.long) |██████████████████████████████████████▎ | ▇▇▅ 96% [10000/10460] in 9s (~0s, 1091.9/s) 2024-05-20 10:30:55.006 | WARNING  | snapmap_archiver.SnapmapArchiver:_coordinate_query_failure:223 - You have been rate limited. Sleeping for [60] seconds...
Location: (decimal.lat,decimal.long) |██████████████████████████████████████▎ | ▄▂▂ 96% [10000/10460] in 1:24 (~4s, 119.9/s) 2024-05-20 10:32:09.328 | WARNING  | snapmap_archiver.SnapmapArchiver:_coordinate_query_failure:223 - You have been rate limited. Sleeping for [60] seconds...
Location: (decimal.lat,decimal.long) |██████████████████████████████████████▎ | ▂▄▆ 96% [10000/10460] in 1:46 (~5s, 94.1/s)

The average rate slowly approaches 0.0/s over the course of many more retries and eventually just hangs. Perhaps a bug was introduced in the refactor?

• While Snapmap Archiver has been decoupled significantly thanks to #13, there is still work to be done to clean up the main SnapmapArchiver.py file and move many of its functions elsewhere. There is also some ugly state management where there are multiple snap caches in use at one time. This can be cleaned.
• print() is handling all the logging at the moment, which should change in favour of a proper logging library.
• There are no tests.

The ability to constrain archived Snaps not only by location but also by time would be useful. Just to illustrate the idea:

snapmap-archiver -o . -l='LAT,LONG' -r 5000 -t 5d  --write-json

snapmap-archiver -o . -l='LAT,LONG' -r 5000 -t 36h --write-json

The new option being -t which accepts some concept of time in either hours or days (minutes seems unneccesasry, but 🤷‍♂️)

Hello, I can't get the new version 2.0.1 to work.

my command :
snapmap-archiver -o ~/Desktop/projet/ -l='123.123,123.123' -l '445.445,445.445'
-r 1000

my error :
'''
Traceback (most recent call last):
File "/home/user/.local/bin/snapmap-archiver", line 8, in
sys.exit(main())
File "/home/user/.local/lib/python3.10/site-packages/snapmap_archiver/init.py", line 34, in main
sm_archiver.main()
File "/home/user/.local/lib/python3.10/site-packages/snapmap_archiver/SnapmapArchiver.py", line 128, in main
self.download_snaps(self.query_coords(coords))
File "/home/user/.local/lib/python3.10/site-packages/snapmap_archiver/SnapmapArchiver.py", line 122, in query_coords
return self._transform_index(to_download.values())
TypeError: SnapmapArchiver._transform_index() takes 1 positional argument but 2 were given
'''

Snaps with text and stickers don't include said graphics in the video file, instead they're stored in an image called overlay.png, and displayed over the top by the browser/app.

We could have an option like --merge-overlay which would use something like ffmpeg or avconv to put the image over the top of the media.mp4 file and export it as a new file.

Nvm