On windows 10 if you do the install command $ go get -u github.com/mholt/timeliner/cmd/timeliner you get the above error.

Not sure how to resolve it. I have the latest go version by the way.

Hi, I just tried installing timeliner on Windows 10 and noticed that the go-sqlite3 dependency requires gcc.

On Windows, gcc is not included and has to be installed manually. Maybe you could add a note about that in the install section of the readme, as long as there will be no binaries available for download.

Using chocolatey, this installs the necessary dependencies on Windows:

choco install golang mingw

Documenting how the database is structured could be useful for starting work on an interface. I'm looking at it and without some digging it's hard to understand what some of the stuff is. For example the "class" for "items" is, I assume based on this however it's not clear

OS: macOS (latest)
GO: go version go1.12.6 darwin/amd64

I am getting this error. Everything is done as per README.md

2019/06/16 19:58:44 [ERROR][google_photos/[email protected]] Processing item graph: processing node of item graph: downloading data file: syncing file: sync timeliner_repo/data/2019/06/google_photos/IMG_5773.JPG: operation not supported (item_id=6)

Another error:

2019/06/16 20:00:21 [ERROR][google_photos/[email protected]] Processing item graph: processing node of item graph: storing item in database: disk I/O error (item_id=XXXXXXX)

I've started this, but pulling content from the Twitter API does not currently add relationships between tweets (like replies).

On trying to use timeliner with Google Photos I am constantly getting a Bad Request error as follows:

 Get https://photoslibrary.googleapis.com/v1/albums?pageSize=50&pageToken=: oauth2: cannot fetch token: 400 Bad Request
Response: {
  "error": "invalid_grant",
  "error_description": "Bad Request"
} <<< - retrying... (attempt 5/10)

Any ideas?

1. What is the data source you want to add?

SMS Backup & Restore (or on Google Play) is the only Android app that I've been satisfied with for backing up my text messages (both SMS and MMS). (I don't want to argue which app is better, or answer why I don't use another app.)

(Sigh, I just found out that it improperly encodes emoji characters. Filed a bug.)

If there is a more standard way of archiving text messages, I'd be willing to use that, but for now, this is the easiest and most straightforward method I could find that works for me.

2. How are items obtained from the data source?

By file import. The app exports all text messages on the device when a backup is performed and stores it locally or on a cloud service. I have mine upload to cloud storage on a weekly basis. It's an XML file that has all the data and metadata that's relevant.

2a. If authentication is required, how does a user create or obtain credentials for Timeliner to access the data?

n/a

2b. If an API is available, what are its rate limits?

n/a

2c. If a file is imported, how is the file obtained?

Transferred from device or downloaded from cloud storage. The app can produce these automatically on a schedule.

2d. If a file is imported, how do we read the file?

There is some (incomplete) documentation here: https://synctech.com.au/sms-backup-restore/fields-in-xml-backup-files/

3. What constitutes an "item" from this data source?

SMS or MMS messages or files.

4. How can items from this data source be related?

Messages can be replies to each other, but this is not really explicit, since SMS and MMS are basically stateless. Most phones just group SMS conversations by phone number ("address") and then thread replies implicitly through their timestamps. MMS conversations have one or more addresses (phone numbers), and all messages with the same set of addresses are implicitly a grouped conversation.

So, relationships of replies can be inferred from timestamp without explicitly saying that one is a reply to another.

5. What constitutes a "collection" from this data source?

Perhaps a conversation between recipients, but again this is implied through the address(es) on the SMS or MMS.

6. What might not be trivial, obvious, or straightforward when implementing this data source?

MMS can be files, SMS is just text. MMS can be just text but might have "rich" text (or emojis) or multiple recipients ("group texts").

SMS and MMS are separate in the data file, but SMS can be related to MMS, and vice-versa. We will probably need to somehow treat them as one list if we want to establish explicit relationships between them.

There's no concept of accounts, but the account ID will need to be the owner's phone number.

All phone numbers (including account ID) will need to be standardized. Recommend E164 format.

A default region/country will need to be available (default to US) if a phone number does not have an explicit country calling code.

Lots of metadata fields are strings with a value of "null" which may need to be explicitly converted to empty or nil string.

I'm running $ timeliner -twitter-replies -twitter-retweets get-latest twitter/joonas_fi:

2019/11/30 16:36:03 [ERROR][twitter/joonas_fi] Getting latest:
	getting items from service:
	processing tweet from API:
	processing tweet 123:
	making item from tweet that this tweet (123) is in reply to (456):
	making item from tweet that this tweet (456) is in reply to (789):
	making item from tweet that this tweet (789) is in reply to (AAA):
	getting tweet that this tweet (AAA) is in reply to (BBB):
	HTTP error: https://api.twitter.com/1.1/statuses/show.json?id=BBB&tweet_mode=extended:
	429 Too Many Requests

(anonymization and line breaks added by me)

Timeliner configures its Twitter client's rate limit ("with some leeway") to 5 900/h. Bursting is disabled for Twitter, so it's 610ms between requests.

What I think is the problem

twitter/api.go use three API endpoints:

Source for limits: https://developer.twitter.com/en/docs/basics/rate-limits

Timeliner's ratelimit is shared across all of those endpoints. Timeliner theoretically lets me do 1 475 reqs/15 min to /statuses/show.json - pushing it over the limit of 900. Now I don't know for real what the ratios of the endpoint call rates are, but if we were to avoid going over the limit with this current "all endpoints have same rate limit" -design, the limit should be re-calculated based on the 900 number.

Another thought: is the 1 500 correct even for user_timeline.json?

This depends on the authorization model, if Timeliner:

a) uses only the app's credentials to read public data and doesn't get authorization from the user
b) gets authorization from the user and operates on behalf of the user (I don't remember seeing any authorization screen, but that might be because I had an API key laying around which I had authorized way back)

A few quotes:

Rate limiting of the standard API is primarily on a per-user basis — or more accurately described, per user access token.

When using application-only authentication, rate limits are determined globally for the entire application. If a method allows for 15 requests per rate limit window, then it allows you to make 15 requests per window — on behalf of your application. This limit is considered completely separately from per-user limits.

Source: https://developer.twitter.com/en/docs/basics/rate-limiting

If I understand correctly, Timeliner is not using "When using application-only authentication", so shouldn't the limit be based on the 900 anyway?

I'm not sure of this. WDYT?

Workaround

This is not a serious issue, since after throttling I can wait a while and continue later.

Code suggestions

I don't know if you're interested in code suggestions, but a couple came to mind while kicking the tires:

1. I came across timeliner.FakeCloser, just in case you're not aware there's a stdlib implementation for that: https://godoc.org/io/ioutil#NopCloser
2. ratelimit.go: you're chucking empty structs (struct{}) on the token channel. Usually when a channel is only used for signalling, I've seen interface{} used so one can just chuck nil's down the channel. I'm not sure if it's more performant but I think it's more semantic. This might be subjective though.

I haven't tested your project yet, but it seems awesome. I'd like to suggest you to consider using Metabase https://www.metabase.com/ over TablePlus. The installation process is very simple and I think this project could benefit a lot from a visualization tool such as Metabase.

Excellent work with timeliner Matt :)

Can I suggest an Amazon data source? Specifically things like order history etc. Or is that out of scope?

Here's the Tweet: https://twitter.com/janl/status/1113015555064201216

Error message:

2019/11/30 18:04:02 [ERROR][twitter/joonas_fi] Getting latest: getting items from service: processing tweet from API: processing tweet 1113180316510957568: making item from tweet that this tweet (1113180316510957568) is in reply to (1113015555064201216): making item from tweet that this tweet (1113015555064201216) embeds (1112473455650172929): media resource returned HTTP status 403 Forbidden: https://pbs.twimg.com/ext_tw_video_thumb/1112471832232259585/pu/img/ywWGTl09hsnLnMOY.jpg

That image URL redirects (when used with browser - different when API use?) to this DMCA warning.

Timeliner cannot cope with this, and trying to re-run Timeliner always gets me this and cannot continue.

Currently on go 1.13.5

root@778c41ca01ec:~/timeliner/cmd/timeliner# ./timeliner add-account twitter/pfrcks                  
panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0x0 pc=0x7c363f]

goroutine 1 [running]:
github.com/mholt/timeliner.authorizeWithOAuth2(0xb24986, 0x7, 0x0, 0x0, 0x0, 0x80, 0xa9a0c0, 0xc000130f01, 0xc0000f6e00, 0xc000130f70)
        /root/timeliner/oauth2.go:57 +0x5f
github.com/mholt/timeliner.DataSource.authFunc.func1(0x7ffcf4a2cee9, 0x6, 0xb246df, 0x7, 0xb24986, 0x7, 0x0)
        /root/timeliner/datasource.go:116 +0x5e
github.com/mholt/timeliner.(*Timeline).Authenticate(0xc0001e77c0, 0x7ffcf4a2cee1, 0x7, 0x7ffcf4a2cee9, 0x6, 0x0, 0xc0001e77e0)
        /root/timeliner/account.go:79 +0x582
github.com/mholt/timeliner.(*Timeline).AddAccount(0xc0001e77c0, 0x7ffcf4a2cee1, 0x7, 0x7ffcf4a2cee9, 0x6, 0x1, 0x0)
        /root/timeliner/account.go:63 +0x2e5
main.main()
        /root/timeliner/cmd/timeliner/main.go:94 +0x554
root@778c41ca01ec:~/timeliner/cmd/timeliner# 

Hey guys,

I'm getting the following error that prevents me from installing timeliner:

go get -u github.com/mholt/timeliner/cmd/timeliner
package github.com/pierrec/lz4/v3: cannot find package "github.com/pierrec/lz4/v3" in any of:
        /usr/lib/go-1.10/src/github.com/pierrec/lz4/v3 (from $GOROOT)
        /home/ngirardin/go/src/github.com/pierrec/lz4/v3 (from $GOPATH)

Best,

Nicolas

1. What is the data source you want to add?

Facebook data exports. Currently there's a date source for data from the API, but a Timeliner user might have already closed their account, leaving behind only their data.

2. How are items obtained from the data source?

Data must be manually imported. It may be sourced from some or all of Facebook's different offerings. Documentation is here.

2a. If authentication is required, how does a user create or obtain credentials for Timeliner to access the data?

N/A

2b. If an API is available, what are its rate limits?

N/A

2c. If a file is imported, how is the file obtained?

Documentation here.

2d. If a file is imported, how do we read the file?

Exports are a ZIP file containing a structure of folders, JSON files, and various media files. The media files have their EXIF data, but additional metadata from Facebook corresponding to each one is in the associated JSON file.

3. What constitutes an "item" from this data source?

It depends how much of the export is considered useful, but photos, posts, milestones etc. should all be considered.

4. How can items from this data source be related?

This should be looked into more closely by someone who actually uses Facebook – my memory for it isn't great as I type this.

5. What constitutes a "collection" from this data source?

Facebook has the concept of photo albums, so they'd constitute a collection. Other than that, I doubt there are many others.

6. What might not be trivial, obvious, or straightforward when implementing this data source?

I'm not sure how easy it would be to make the relationship between data from the JSON files and actual media files themselves clearer.

Bonus: How do you like Timeliner? How much data are you preserving with it? Which existing data sources do you use?

I haven't started using it yet, but I figure a Facebook data export would serve as the foundation for my starting to use this on my VPS.

I'm closing my Facebook account as I type this, but it'd be good to hang on to the backed-up data in a way I might be able to browse in future. I've observed from the wiki that there's not yet support for loading from a full backup taken from Facebook (whether HTML or JSON). I'd hope there's some support for this thanks to this issue.

I get this error when attempting ./timeliner add-account google_photos/[email protected]

panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0x0 pc=0x4367335]

goroutine 1 [running]:
github.com/mholt/timeliner.authorizeWithOAuth2(0x463ed5a, 0x6, 0x4a16410, 0x1, 0x1, 0x4363f34, 0x45cde60, 0xc000100200, 0x0, 0x0)
	/Users/craig/go/src/github.com/mholt/timeliner/oauth2.go:57 +0x65
github.com/mholt/timeliner.DataSource.authFunc.func1(0x7ffeefbffade, 0x18, 0x4641041, 0xd, 0x463ed5a, 0x6, 0x4a16410)
	/Users/craig/go/src/github.com/mholt/timeliner/datasource.go:116 +0x3c
github.com/mholt/timeliner.(*Timeline).AddAccount(0xc00000d1a0, 0x7ffeefbffad0, 0xd, 0x7ffeefbffade, 0x18, 0x1, 0x0)
	/Users/craig/go/src/github.com/mholt/timeliner/account.go:69 +0x754
main.main()
	/Users/craig/go/src/github.com/mholt/timeliner/cmd/timeliner/main.go:82 +0x3b9

Currently, the metadata is stored as a blob in the database and it's not straightforward to read and parse from within SQL or other programming languages.

I would suggest to store the metadata in separate tables, but I am not sure what a good design would be for that.

A short-term solution could be to store it as serialized text (e.g. json).

What do you think?

➜ ./timeliner add-account google_photos/[email protected]
panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0x0 pc=0x7439b0]

goroutine 1 [running]:
github.com/mholt/timeliner.authorizeWithOAuth2(0x9d6631, 0x6, 0xe35f60, 0x1, 0x1, 0x7405bc, 0x80, 0x95bb40, 0x4000126f28, 0x7405f8)
/home/sainyam/go/src/github.com/mholt/timeliner/oauth2.go:57 +0x50
github.com/mholt/timeliner.DataSource.authFunc.func1(0x7fdc21d5fa, 0x20, 0x9d8bab, 0xd, 0x9d6631, 0x6, 0xe35f60)
/home/sainyam/go/src/github.com/mholt/timeliner/datasource.go:116 +0x68
github.com/mholt/timeliner.(*Timeline).Authenticate(0x400000f0a0, 0x7fdc21d5ec, 0xd, 0x7fdc21d5fa, 0x20, 0x0, 0x400000f0c0)
/home/sainyam/go/src/github.com/mholt/timeliner/account.go:79 +0x4a8
github.com/mholt/timeliner.(*Timeline).AddAccount(0x400000f0a0, 0x7fdc21d5ec, 0xd, 0x7fdc21d5fa, 0x20, 0x1, 0x0)
/home/sainyam/go/src/github.com/mholt/timeliner/account.go:63 +0x2f0
main.main()
/home/sainyam/go/src/github.com/mholt/timeliner/cmd/timeliner/main.go:91 +0x48c

This is happening at the first stage,

https://github.com/perkeep/perkeep

Timeliner seems pretty similar, if less complex...

The Telegram desktop app has an option "Export Telegram data" as a json (or html) file including chats, contacts, media and a whole lot more...

I will try to integrate it if I find some time ;)

Would be useful to be able to download my own photos from Instagram.

I get the following error message, and timeliner consistently doesn't download all pictures

2019/05/29 05:02:01 [ERROR][google_photos/xxx@xxx] Processing item graph: processing node of item graph: assembling item for storage: getting item metadata: parsing width as int: strconv.Atoi: parsing "": invalid syntax (width=)

I'm trying to use this directly on my NAS (a Synology DS218j), so I'm runnig it via Linux command line with no browser access.

I have timeliner installed and ready to run, but I can't authenticate as I can't open a browser within command line.

Is there a way to get around that and authenticate via the command line?

I put in the command to setup google photos. It pulls up the browser to login. I get all the way to where I'm confirming my choices to allow my app access. Then it just churns with a "waiting for localhost" in the bottom left. Am I just not patient enough? Does this take a long time?

Test data and db with all available features used (images, videos, posts, relationships etc) would be useful for development of an interface as it would allow for development of features even if you don't have your own data with all features.

Can this notify you when a data sources data has changed ?
For example when you added a new google photo. Or if you edited it or deleted it.
Its basically an Event Stream.
Alot liek CDC. https://en.wikipedia.org/wiki/Change_data_capture

Not all Data sources would support this i am guessing.

Anyway then the system would be able to see the event and then go get the blob if it wanted to.

just an idea.

I suggest documenting the flags, and their usage, in the readme file.
They can be found in main.go or via "timeliner -help"

For example:

timeliner -repo /Volumes/XDrive get-all google_photos/[email protected]

My Google Photos storage reports 10.31GB used, but a full rip with timeliner only comes up to 6.7GB.

With google file conversion (such as gdoc => docx).

Someone correct me if I'm wrong but it appears videos are excluded in the get-all command.
It would be nice if they were included as well.

In Europe, a personal data archive can be downloaded from https://privacy.apple.com/

Your download will include:

    App usage and activity information as spreadsheets or files in JSON, CSV, XML or PDF format.
    Documents, photos and videos in their original format.
    Contacts, calendars and bookmarks in VCF, ICS or HTML format.

Your download will not include app, book, film, TV programme or music purchases.

Detailed list of apps and services:

    Apple Media Services information
    Apple ID account and device information
    Apple Online Store and Retail Store activity
    Apple Pay activity
    AppleCare support history, repair requests and more
    Game Center activity
    iCloud Bookmarks and Reading List
    iCloud Calendars and Reminders
    iCloud Contacts
    iCloud Drive files and documents
    iCloud Mail
    iCloud Notes
    iCloud Photos
    Maps Report an Issue
    Marketing communications, downloads and other activity
    Other data

Hi all,
Just came across this project. In the wiki it mentions that location data is stripped when using the Google Photos API?
I am currently using rclone to pull and backup my Google Photos account to my personal server through Google Drive. All of my media pulled using rclone retains the GPS data (lat/lon/alt).

Two thoughts.

1. I would rather use my own script (or some other program) to sync my Google Photos account. Would it be easy for timeliner to import from a local folder?
2. I am not familiar with the Google Photos API, but location data is pretty important (especially if you don't have Location History). It seems alternative methods to import photos with the location data would be of high importance.

hiya,

I'm having a little trouble with oauth tokens.

I've set up timeliner and a google photos data source and managed to start downloading 9GB+.

Now, I receive an error saying the token needs to be refreshed (below)

I've tried creating new API keys and removing timerliner from my appl permissions within google but no joy. I haven't even been prompted to re-add app permissions within google.

Any help would be great

Dan

Currently when I'm either downloading from sources or importing files, very little output is written to the screen. I'd love it if I can see some kind of progress being made or some kind of logging information being printed. Even if I have to add a flag to get it.

Thanks for this great tool.

Any idea what must be provoking this error:
Processing item graph: processing node of item graph: replacing data file with identical existing file: querying DB: database is locked

First of all thank you @mholt for this tool. I was looking at the source code to implement my own file writer so that I can save files in remote location ex. perkeep, s3 and others. How hard do you think is to abstract that part?
Thanks.

Type

Should it have a web interface or a desktop one? Maybe both? Possibly add an API to make it easier to create different types of interfaces. Should it be available from other devices? For example through the web or a phone app. Should import be available through the interface and if you have "external" interfaces (phone, web etc) should you be able to import it from there?

I think an API and a local only interface for import. Import (and other modification) shouldn't be available on other devices unless password protection is added.

Design (for a graphical interface)

A timeline sounds like a good idea (based of name). Should you be able to sort it by type (video, audio, image, place etc)? Separate more detailed views for each month while only providing a overview on the timeline?

I think a timeline with general month views and then a more detailed monthly view would be good, since it's supposed to be a long term collection then a general yearly view might be good too.

Any more suggestions?

[ERROR][twitter/jacroe] Downloading all: getting items from service: getting next page of tweets: reading response body: json: cannot unmarshal number into Go struct field tweetGeo.coordinates of type string

Unsure what causes this error; this is just immediately after I run timeliner get-all twitter/jacroe.

Hello Google plus is shutting down end of march 2019. Is that possible to include it ?

I made a proof of concept of Slack=>Gmail to benefit of Gmail search engine. But a local version could be even better !

I'd like to try Timeliner out, but I'm having difficulties with installing it. I've also never used Go before.

I've spent a lot of time trying and failing under Windows 10 and Ubuntu 18 LTS under WSL. (I haven't found any info re supported platforms. If I'd just missed it, please point me in the right direction.)

I'd installed golang successfully, then tried the go get -u github.com/mholt/timeliner/cmd/timeliner.
At first it threw up errors about missing 'gcc'. I've tried installing MinGW on Windows and one of the suggested packages from apt on Ubuntu.

After that, the Ubuntu go get just ran for a while, then died without printing any messages. Under Windows it threw up an error about a failing SQLite compilation and died as well.

Are these just completely unsupported scenarios? How should I go about running Timeliner?
(I've been writing something similar in the past month, so I'm especially eager to try it out.)

I couldn't find in the documentation, how does timeliner handle dupliates?
let say if I have same photo on google photos and facebook, when it downloads it, can it detect duplicates or warn me?

or does it automatically keep one of them ?
what happens in case of date conflict ? like one of them has a wrong date and one of them correct date.

Possible data source in the future.

1. What is the data source you want to add?

Gmail - the ability to download emails and attachments, with very heavy filtering options. Having a record of emails would be useful because my family often documents their lives to the rest of us through emails.

2. How are items obtained from the data source?

Gmail API via Google OAuth2

2a. If authentication is required, how does a user create or obtain credentials for Timeliner to access the data?

Google OAuth2 is already documented in the wiki.

2b. If an API is available, what are its rate limits?

250 queries per user per second: https://developers.google.com/gmail/api/v1/reference/quota

2c. If a file is imported, how is the file obtained?

n/a

3. What constitutes an "item" from this data source?

An email or attachment.

4. How can items from this data source be related?

Emails can have attachments. Emails can be replies to other emails.

5. What constitutes a "collection" from this data source?

Perhaps a conversation (a thread of messages); but that might be redundant if emails can be related as replies to other ones.

6. What might not be trivial, obvious, or straightforward when implementing this data source?

A lot of filtering will need to be configurable, since Gmail contains a messy amount of noise; somehow I have Google Voice messages, Hangout chats, and other things that aren't really emails in my Gmail account which also show up in the API.

Will need to filter by label, subject, sender, To, other headers, etc. Optionally include attachments (from some emails only, or files of a certain type or size, etc).

I'd like my retrieved photos and videos to have a creation/modification date as is in the filename of the files retrieved from Google Photos.