tagatac / bagoup Goto Github PK

There has been a report of exports containing blank messages starting on 10/24/2022, the release date of Mac OS Ventura.

Is your feature request related to a problem? Please describe.
Currently (as far as I could tell) there is no way to back up messages within a specified time period, e.g. within the last year or from 2019-2020.

Describe the solution you'd like
Some option that allows the user to backup messages within a specific date range.

Additional context
I think this would pair well with #33, if you're considering both for implementation. I only really need to offload older messages so I can free up iCloud storage, so these two features in tandem would be fantastic for what I'm trying to do.

It seems that the messages are not ordered, but extracted as is from the database?

For older messages in an iMessage file iMessage;-;SENDER_ADDRESS.txt, the history seems to be out of order. It seems that at least the years are "somewhat ordered" but not consistently. I'm unsure what ordering this is.

Also I noticed that the order for iMessage files whose sender address is an email is roughly descending (above new, bottom old), while for a sender address that's a phone number it's the opposite, roughly ascending (above old, bottom new).

It would probably make sense to order them, before writing the file. Ideally, this would also work well together with #8 which merges iMessage files of the same contact with different sender addresses.

Describe the bug
Using -a to export attachments which are directories (such as Apple iWork documents) produces an error which halts execution

To Reproduce
Steps to reproduce the behavior:

1. Send an Apple Pages file as an attachment to an iMessage
2. Execute bagoup -c example.vcf -a
3. A series of WARN messages are logged and then:

BAGOUP RESULTS:
bagoup version: 2.1.2 Darwin/x86_64
Export folder: "messages-export"
Export files written: 317
Chats exported: 345
Messages exported: 6781
Attachments copied: 866
	image/heic: 7
	image/jpeg: 500
	image/tiff: 14
	video/quicktime: 4
	application/pdf: 4
	text/x-vlocation: 3
	text/vcard: 2
	video/3gpp: 1
	image/png: 56
	application/octet-stream: 271
	image/gif: 2
	audio/amr: 2
Attachments referenced or embedded: 872
	application/pdf: 4
	image/heic: 7
	image/png: 56
	application/octet-stream: 273
	image/tiff: 14
	video/quicktime: 4
	text/x-vlocation: 3
	image/jpeg: 504
	image/gif: 2
	audio/amr: 2
	text/vcard: 2
	video/3gpp: 1
Attachments embedded: 0
Attachments missing (see warnings above): 6
HEIC conversions completed: 0
HEIC conversions failed (see warnings above): 0
Time elapsed: 5.305668045s

4. The error is then shown:

ERROR: export chats: chat file "messages-export/<contact>/iMessage;-;[email protected]" - message <message_id>: copy attachment "/Users/$USERNAME/Library/Messages/Attachments/<path/to/attachment>/example.pages" to "messages-export/<contact_name>/attachments": read /Users/$USERNAME/Library/Messages/Attachments/<path/to/attachment>/example.pages: is a directory

5. bagoup terminates with the remaining messages from that contact and subsequent contacts (alphabetically) not exported

Expected behavior

• Attachments using a directory structure should be created in the export path using the same directory structure
• The error should not be considered fatal; execution should continue from the next message

Environment (please complete the following information):

• OS: [e.g. Mac OS 12.5] MacOS 12.6
• Version [e.g. v2.0.4] (obtain with bagoup --version) bagoup version 2.1.2 Darwin/x86_64

Describe the bug
When the command is run without --separate-chats flag, it tries to merge chat history from multiple phone numbers and emails under the same contacts, and it creates a .txt file by naming it with the concatenation of all those phone numbers and emails, and this could create a file name exceeds the length limit.

Probably not many people will have this problem though.

To Reproduce
Steps to reproduce the behavior:
just run the commands without --separate-chats, and if you have a contacts with more than 4 phone numbers and more than 4 emails, you will likely see "file name too long" error.

Expected behavior
Should truncate the file name if too long.

Environment (please complete the following information):

• OS: macOS 12.6
• Version 2.1.2 Darwin/arm64

Add a parameter for specifying the location of copied attachments from which to create PDFs.

Installing using Homebrew fails with error:

Running `brew update --auto-update`...
==> Auto-updated Homebrew!
Updated 2 taps (homebrew/core and homebrew/cask).
==> New Formulae
bdftopcf                   liburing                   weasyprint
fonts-encodings            lmfit

==> Downloading https://ghcr.io/v2/homebrew/core/make/manifests/4.3-1
Already downloaded: /Users/damon/Library/Caches/Homebrew/downloads/55bed1d09351ed4efb7121971eb6d802f16f0d4cc9f3b46632118e93c0dd9519--make-4.3-1.bottle_manifest.json
==> Downloading https://ghcr.io/v2/homebrew/core/make/blobs/sha256:4349e2c715f78
Already downloaded: /Users/damon/Library/Caches/Homebrew/downloads/dba5044d6af4d00b1c3648ab951e079cad358e298f66296f453265b47c0061db--make--4.3.monterey.bottle.1.tar.gz
==> Downloading https://github.com/tagatac/bagoup/archive/refs/tags/v2.1.1.tar.g
Already downloaded: /Users/damon/Library/Caches/Homebrew/downloads/02a49b0d4ead99915bb047de4e3feb4fd503e3d076290741e890a9de496fb243--bagoup-2.1.1.tar.gz
==> Installing bagoup from tagatac/bagoup
Error: Your Command Line Tools are too outdated.
Update them from Software Update in System Preferences or run:
  softwareupdate --all --install --force

If that doesn't show you any updates, run:
  sudo rm -rf /Library/Developer/CommandLineTools
  sudo xcode-select --install

Alternatively, manually download them from:
  https://developer.apple.com/download/all/.
You should download the Command Line Tools for Xcode 13.4.

I'm running Xcode 14.0 on MacOS 12.5.1 which MacOS is telling me does not require updating. I haven't gone through the troubleshooting steps suggested by brew install after softwareupdate yet as the problem seems to be that my Xcode is newer than expected rather than needing updating? Happy to go through the troubleshooting steps suggested if you think that will help.

2022/07/21 10:07:03 ERROR: export chats: write/close chat file "20220721bagouppdf/Apple/iMessage;-;urn:biz:b15ed773-9eed-11e7-baa2-7b88b04daa8e.pdf": write out PDF: Loading pages (1/6)
Error: Failed to load file:///Users/tag/Library/Messages/Attachments/35/05/491787E2-8940-4DA6-9159-7908BDB5339B/og-default?wid=1200&hei=630&fmt=jpeg&qlt=95&.v=1525370171638.jpeg, with network status code 203 and http status code 0 - Error opening /Users/tag/Library/Messages/Attachments/35/05/491787E2-8940-4DA6-9159-7908BDB5339B/og-default: No such file or directory

This is probably related to wkhtmltopdf/wkhtmltopdf#4406

For example, --preserve-paths without --copy-attachments doesn't make sense.

My conversion stalled in the middle with this error:

decode HEIC image: error reading "ftyp" box: got box type "$\x14Ex" instead

Originally posted by @richardsprague in #16 (comment)

https://pkg.go.dev/mod/github.com/jessevdk/[email protected]

• Offer parameters for:
  • DB file location
  • contact file location
  • backup output location
  • self handle
• Output usage from README

It seems like typedstream-decode is adding some newlines.

Hello !
I'm encountering a fatal error when I run bagoup :

~# bagoup --db-path=chat.db 2022/07/15 12:03:41 0 messages successfully exported to folder "messages-export" 2022/07/15 12:03:41 ERROR: export chats: get attachment paths: get path for attachment 25 to message 414: read data for attachment ID 25: sql: Scan error on column index 0, name "filename": converting NULL to string is unsupported

This happens wether I specify the chat.db path (previously copied in another location) or when I just use the default path (while granting full disk access to my terminal emulator)

I think that some attachments are broken ?

https://github.com/tagatac/bagoup/community

At least as recently as Mac OS 10.11:

2022/07/24 05:26:40 ERROR: export chats: get message IDs for chat ID 75: query chat_message_join table for chat ID 75: no such column: message_date

https://github.com/golang-standards/project-layout

Instead of issuing an HTTP request for each one

Several possible routes here:

• use my own fork of goheif
• shell out to use magick
• compile libheif into bagoup
• reimplement relevant portions of libheif in bagoup

Three main benefits:

• improve stability
• get rid of constant-conversion warnings
• expose an interface for easy testing

magick appears to be stable at the cost of adding another system dependency.

Some more triage is needed. At least sharing a map location seems to be one cause for blank messages.

With embedded images.

Describe the bug
When bagoup is run with -a flag, and when a chat history contains more than one attachment files with the same file name, it will rename the files with the same name except the first one, in order to copy the into the same subdirectory. However the reference to that file name in the text transcript is not update.

For example, when you see files being renames:

2022/09/23 14:05:35 WARN: copy "/Users/***/Library/Messages/Attachments/da/10/at_1_26E0DB19-890A-4C9E-874A-397735C7B9E5/Image-1.jpeg" to "/Users/***/Downloads/messages_export1/John Smith/attachments" - "Image-1.jpeg" already exists; using "Image-1-2.jpeg" instead

You are still seen multiple references to in the transcript:

[2020-11-05 17:59:13] Me: I really hope *****
<attached: Image-1.jpeg>
<attached: Image-1.jpeg>
[2020-11-05 17:59:27] John: I think it’s coming 

To Reproduce
Just run the command with -a, and if you have duplicated file names in the same conversation, this behavior will appear.

Expected behavior
The reference in the transcript text should also be updated when the name of the file referred to is changed

Environment (please complete the following information):

• OS: macOS 12.6
• Version 2.1.2 Darwin/arm64

Describe the bug
When running the program with -a flag to copy attachments, saw Error that says the directory already exists

To Reproduce
Steps to reproduce the behavior:

1. Running in terminal bagoup -o /Users/***/Downloads/messages_export -c /Users/***/Downloads/all_contacts.vcf --separate-chats -a
2. See error:

2022/09/11 20:24:07 BAGOUP RESULTS:
bagoup version: 2.1.1 Darwin/arm64
Export folder: "/Users/***/Downloads/messages_export"
Export files written: 14
Chats exported: 14
Messages exported: 25
Attachments copied: 0
Attachments referenced or embedded: 0
Attachments embedded: 0
Attachments missing (see warnings above): 0
HEIC conversions completed: 0
HEIC conversions failed (see warnings above): 0
Time elapsed: 2.145910917s
2022/09/11 20:24:07 ERROR: export chats: create directory "/Users/***/Downloads/messages_export/+1**********/attachments": mkdir /Users/***/Downloads/messages_export/+1**********/attachments: file exists

Expected behavior
If the existence of a directory prevents mkdir, it should be skipped

Screenshots
If applicable, add screenshots to help explain your problem.

Environment (please complete the following information):

• OS: macOS 12.5.1
• Version bagoup version 2.1.1 Darwin/arm64

At the moment, they are just using actual syscalls:

It's great to see an active script to export iMessages on macOS. Thank you!

It would be great if it could export attachments as well. Not necessarily image as described in #5, but also videos, PDFs, and arbitrary files.

The message file should reference them somehow. For comparison, a WhatsApp export references them in the _chat.txt message file like this:

[01.02.21, 15:16:17] John: Hello World.
[01.02.21, 15:16:18] John: ‎This message was deleted.
[01.02.21, 15:16:19] John: ‎<attached: 00000121-AUDIO-2021-02-01-15-16-19.opus>
[01.02.21, 15:16:20] John: ‎<attached: 00000124-PHOTO-2021-02-01-15-16-20.jpg>

Describe the bug
__kIMOneTimeCodeAttributeName is occasionally being included in the message.

To Reproduce
Run bagoup.

Expected behavior

[2023-12-17 21:27:07] Google Codes: G-123456 is your Google verification code.

Actual behavior

[2023-12-17 21:27:07] Google Codes: G-123456    "__kIMOneTimeCodeAttributeName" =     {
        code = 123456;
        displayCode = "G-123456";
    };
} is your Google verification code.

Environment (please complete the following information):

• OS: Mac OS 14.5
• Version: v2.4.6

Is your feature request related to a problem? Please describe.
Ability to export conversation from just one contact rather than entire iMessages

Describe the solution you'd like
add a flag:
--limit-to-contact="+15551124456"

Describe alternatives you've considered
Right now I have to export everything and then delete all but the contact I want. Takes a long time

Additional context
n/a

Is your feature request related to a problem? Please describe.
Currently you cannot point output path to an existing directory, and bagoup insist on creating an empty directory and backup everything.

Describe the solution you'd like
Would it be possible to backup incrementally, i.e. find new messages sent and received after the last backup, and add these new ones to an existing backup directory?

Additional context
Just a thought. This may be a bit difficult to implement, it will need either 1) keep an internal database on which messages it has already backed up, or 2) scan the backup files to determine which ones are old, prior to each backup.

To allow use of MockPDFGenerator in testing.

Preferably also break OS.NewOutFile into OS.NewTextOutFile and OS.NewPDFOutFile instead of passing isPDF and conditionally making one or the other.

For 2 reasons:

1. performance
2. to remove a dependency

Follow-up to #42

PDF creation is much slower than plaintext creation.

The concern is that querying for the attributedBody column may not be valid if that column does not exist.

The -c option is great, as it bundles iMessages and SMS for the same contact in one folder.

Unfortunately, iMessage allows to change the sender address. The contact may use their phone number, or may use any one of their email addresses associated with the Apple ID. This leads to multiple iMessage;-;SENDER_ADDRESS.txt files for the same contact, if they ever changed their sender address.

It would be great if there was an option to combine those multiple iMessage;-;SENDER_ADDRESS.txt files for the same contact.