arachsys / mailflow Goto Github PK

MailFlow
========

MailFlow is a plugin for Apple Mail on macOS 10.12 to 12.x, fixing it to
emit RFC2646 format=flowed plain text messages and taming its enthusiasm for
quoted-printable transfer encoding. Quoting and attribution when replying is
also improved, and a long-standing bug in the display of plain text messages
is fixed.

Mail was once quite a good 'net citizen. It could be configured to generate
plain text email, and would do so with the text neatly wrapped at 76
columns. In addition, it used the RFC2646 format=flowed extension to
indicate that these wrapped paragraphs could be reflowed. Thus mailing list
archives and traditional mail clients saw readable 80-column text without
noisy encoding, but more sophisticated readers could re-fill the paragraphs
to suit the display width.

Unfortunately, Mail is poorly maintained and has declined in quality over
recent years, with many bugs introduced and incompetent design choices made.
Current releases have ditched format=flowed for plain text parts, instead
opting to emit each paragraph as a single long line. If this is longer than
77 characters, the text is mangled with a quoted-printable transfer
encoding. The resulting email leaves a very visible mess in mailing list
archives on the web, as well as drawing understandable ire from recipients
with traditional unix mail clients.

Fortunately, even if Mac users find themselves attracted to Apple Mail for
its convenient reading interface and good platform integration, all is not
lost. Mail has been furnished with a plugin interface, albeit an
undocumented one, and MailFlow hooks into this to improve the situation.


Compatibility
-------------

MailFlow is currently compatible with Apple Mail 10.0 to 15.x included in
macOS 10.12 (Sierra) to 12.x (Monterey).

It does not yet support Mail 16.x included in macOS 13.x (Ventura). This
makes major changes to the message editor, requiring non-trivial fixes to
the way MailFlow hooks into the composer. In particular, it is no longer
obvious how to access the message DOM tree, which MailFlow relies on to
correct attribution lines, detect plain text mode and extend indentation.
The format=flowed transformation of outbound messages still works fine.

I do not use macOS apart from maintaining MailFlow and MailWrap, and only
have occasional access to machines running the most recent versions. As far
as I can tell, the class-dump utility which I relied on to develop MailFlow
is also broken by the latest OS release. I would welcome input from anyone
with better insight into the Mail.app changes, or just with more patience
to reverse-engineer its internals once again. Please do get in touch with
Chris Webb <[email protected]>.


Installation
------------

To install, clone the git repository or unpack the source tar.gz, change to
the source directory and run 'python install.py' or 'python3 install.py'.
Your terminal will need access to ~/Library/Mail/Bundles/ which you can
grant in the Privacy and Security tab of System Settings.

The installer and plugin work with both the system Python 2.7 and more
recent Python 3.x, but py2app and pyobjc are required. The installer will
prompt you to install these with pip/pip3 if they can't be found.

Plugin bundles contain a list of UUIDs identifying versions of Mail with
which they are compatible. The install.py script extracts the correct UUID
from the installed version of Mail, generates a MailFlow.bundle to match,
and installs it in ~/Library/Mail/Bundles/. You will need to quit and
relaunch Mail for the plugin to be registered.

On macOS 10.14 and later, the plugin must be explicitly enabled in Mail
Preferences or Settings. Choose 'Manage Plug-ins...' from the General tab,
tick MailFlow.mailbundle, then choose 'Apply and Restart Mail'.

On macOS 11.0 and later, the plugin also needs to be ad-hoc signed and
authorised before it will work. The installer will run

  codesign -f -s - ~/Library/Mail/Bundles/MailFlow.mailbundle
  spctl --add --label MailFlow ~/Library/Mail/Bundles/MailFlow.mailbundle
  spctl --enable --label MailFlow

for you, but spctl will require your password to allow the changes. Many
thanks to A. Wilcox (awilfox) for providing these signing instructions on
their Cat Fox Life blog.

If the sandbox doesn't allow Mail.app to read the directory where your
Python is installed, you may need to change 'semi_standalone' to False in
the setup options in install.py. This issue doesn't affect /usr/bin/python
or /usr/bin/python3, but has been reported with Homebrew Python running
from /opt/homebrew/bin/python3 and /opt/homebrew/Cellar/[email protected]/.

If you use the system Python 2.7 on macOS 11.0 or later, you will need to
set SYSTEM_VERSION_COMPAT=0 in the environment when running install.py:

  SYSTEM_VERSION_COMPAT=0 /usr/bin/python install.py

Without this, a horrible Apple hack will cause the installer to detect the
OS version incorrectly as 10.16. It aborts with a warning if that happens.

Sometimes when Mail is updated, its compatibility UUID changes. Mail will
then disable plugins, moving them from 'Bundles/' to 'Bundles (Disabled)/'.
The user is notified when this happens, and it is sufficient to simply run
the install.py script again. The old disabled bundle will be cleared away,
and a new one built and installed to match the new version of Mail.


Features
--------

Most of MailFlow's functionality should be transparent to a Mail user unless
the raw source of sent messages is examined. Its primary feature is to break
paragraphs into reflowable lines in RFC2646 format=flowed format and disable
quoted-printable transfer encoding for plain text email.

In outbound plain text messages, lines are broken at word boundaries to wrap
the text to 76 columns wide, and a trailing space is added wherever a line
has been broken. Where a line to be broken is quoted, the continuation line
is quoted to the same level. Finally, to signal the message has been flowed,
the parameters format=flowed and delsp=yes are added to the text/plain
content-type. The result is a visually clean plain text message, correctly
wrapped for 80 column displays, but clients aware of the format=flowed
extension can spot the trailing spaces, reassemble paragraphs and reflow
them to fit displays of differing widths. This is especially useful for
mobile devices.

Unlike Apple's original format=flowed implementation, MailFlow will never
break a line within a word, even if that word is longer than 76 characters.
This was typically an issue with long URLs pasted into messages: breaking
these to fit an 80-column display usually causes more problems than it
solves, and some format=flowed aware clients do not fully support the
delsp=yes modifier needed to reassemble them. However, as with Apple's
original implementation, MailFlow implements space stuffing for unquoted
lines beginning with ' ' or 'From '.

Lines indented with whitespace are not flowed, whether or not they are
quoted, and any trailing whitespace is removed to avoid clients from trying
to reflow them. This provides a convenient way to include non-reflowable
content such as quotes, code samples, aligned columns or ASCII art whilst
composing a plain text message.

For convenience when indenting text blocks, the built-in Increase/Decrease
Indentation operations are extended to work on plain text messages. These
will insert or remove two spaces at the start of the current line or all
lines overlapping the current selection.

Flowed format is not appropriate for some messages, such as those containing
inline patches. To disable the use of flowed text for an individual message,
hold down the Option key when clicking on the Send button in the toolbar, or
when selecting Send from the Message menu. MailFlow will still restrict
unnecessary use of quoted-printable even when format=flowed is disabled.

By default, Mail uses a quoted-printable transfer encoding for text/plain
parts whenever they contain non-ASCII characters or a line longer than 77
characters. MailFlow relaxes this behaviour for text/plain, using a 7-bit
transfer encoding for plain ASCII and an 8-bit transfer encoding otherwise,
provided the lines do not exceed the 998 column limit imposed by SMTP. Note
the outbound mail server must support the 8BITMIME ESMTP extension
(RFC1653/RFC6152) but all modern SMTP servers are fine with this.

MailFlow will trim the excessively verbose attribution line Mail inserts
when composing a reply, i.e.

  On 8 Apr 2014, at 10:08:34, Chris Webb <[email protected]> wrote:

becomes

  Chris Webb <[email protected]> wrote:

Since version 8.0, Mail has a bug which causes the attribution line to be
quoted as if it were part of the original message. MailFlow will also fix
this whilst trimming the attribution line.

MailFlow makes it easier to forward messages as proper MIME attachments
instead of quoting them inline. Mail has always supported MIME forward but
relegated it to the Message menu without a shortcut or toolbar button,
instead encouraging ugly inline forward. MailFlow reconfigures the standard
forward buttons, keyboard shortcut and menu item to use MIME.

To override this and forward a message inline, hold down the Option key when
clicking on the Forward button in the toolbar or when selecting Forward from
the Message menu.

Finally, MailFlow addresses a very long-standing bug in Mail, which drops
one leading space from every indented line of a plain text message in the
message viewer. When converting the text/plain part to HTML for display in a
WebView, all but the first leading space is rendered as '&nbsp;' so, for
example, a line '  foo' becomes '<BR> &nbsp;foo'. However, WebKit will then
display this as ' foo' not '  foo'.

To work around this bug, MailFlow translates a space following a '<BR>' (or
at the start of the message) into '&nbsp;' before a plain text part is
rendered in a WebView.


Configuration
-------------

MailFlow reads two preferences from the com.apple.mail domain. These can be
set at the command line with the macOS defaults command:

  defaults write com.apple.mail MailFlow -dict-add FixAttribution -bool false
  defaults write com.apple.mail MailFlow -dict-add FixAttribution -bool true
    - configure MailFlow to strip the verbose date and time information
      from the attribution line when composing a reply. The default is on.

  defaults write com.apple.mail MailFlow -dict-add FlowWidth -int NN
    - break lines at a width of NN characters when flowing text. The
      default is 76. Set to 0 to disable format=flowed but still restrict
      unnecessary use of quoted-printable.


pbmbox
------

For command-line users, the MailFlow distribution also includes a small
utility, pbmbox. When messages are selected and copied in Mail, they are
added to the clipboard as RFC822MessageDatasPboardType objects. pbmbox
decodes these objects and emits the messages in unix mbox format on stdout.

To install, copy it to a directory in your PATH and make it executable:

  sudo install -m 0755 pbmbox.py /usr/local/bin/pbmbox

A typical use is importing a patch series from email into a git repository.
Select the messages in Mail, copy them with Command-C, and then run

  pbmbox | git am

within the repository.

By default, the mboxrd format is used: lines beginning with />*From / are
quoted with one additional leading '>'. This encoding avoids corruption of
messages and is always reversible. If the -n or --no-quote-from option is
given, pbmbox will not attempt to quote 'From  ' lines. This is sometimes
useful for simple command-line handling of a single message, where no
ambiguity can result from an unquoted 'From '.


Copying
-------

This software was written by Chris Webb <[email protected]> and is
distributed as Free Software under the terms of the MIT license in COPYING.