Forward not working for me about mailu

[kaiyou]

First thank you for the SES idea, i just opened #27 for this purpose.

Regarding your forwarding issue, I remember we had an issue with the forwarding sieve script a month ago, did you pull the latest IMAP image?

If you did, could you please grab the logs from both your SMTP and IMAP container around the time you send the test email that does not seem to be forwarded? Forwarding is actually handled by the IMAP container using sieve scripts, then the SMTP container takes care of sending the outgoing message (as opposed to aliases, that are handled directly by the SMTP server).

In you IMAP log, you should see something in the lines of:

Jul 24 09:14:48 lmtp([email protected]): Info: xyz: sieve: msgid=<[email protected]>: forwarded to <[email protected]>
Jul 24 09:14:48 lmtp([email protected]): Info: xyz: sieve: msgid=<[email protected]>: stored mail into mailbox 'INBOX'

The message is first forwarded by the sieve script then delivered locally by the LDA (the forwarding feature forwards a copy of the email, if you prefer emails not to be delivered locally, you should use an alias instead of a user).

Then on the SMTP container:

postfix/lmtp[378]: EC8CEB19F: to=<[email protected]>, relay=imap[172.18.0.10]:2525, delay=3.7, delays=1.1/0/0.01/2.6, dsn=2.0.0, status=sent (250 2.0.0 <[email protected]> xyz Saved)
postfix/qmgr[28]: EC8CEB19F: removed
postfix/smtp[379]: 1ECDAB1A0: to=<[email protected]>, relay=gmail-smtp-in.l.google.com[74.125.133.26]:25, delay=30, delays=2.1/0.01/28/0.27, dsn=2.0.0, status=sent (250 2.0.0 OK 1469351716 sw9si9853072wjb.19 - gsmtp)
postfix/qmgr[28]: 1ECDAB1A0: removed

First the message is delivered locally (forwarded to Dovecot using LMTP), then the forwarded message queued by the IMAP container is sent to Gmail.

[jkarlosb]

I have noticed now that sometimes, with normal emails (without forwarding), I get this error with the relayhost, then the email isn't sent:

postfix/qmgr[594]: BB922291: from=<[email protected]>, size=611, nrcpt=1 (queue active)
postfix/smtp[606]: BB922291: to=<[email protected]>, relay=none, delay=1120, delays=1090/0.01/30/0, dsn=4.4.3, status=deferred (Name service error for name=email-smtp.eu-west-1.amazonaws.com type=AAAA: Malformed or unexpected name server reply)

I think that this is due to IPv6 problem in DNS (AAAA registry), but some time later I get this log in postfix:

postfix/qmgr[594]: BB922291: from=<[email protected]>, size=611, nrcpt=1 (queue active)
postfix/smtp[620]: BB922291: to=<[email protected]>, relay=email-smtp.eu-west-1.amazonaws.com[52.51.170.35]:25, delay=2295, delays=2290/0.01/5.1/0.13, dsn=2.0.0, status=sent (250 Ok 010201561e4dcb65-22b2a3ec-faeb-490e-a091-1a26bb7d7e90-000000)
postfix/qmgr[594]: BB922291: removed

The email is sent correctly throught Amazon SES relayhost.

In my main.cnf file in postfix containter, I have:

relayhost = [email-smtp.eu-west-1.amazonaws.com]:25

On the another hand, I get correct imap/dovecot logs when I send email for forward (forwarding emails with copy in the server, i.e. user, not alias):

Jul 24 19:54:06 lmtp([email protected]): Info: pnz8J9oclVe0BAAAWxxAHA: sieve: msgid=<CABbJG=7EfsX2s4BLX02ZtVE6QVH5=[e-mail]>: forwarded to <[email protected]>
Jul 24 19:54:06 lmtp([email protected]): Info: pnz8J9oclVe0BAAAWxxAHA: sieve: msgid=<CABbJG=7EfsX2s4BLX02ZtVE6QVH5=[e-mail]>: stored mail into mailbox 'INBOX'

But I encounter the same error in postifix logs:

postfix/qmgr[594]: AF548292: from=<[email protected]>, size=3010, nrcpt=1 (queue active)
postfix/smtpd[635]: disconnect from freeposteio_imap_1.freeposteio_default[172.18.0.7] ehlo=1 mail=1 rcpt=1 data=1 quit=1 commands=5
postfix/lmtp[640]: 9E98F291: to=<[email protected]>, relay=imap[172.18.0.7]:2525, delay=9.1, delays=5/0/0.01/4.1, dsn=2.0.0, status=sent (250 2.0.0 <[email protected]> pnz8J9oclVe0BAAAWxxAHA Saved)
postfix/qmgr[594]: 9E98F291: removed
postfix/smtp[641]: AF548292: to=<[email protected]>, relay=none, delay=34, delays=4.1/0.01/30/0, dsn=4.4.3, status=deferred (Name service error for name=email-smtp.eu-west-1.amazonaws.com type=AAAA: Malformed or unexpected name server reply)

In this case I never received the forward email.

If I run nslookup from postfix container I get:

# nslookup email-smtp.eu-west-1.amazonaws.com
nslookup: can't resolve '(null)': Name does not resolve

Name:      email-smtp.eu-west-1.amazonaws.com
Address 1: 52.49.159.188
Address 2: 52.51.170.35
Address 3: 54.229.133.103

If I run it from server:

# nslookup email-smtp.eu-west-1.amazonaws.com
Server:     172.31.0.2
Address:    172.31.0.2#53

Non-authoritative answer:
email-smtp.eu-west-1.amazonaws.com  canonical name = ses-smtp-eu-west-1-prod-345515633.eu-west-1.elb.amazonaws.com.
Name:   ses-smtp-eu-west-1-prod-345515633.eu-west-1.elb.amazonaws.com
Address: 52.51.170.35
Name:   ses-smtp-eu-west-1-prod-345515633.eu-west-1.elb.amazonaws.com
Address: 54.229.133.103
Name:   ses-smtp-eu-west-1-prod-345515633.eu-west-1.elb.amazonaws.com
Address: 52.49.159.188

I tried install drill (a dig-like for Alpine) but I get this error:

# apk add --update --no-cache drill
fetch http://dl-cdn.alpinelinux.org/alpine/v3.4/main/x86_64/APKINDEX.tar.gz
fetch http://dl-cdn.alpinelinux.org/alpine/v3.4/main/x86_64/APKINDEX.tar.gz
fetch http://dl-cdn.alpinelinux.org/alpine/v3.4/community/x86_64/APKINDEX.tar.gz
fetch http://dl-cdn.alpinelinux.org/alpine/v3.4/community/x86_64/APKINDEX.tar.gz
(1/2) Installing ldns (1.6.17-r3)
(2/2) Installing drill (1.6.17-r3)
Executing busybox-1.24.2-r9.trigger
OK: 21 MiB in 31 packages

# drill email-smtp.eu-west-1.amazonaws.com
Error: error sending query: Could not send or receive, because of network error

Ok, if we can solve this problem, I could help you to implement it in the project and complete the #27 feature, if you want ;)

Thanks for all.

[kaiyou]

Sorry about the delay. What you are describing is very similar to an issue that I have recently been experiencing on some Docker 1.11 servers with containers performing a lot of DNS queries (including my Postfix servers). It is not related to Freeposte but to Docker itself on containers that tend to perform many DNS queries.

Docker 1.11 introduced per-container UDP socket caching for DNS queries. There is a bug in the initial implementation, where a late reply is not flushed from the DNS socket, but then forwarded as a reply to the next request, and subsequent replies are shifted. When the wrong reply is forwarded, it is ignored by the container DNS stack because the reply id does not match, leading to a timeout. More details on the issue thread: moby/moby#22185 .

The issue was fixed two days ago and the patch is available in a Docker RC. Until the patch is introduced into stable, the suggested workaround consists in enabling the use-vc DNS option on the host, then restarting the containers. This will switch to TCP sockets for DNS resolution and the quirky UDP caching feature will not mess around. If your host does not perform huge amounts of queries, the TCP overhead should not be too much of an issue, simply add this line to your /etc/resolv.conf:

options use-vc

Then restart your containers.

[jkarlosb]

Thank you for your response. I added options use-vc in resolv.conf of my host and after I restarted my containers, but this workaround not working for me. I haven't many DNS queries, I runned this docker compose in a new AWS t2.micro instance for testing.

Now I understand better the issue. There are two different problems. These problems only happen with Amazon relayhost configured. (I configured this way)

First problem. When I send an email from a verified SES domain, this has a "random" behavior of DNS resolve, sometimes through this error:

postfix/smtp[641]: AF548292: to=<[email protected]>, relay=none, delay=34, delays=4.1/0.01/30/0, dsn=4.4.3, status=deferred (Name service error for name=email-smtp.eu-west-1.amazonaws.com type=AAAA: Malformed or unexpected name server reply)

Sometimes this another error:

postfix/smtp[295]: DFF04150: to=<[email protected]>, relay=none, delay=102, delays=4/0.01/98/0, dsn=4.4.1, status=deferred (connect to email-smtp.eu-west-1.amazonaws.com[52.51.170.35]:25: Operation timed out)

Sometimes it works fine (smtp response code 250):

postfix/smtp[137]: 34EAD138: to=<[email protected]>, relay=email-smtp.eu-west-1.amazonaws.com[52.51.170.35]:25, delay=561, delays=554/0.01/7.6/0.19, dsn=2.0.0, status=sent (250 Ok 010201563173d191-409b5b42-eaa4-4dc6-8c80-fb3646b0723d-000000)

I was trying to resolve this error with this topic of serverfault, changing the protocol to IPv4 with inet_protocols = ipv4 in /etc/postfix/main.cf, but I get the next result :

postfix/smtp[1022]: 4733E292: to=<[email protected]>, relay=none, delay=455, delays=430/0.02/25/0, dsn=4.4.3, status=deferred (Name service error for name=email-smtp.eu-west-1.amazonaws.com type=A: Malformed or unexpected name server reply)

I was trying with another postfix server in a docker container and this error don't happen. Therefor I don't know if this problem is due to a docker bug or a another issue related a DNS configuration.

Second problem (initial problem). I want to send and receive emails with my Gmail client. I can send through Postfix smtp container, but I want receive through Postfix forward feature or IMAP (Gmail don't let to act itself how a IMAP client). Postfix forward email if sender account is from my domain, i.e. it is verified, but I can't forward emails from external accounts with SES (either keeping copy of email in server, alias or virtual), this happens when I try:

postfix/smtp[294]: CFCF1134: to=<[email protected]>, relay=email-smtp.eu-west-1.amazonaws.com[54.229.133.103]:25, delay=13, delays=4.1/0.01/9.2/0.16, dsn=5.0.0, status=bounced (host email-smtp.eu-west-1.amazonaws.com[54.229.133.103] said: 554 Message rejected: Email address is not verified. The following identities failed the check in region EU-WEST-1: [email protected], =?UTF-8?Q?MYNAME_LASTNAME?= <[email protected]> (in reply to end of DATA command))
postfix/cleanup[292]: 3A021151: message-id=<[email protected]>
postfix/qmgr[26]: 3A021151: from=<>, size=5953, nrcpt=1 (queue active)
postfix/bounce[296]: CFCF1134: sender non-delivery notification: 3A021151
postfix/qmgr[26]: CFCF1134: removed
postfix/scache[297]: warning: smtputf8_enable is true, but EAI support is not compiled in
postfix/smtp[294]: 3A021151: to=<[email protected]>, relay=email-smtp.eu-west-1.amazonaws.com[54.229.133.103]:25, delay=7.6, delays=0/0/7.6/0, dsn=5.0.0, status=bounced (host email-smtp.eu-west-1.amazonaws.com[54.229.133.103] said: 501 Invalid MAIL FROM address provided (in reply to MAIL FROM command))

[Amazon smtp response codes]

These days I was reading and It is due to Amazon SES restrictions. Only verified domains can send emails through SES, and when I use a relay feature, really I am send from external domain (e.g. gmail.com). I have the same problem that there in serverfault here and here

I don´t know which is the best solution to implement:

• To use sender_canonical_maps to envelope sender in Postfix how as suggested in serverfault links.
• To use a fetchmail daemon for to connect by IMAP and to forward the emails after.
• To use the receive email service of Amazon SES through AWS Lambda. This is a project in GitHub.
• I am thinking now... I don't know if is possible to use transport, with transport_maps in the main.cf for to split sender domains in two types: verified domains, to relay to Amazon SES; and rest of domains, to relay to local smtp (no relayhost)...
• Any other ideas?

Do you think about this?

[jkarlosb]

Ok, I solved the second problem. I was reading about transport maps and I saw that transport maps only can relay to others hosts based in recipient destination addresses or domains... so that wasn't what I wanted.

Then I found the next entry for main.cf: sender_dependent_relayhost_maps, this entry is how transport_maps but based on sender. That is the solution. I tested it and it works.

[kaiyou]

Very nice about the second problem. I will think about embedding common settings or the ability to set specific postfix options from the main freeposte.env file, so that you do not lose your settings when upgrading or recreating your containers.

Regarding the first problem, I set up a lab server talking to SES and I am unable to reproduce for now. Could you share your Docker version (docker version output)? Did you try your Freeposte configuration on a different server (mostly different Docker version)?

Finally, I still cannot understand your use case exactly. You are trying to send emails with your gmail.com identity through Postfix then SES? If so, first SES won't work as you already discovered, but due to DMARC and you being unable to DKIM-sign as gmail.com, your messages will end up in junk folders if not dropped by the recipient. For receiving emails, I believe that the Web GMail client supports IMAP, and so does the Inbox application.

[jkarlosb]

Yes, the ability to set specific postfix options from the main freeposte.env file I think that is a great idea, even including a config folder with dovecot config files, postfix config files (main.cf and eventual lookup tables file for some entries) , etc. and when recreating the containers, this entries of these config files override the entries of main configuration. If this files are empty then we do nothing.

My docker server and client are in the same host (ubuntu 14.04) and they have the same version:

$ docker version
Client:
 Version:      1.11.2
 API version:  1.23
 Go version:   go1.5.4
 Git commit:   b9f10c9
 Built:        Wed Jun  1 21:47:50 2016
 OS/Arch:      linux/amd64

Server:
 Version:      1.11.2
 API version:  1.23
 Go version:   go1.5.4
 Git commit:   b9f10c9
 Built:        Wed Jun  1 21:47:50 2016
 OS/Arch:      linux/amd64

But, I have good news, I rebuilt the containers and this problem disappeared. It may to be that I changed some configuration in postfix (searching a forward solution) but now it works perfectly.

Ok, I explain better my case. I am trying to send emails with my web gmail client as a MUA through Postfix (configured to relay to Amazon SES - MTA-to-MTA communication). That now works fine with this configuration, Amazon SES as a nexthop/gateway in relayhost entry of main.cf. Any email sent from @mydomain.com is relayed through SES perfectly.

I want to use web Gmail client as MUA also for retrieve my mails through IMAP (I don't want to use another MUA, as Roundcube webmail) but this is not possible because Gmail only lets to check mail from other accounts using POP3 (as I think to have understood). Then the solution is forwarding all inbound mail to my @gmail.com account.

Amazon SES has a restrictions, only verified domains are allowed to relay mail. So when arriving a email from a external domain (e.g. @yahoo.com) to @mydomain.com, Postfix delivery the message to the LDA (dovecot) and It message is saved to inbox. Until then everything right. Next the forwarded message queued by the LDA container is sent through MTA Postfix to my @gmail.com account (relayed through Amazon SES). And here is the problem. SES don't let delivery mails from unverified domains as @yahoo.com, so reject this message with 554 code:

postfix/smtp[294]: CFCF1134: to=<[email protected]>, relay=email-smtp.eu-west-1.amazonaws.com[54.229.133.103]:25, delay=13, delays=4.1/0.01/9.2/0.16, dsn=5.0.0, status=bounced (host email-smtp.eu-west-1.amazonaws.com[54.229.133.103] said: 554 Message rejected: Email address is not verified. The following identities failed the check in region EU-WEST-1: [email protected], =?UTF-8?Q?MYNAME_LASTNAME?= <[email protected]> (in reply to end of DATA command))

I solved this problem with sender_dependent_relayhost_maps entry for using a Amazon SES as relayhost only for delivery internal domain senders, but postfix directly for delivery external domain senders. This way forward works, all works perfectly.

I hope I explained better.

[kaiyou]

Okay, thank you very much for the details. I do understand the use case now. Indeed I see no other solution than forwarding to your GMail account (the mobile application has an embedded IMAP client but I don't think the Web client does).

Regarding the custom configuration files and/or environment variables, I'll paste your ideas to the other issue. Thank you.

[jkarlosb]

Thank you very much for all @kaiyou, I will follow this project very close and I will help if I can.

[jkarlosb]

I think I found the solution to my first problem, I think that it was due to a bad configuration of main.cf in smtp_tls_CAfile parameter. If anyone can help.

[kaiyou]

Could you elaborate on the problem? I thought forwarding was working fine again? Regarding the CAfile, you should be using the system CA, but I think that the default value is fine on Alpine, so you should not need to override it if you plan on using standard CA certificates.

[jkarlosb]

Yes, forwarding works fine. No problem with this. I just wanted to know what happened with the first problem and why, to finally close the issue, because I wanted to be sure that this problem doesn't occur anymore.

I tested and I noticed that the problem was that I had followed step by step Amazon instructions to configure integration Postfix with SES:

9. Tell Postfix where to find the CA certificate (needed to verify the Amazon SES server certificate). You could use a self-signed certificate or you could use default certificates as follows:

If running on the Amazon Linux AMI:

sudo postconf -e 'smtp_tls_CAfile = /etc/ssl/certs/ca-bundle.crt'

If running on Ubuntu Linux:

sudo postconf -e 'smtp_tls_CAfile = /etc/ssl/certs/ca-certificates.crt'

How I didn't understand if the CAfile override was necessary, then I overrided this. That was the problem, now I understand this and I think like you, that the default value on Alpine is perfect and it works fine.

So all good.