poolporg / filter-rspamd

OpenSMTPD filter integration for the Rspamd daemon

Hi there 👋

• 🔭 part-time IT Architect @ Veepee
• ⭐️ freelance architect/developer/consultant or opensource developer on my free time: I'm available for freelance projects!
• 📚 I occasionally teach or give private talks on various topics
• 🚀 Personal blog at poolp.org

I'm @poolpOrg

filter-rspamd's Issues

No longer available as an OpenBSD package?

Been using OpenSMTPD for a few years now and finally getting around to putting some proper spam filtering in place, however this filter is no longer available as a package on OpenBSD:

# pkg_info -Q opensmtpd
libopensmtpd-0.7
opensmtpd-extras-6.7.1v0
opensmtpd-extras-mysql-6.7.1p0v0
opensmtpd-extras-pgsql-6.7.1p0v0
opensmtpd-extras-python-6.7.1v0
opensmtpd-extras-redis-6.7.1v0
opensmtpd-filter-admdscrub-0.1
opensmtpd-filter-dkimsign-0.5p2
opensmtpd-filter-dnsbl-0.3
p5-OpenSMTPd-Filter-0.0.2

This is on -current: OpenBSD 7.3-current (GENERIC) #1193: Sun Jun 18 09:32:39 MDT 2023

Provide information whether the user is authenticated

So I am running OpenSMTPD 6.6.0p0 from Git since yesterday, everything seems to work fine except DKIM signing for outgoing mail.
Upgraded from some ooooold version that was providing ooooold Rspamd and DKIM signer filters 😆
Now this filter-rspamd is expected to do both I presume.

However, I don't see a clear way to tell Rspamd that it should sign my outgoing mail, quoting https://rspamd.com/doc/modules/dkim_signing.html

To be eligible for signing, a mail must be received from an authenticated user OR a reserved (local) IP address OR an address in the sign_networks map (if defined)

From the description above only "authenticated user" flag is something that is reliable for me. I cannot do any assumptions based on IP, because IP reported to Rspamd is my client IP which depending where I am can be anything. Also I don't want to have Spam check on my outgoing mail, because it does not make sense.

The Rspamd Protocol provides User field for that purpose, but I can see in filter-rspamd code that this field is not supplied when SMTP client was authenticated in OpenSMTPD. Also it seems that OpenSMTPD does not provide this information to their filters (or maybe I am wrong?)

Or maybe there is another way to distinguish outgoing and incoming mail? @poolpOrg How do you do it? 😉

Here's an anonymized excerpt from my config:

filter "rspamd" proc-exec "/usr/local/bin/filter-rspamd"

listen on myipv4 port 25  hostname mydomain tls pki mydomain filter "rspamd"
listen on myupv6 port 25  hostname mydomain tls pki mydomain filter "rspamd"
listen on myipv4 port 587 hostname mydomain tls-require pki mydomain auth mask-src filter "rspamd"
listen on myipv6 port 587 hostname mydomain tls-require pki mydomain auth mask-src filter "rspamd"

Many thanks for this effort. I also became a mean inferior patron of your project 🙄 . Let's see where this goes.
Best regards,
Adam

is message rewrite supported?

Hi,

I'm using OpenSMTPD (6.8.0p2-3) and Rspamd (2.7-1) with filter-rspamd (0.1.7-1+b5) on Debian 11.

I want to add a text/html signature (disclaimer) to the end of every outgoing email from my domain. Rspamd has a function for this: lua_mime.add_text_footer()https://rspamd.com/doc/lua/lua_mime.html#fa3d82. An example implementation can be found at https://gist.github.com/vstakhov/3dda60a5638a6aefd454973bd687fc21.

After writing the lua code i can see the text is correctly modified with task:get_message() https://rspamd.com/doc/lua/rspamd_task.html#mcea29, however OpenSMTPD delivers the original message without the signature appended to text/html part.

The README.md https://github.com/poolpOrg/filter-rspamd#features of this project does not say anything about full message rewrite. The lua script changes both headers and content.

Is this feature supported? If no, is there a workaround to implement this functionality?

Thanks

filter-rspamd broken

For some reason, I think that with the latest snapshot version of rspamd, filter-rspamd has stopped working. In the interim, I've disabled that filter and emails are again being delivered to my inbox. I am happy to do whatever debugging you guys would like me to. Please tell me how I can help. Thanks much!

Greylist issue in combination with disabled greylisting in Rspamd.

I'm finding that if you overwrite the greylist action in Rspamd local.d/actions.conf and then disable greylisting, Rspamd will blindly return 'greylist' action instead of the actual greylist module action which is 'soft reject'.

I'm wondering if the filter should respond to 'greylist' action at all, because it's not the action that Rspamd returns from its greylist module. 'soft reject' should probably be the only action the filter accepts for a temporary reject.

Dead, blocked or misconfigured Rspamd url.

I'm in the process of moving our server to a new machine bit by bit and testing Rspamd over a network. While everything works nicely with Rspamd over network (or locally) there isn't actually any notification if Rspamd request fails and mails are passed.

Should the filter maybe log if Rspamd http requests fail?

I also thought about pinging Rspamd server at startup with rspamd/ping, but that might not be a good idea if Rspamd starts after OpenSMTPD.

Permission issue when trying to access the socket?

I’ve setup the worker-normal socket to be 660 rspamd:rspamd (0660 in octal mode for the configuration file), but also added smtpd user to the rspamd group.

However, when smtpd starts in this configuration, it fails with:

'/run/rspamd/normal.sock' err: 'dial unix /run/rspamd/normal.sock: connect: permission denied'

It works OK if the socket is 666, but I think this is not a good idea from a security standpoint.

Verbose X-Spam-Symbols.

What do you think about having another argument to the script that would enable different levels of verbosity to X-Spam-Symbols?

We could then go from:

BAYES_SPAM
BAYES_SPAM(5.10)
BAYES_SPAM(5.10)[100.00%]

Rspamd supplies all that info anyway.

impossible for milter_headers to add a custom header

rspamd has a module to insert custom headers : https://rspamd.com/doc/modules/milter_headers.html

But this filter only support a few hardcoded headers.

I think the problem is here :

filter-rspamd/filter-rspamd.go

Lines 510 to 525 in 9e74486

	/**
	* Prefix auth headers to incoming mail in proper order.
	*/
	if len(authHeaders) > 0 {
	hdrs := []string{
	"ARC-Seal",
	"ARC-Message-Signature",
	"ARC-Authentication-Results",
	"Authentication-Results"}
	for _, h := range hdrs {
	if authHeaders[h] != "" {
	writeHeader(s, token, h, authHeaders[h])
	}
	}
	}

Only the headers in hdrs will be inserted by the filter.

Here is an example of the headers that rspamd can return:

curl -s -X POST http://localhost:11333/checkv2 | jq .milter.add_headers
{
  "X-Spamd-Bar": {
    "value": "+++++++++++++++",
    "order": -1
  },
  "X-Spamd-Result": {
    "value": "default: True [15.00 / 15.00];\r\n\tCOMPLETELY_EMPTY(15.00)[]",
    "order": -1
  },
  "X-Rspamd-Action": {
    "value": "reject",
    "order": -1
  },
  "X-Spam-Status": {
    "value": "Yes, score=15.00",
    "order": -1
  },
  "Authentication-Results": {
    "value": "localhost;\r\n\tnone",
    "order": 1
  },
  "X-Rspamd-Server": {
    "value": "localhost",
    "order": -1
  },
  "X-Spam-Level": {
    "value": "***************",
    "order": -1
  }
}

I can rewrite this, but I'd like to know what is the rationale for this code:

How does the order matter ?
Does this matter for other headers (such as those given above) ?

filter-rspamd crash

rspamdQuery() ends by writing back the session to the sessions map however this should not be done this way but through a channel so we don't have concurrent writes to the map.

Bad DKIM signatures over various whitespace problems

@vstakhov

Sometimes, emails get an invalid DKIM signature. I wrote about these to [email protected], but perhaps here's a better place. It's a long thread so I'll reproduce highlights of it here:

Using a fairly typical OpenSMTPD+rspamd setup, I'm finding that emails sent that have the ^L escape in them or end with a trailing space and a newline come out with an invalid DKIM signature. Something basic like:

filter rspamd proc-exec "filter-rspamd"
listen on ... filter rspamd

Everything else is otherwise pretty default and vanilla.

Here are two emails that exhibit the issue in mbox format, so you can open these with mutt -f ./file.mbx and then use b to bounce them through opensmtpd+rspamd.

$ base64 -d > naughty-email1.mbx

RnJvbSA5YjM1Mzg5NWViZGUyZDgzZTA5MTk4YTYzZGJjYmVlMmNmNTg5OWQ0IE1vbiBTZXAgMTcg
MDA6MDA6MDAgMjAwMQpGcm9tOiAiSmFzb24gQS4gRG9uZW5mZWxkIiA8SmFzb25AengyYzQuY29t
PgpEYXRlOiBUdWUsIDI2IEp1bCAyMDIyIDAwOjIwOjIxICswMjAwClN1YmplY3Q6IHRlc3QgY29y
cnVwdGlvbiB3aXRoIGEgXkwgbWVzc2FnZQpNSU1FLVZlcnNpb246IDEuMApDb250ZW50LVR5cGU6
IHRleHQvcGxhaW47IGNoYXJzZXQ9VVRGLTgKQ29udGVudC1UcmFuc2Zlci1FbmNvZGluZzogOGJp
dAoKZGlmZiAtLWdpdCBhL0xJQ0VOU0VTIGIvTElDRU5TRVMKaW5kZXggY2QwNGZiNmU4NC4uNTMw
ODkzYjFkYyAxMDA2NDQKLS0tIGEvTElDRU5TRVMKKysrIGIvTElDRU5TRVMKQEAgLTM4OSwyNiAr
Mzg5LDMgQEAgQ29weXJpZ2h0IDIwMDEgYnkgU3RlcGhlbiBMLiBNb3NoaWVyIDxtb3NoaWVyQG5h
LW5ldC5vcm5sLmdvdj4KICBZb3Ugc2hvdWxkIGhhdmUgcmVjZWl2ZWQgYSBjb3B5IG9mIHRoZSBH
TlUgTGVzc2VyIEdlbmVyYWwgUHVibGljCiAgTGljZW5zZSBhbG9uZyB3aXRoIHRoaXMgbGlicmFy
eTsgaWYgbm90LCBzZWUKICA8aHR0cHM6Ly93d3cuZ251Lm9yZy9saWNlbnNlcy8+LiAgKi8KLQwK
ClRoZSBhYm92ZSBzaG91bGQgY2F1c2UgaXNzdWVzLgo=

$ base64 -d > naughty-email2.mbx

RnJvbSA5YjM1Mzg5NWViZGUyZDgzZTA5MTk4YTYzZGJjYmVlMmNmNTg5OWQ0IE1vbiBTZXAgMTcg
MDA6MDA6MDAgMjAwMQpEYXRlOiBNb24sIDEwIE9jdCAyMDIyIDE2OjE5OjM5ICswMjAwCkZyb206
IGphc29uQHp4MmM0LmNvbQpUbzogamFzb25AengyYzQuY29tClN1YmplY3Q6IG9oIG5vIGFub3Ro
ZXIgb25lIG9mIHRoZXNlIHRlc3RzCk1JTUUtVmVyc2lvbjogMS4wCkNvbnRlbnQtVHlwZTogdGV4
dC9wbGFpbjsgY2hhcnNldD11dGYtOApDb250ZW50LURpc3Bvc2l0aW9uOiBpbmxpbmUKCiAK

Try sending these messages through OpenSMTPD + rspamd, and you'll find that invariably the signature is wrong.

RFC: opensmtpd-filters-go library and filter port

Hi Gilles,

over the last couple of months I worked on a Go library for implementing opensmtpd filters (opensmtpd-filters-go. It's somewhat based around the code of filter-rspamd, but focused on making implementation as easy as possible by wrapping each event sent by opensmtpd into an interface.

Also, as far as I can tell, it seems that calling Print* from goroutinesnot thread-safe appears to be (if you can what I did theseere). So I added a SafePrintf|ln implementation that uses channels to ensure that messages aren't picked apart by the kernel during concurrent operations.

I also started porting filter-rspamd as a proof of concept and wrote dnsbl, trace and greylist implementations. All of this is pretty much under active development, but I'd be interested in your thoughts.

And a happy new year to you :).

Cheers,
Jonas

filter-rspamd exits when tx-mail sends 4 params instead of 3

On a server running OpenSMTPD 6.6.4p1 with rspamd 2.2, I've had a couple smtpd shutdowns happen due to the filter-rspamd plugin reporting "invalid input, shouldn't happen."

After the second or third one, I swapped all log.Fatal() calls to log.Fatalf() with the params in the logging message, like so:

@@ -153,7 +153,7 @@ func txBegin(s *session, params []string) {
 
 func txMail(s *session, params []string) {
        if len(params) != 3 {
-               log.Fatal("invalid input, shouldn't happen")
+               log.Fatalf("invalid input, shouldn't happen: %v", params)
        }
 
        if params[2] != "ok" {

Which revealed that in some cases, at least, we're getting a tx-mail message with four params instead of three:

Mar 24 13:53:35 XXX smtpd[2268]: rspamd: 2020/03/24 13:53:35 txMail: invalid input, shouldn't happen: [0797092b CleaniX [email protected] ok]

Might you have any suggestions for why this would be the case, or how I could prepare a patch to handle this?

rspamd + opensmtpd doesnt seem to block gtube tests

using the configuration below it doesnt appear that rspamd is actually doing anything when sent XJS*C4JDBQADN1.NSBN3*2IDNEN*GTUBE-STANDARD-ANTI-UBE-TEST-EMAIL*C.34X. I do see the message land in the rspamd log and it gets delivered to the maildir via opensmtpd.

However when i used rspamd with postfix milter protocol it rejected the gtube message and sent a notification to the sender.

I tested on the same system with same rspamd config. Would much rather use opensmtpd then postfix but i am not sure how to verify or configure what it will do with the spam. I believe i am missing something simple.

filter "rspamd" proc-exec "filter-rspamd"

pki xxx.com cert "/var/lib/acme/live/xxx.com/fullchain"
pki xxx.com key "/var/lib/acme/live/xxx.com/privkey"

listen on eth0 tls pki xxx.com filter "rspamd"
listen on eth0 port 587 tls-require pki xxx.com auth

action "local" maildir
action "relay" relay

match from any for domain xxx.com action "local"
match from local for any action "relay"
match auth from any for any action "relay"

for reference here is the rspamd log entry for the message. The message is not getting rejected like it does with postfix

2023-10-13 09:36:32 #21111(normal) <2a5462>; task; rspamd_task_write_log: id: <redacted>, qid: <c372279c>, ip: redacted, from: <redacted>, (default: F (no action): [1.71/15.00] [MISSING_SUBJECT(2.00){},R_SPF_ALLOW(-0.20){+mx:c;},MIME_GOOD(-0.10){text/plain;},XM_UA_NO_VERSION(0.01){},ARC_NA(0.00){},ASN(0.00){asn:63949, ipnet:redacted, country:SG;},DMARC_NA(0.00){redacted;},FROM_EQ_ENVFROM(0.00){},FROM_HAS_DN(0.00){},MID_RHS_MATCH_FROM(0.00){},MIME_TRACE(0.00){0:+;},PREVIOUSLY_DELIVERED(0.00){redacted;},RCPT_COUNT_ONE(0.00){1;},RCVD_COUNT_TWO(0.00){2;},RCVD_TLS_ALL(0.00){},RCVD_VIA_SMTP_AUTH(0.00){},R_DKIM_NA(0.00){},TO_DN_ALL(0.00){},TO_MATCH_ENVRCPT_ALL(0.00){}]), len: 907, time: 829.999ms real, 23.565ms virtual, dns req: 16, digest: <506b288a17d5df1dce7fd8c98b0dfe60>, rcpts: <redacted>, mime_rcpts: <redacted>

rspamd: failed to decode JSON response (Rspamd v3.3)

Just a heads up that I've encountered the following after updating Rspamd from v3.2 to v3.3 just now:

Oct  3 20:52:06 orc smtpd[212486]: info: OpenSMTPD 6.8.0p2 starting
Oct  3 20:52:15 orc smtpd[212491]: 03c6e38b6a06134d smtp connected address=10.0.8.2 host=<unknown>
Oct  3 20:52:15 orc smtpd[212491]: 03c6e38b6a06134d smtp tls ciphers=TLSv1.3:TLS_AES_256_GCM_SHA384:256
Oct  3 20:52:15 orc smtpd[212491]: 03c6e38b6a06134d smtp authentication [email protected] result=ok
Oct  3 20:52:15 orc smtpd[212490]: rspamd: failed to decode JSON response
Oct  3 20:52:15 orc smtpd[212491]: 03c6e38b6a06134d smtp failed-command command="DATA" result="421 server internal error"

Curiously incoming mails seem to work. Now how did this Go work... :)

X-Spam-Symbols

we should add the X-Spam-Symbols header when we match

rspamd settings has wrong example

README contains wrong example of user settings.

outgoing {
    id = "outgoing";
    apply {
        enable_groups = ["dkim"];
        actions {
            reject = 100.0;
            greylist = 100.0;
            "add header" = 100.0;
        }
    }
}

enable_groups shall be written as groups_enabled. See documentation - https://rspamd.com/doc/configuration/settings.html

Optionally connect to rspamd with unix socket

Hello Gilles,

I usually use unix sockets to connect to local services and I would like to do the same with rspamd1. However I can not use your filter this way.

Indeed filter-rspamd -url unix:///run/rspamd/normal.sock doesn't work, even with appropriate permissions, so I guess unix socket are not supported, or my syntax is incorrect ?

Kévin.