Traefik about docker-fail2ban

[crazy-max]

@gurabli I have made some modifications to allow to specifiy the iptables chain to which the Fail2Ban rules should be added. See this section for more info.

[onedr0p]

Post the output of iptables again

[crazy-max]

I have tested the conf and it works well for me :

fail2ban service output

fail2ban    | 2018-11-18 22:34:05,947 fail2ban.jail           [1]: INFO    Creating new jail 'traefik-auth'
fail2ban    | 2018-11-18 22:34:05,950 fail2ban.jail           [1]: INFO    Jail 'traefik-auth' uses poller {}
fail2ban    | 2018-11-18 22:34:05,951 fail2ban.jail           [1]: INFO    Initiated 'polling' backend
fail2ban    | 2018-11-18 22:34:05,966 fail2ban.filter         [1]: INFO    Added logfile: '/var/log/traefik/access.log' (pos = 0, hash = 57d2a34fe32529ffd7b37c75e28ff3e7)
fail2ban    | 2018-11-18 22:34:05,969 fail2ban.filter         [1]: INFO      maxRetry: 3
fail2ban    | 2018-11-18 22:34:05,971 fail2ban.filter         [1]: INFO      encoding: UTF-8
fail2ban    | 2018-11-18 22:34:05,976 fail2ban.filter         [1]: INFO      findtime: 600
fail2ban    | 2018-11-18 22:34:05,979 fail2ban.actions        [1]: INFO      banTime: 600
fail2ban    | 2018-11-18 22:34:06,046 fail2ban.jail           [1]: INFO    Jail 'sshd' started
fail2ban    | 2018-11-18 22:34:06,059 fail2ban.jail           [1]: INFO    Jail 'traefik-auth' started
fail2ban    | Server ready
fail2ban    | 2018-11-18 22:34:34,843 fail2ban.filter         [1]: INFO    [traefik-auth] Found 10.0.0.2 - 2018-11-18 22:34:34
fail2ban    | 2018-11-18 22:34:43,509 fail2ban.filter         [1]: INFO    [traefik-auth] Found 10.0.0.2 - 2018-11-18 22:34:43
fail2ban    | 2018-11-18 22:34:49,528 fail2ban.filter         [1]: INFO    [traefik-auth] Found 10.0.0.2 - 2018-11-18 22:34:49
fail2ban    | 2018-11-18 22:34:49,789 fail2ban.actions        [1]: NOTICE  [traefik-auth] Ban 10.0.0.2

sudo iptables -L

...

Chain DOCKER-USER (1 references)
target     prot opt source               destination
f2b-traefik-auth  tcp  --  anywhere             anywhere             multiport dports http,https
RETURN     all  --  anywhere             anywhere

...

Chain f2b-traefik-auth (1 references)
target     prot opt source               destination
REJECT     all  --  10.0.0.2  anywhere             reject-with icmp-port-unreachable
RETURN     all  --  anywhere             anywhere

[onedr0p]

@gurabli did you pull the new docker image?

[onedr0p]

Nice, just made those changes :)

[gurabli]

@gurabli did you pull the new docker image?

Oh, n00b mistake. I pulled the new docker image, and now it is working! Many thanks for the great support!

I do have some other problem, I'm still investigating this, but it looks that the Traefik filter failregex doesn't detect logins properly. I navigate to the Traefik dashboard, auth pops up, and this is already considered as one failed attempt. I enter the proper user/pass, it logs in. If I do a refresh, than I get a ban. Not sure what can it be. I cleared the logs, db.

[crazy-max]

I saw that too @gurabli, maybe this as something to do with authentication header. You should open an issue about that on their repository.

[gurabli]

I saw that too @gurabli, maybe this as something to do with authentication header. You should open an issue about that on their repository.

I would like, but don't get it what should I ask properly :)
Will see if this is happening with other auth or just for Traefik dashboard.

[crazy-max]

Hi @gurabli,

I was thinking to develop a feature for this container to listen logs of others containers through docker api. But since docker-ce 18.09.0 we can use a "local" log driver so it would be easy to use your traefik's logs with the fail2ban image.

Concerning the SSH jail, there is an example in the README. For other jails, feel free to open a PR if you have some nice jails 🙂

[onedr0p]

@gurabli see here how I managed to use this with Traefik.

[gurabli]

Thanks for this, I will try to configure accordingly.

Will this enable me to monitor processes that are proxied by Traefik (but not using basic auth, but their own login form, like Sonarr, Radarr, etc)?

[onedr0p]

I do not believe so. What I do is use basic auth on all my services and disable application auth, for example in Radarr and Sonarr. It's very easy to set up with Traefik. Let me know if you want an example.

Moreover, I use Traefik for basic auth with all my external facing applications except ones that require auth like Ombi, Tautulli, Guacamole, Nextcloud etc...

[crazy-max]

@onedr0p I think we could create some jails examples in this repository against Docker images with a compose example. What do you think ?

[gurabli]

I managed to configure, at least partially.
I think I have successfully enabled Traefik and SSH jails. The only problem now I see is the mail notifications, something I would like to disable. I didn't configure mail in fail2ban container, so I have the following errors in logs:
2018-11-18 17:45:22,312 fail2ban.actions [1]: ERROR Failed to execute ban jail 'sshd' action 'sendmail-whois-lines' info....

and

  File "/usr/lib/python2.7/site-packages/fail2ban/server/actions.py", line 405, in __checkBan


    action.ban(aInfo)


  File "/usr/lib/python2.7/site-packages/fail2ban/server/action.py", line 459, in ban


    raise RuntimeError("Error banning %(ip)s" % aInfo)


RuntimeError: Error banning 45.119.xxx.xxx

How can I disable mail send action, and to have everything only in logs?

EDIT:
I guess I need to change F2B_ACTION but the question is, to what? What actions can I choose from?

[crazy-max]

@gurabli Try with %(action_)s

[onedr0p]

@gurabli I know this isn't a solution to your problem but I don't expose ssh to the outside world. Instead I use Guacamole with the Duo plugin for 2FA to do that. IMO it is much safer.

[gurabli]

@crazy-max I tried with %(action_)s and it looks it did the trick. Thanks!
@onedr0p Yes, that is one solution, but now I need to protect port 22.

I think traefik jail is not working. It registers the failed auth login attempts, IP is reported as banned, but I still receive the login prompt in browser. Any ideas why?

[onedr0p]

@gurabli I would double check the logs. How are you banning IPs? Check iptables if you're banning IPs that way.

[gurabli]

I'm banning IPs with iptables-allports The iptables rules are added, I see the IP listed. Still, traefik.mydomain.com is accessible from the same (VPN) IP, and if I enter the correct credentials, than I can login. In logs I see the IP stated as already banned. Check it, try to access your traefik domain over VPN and after 3 failed attempts (or the number you have configured) see what comes up.
SSHD jail is working fine, connection is refused.

[onedr0p]

Silly question, but are you banning IPs on the node traefik is running on? Traefik and fail2ban container should be running on the same node.

[gurabli]

No, it is not a silly question. I am thinking of the same, I mean the network.
By node do you mean docker network where traefik is running?

[onedr0p]

Node meaning the host / server.

[gurabli]

Well, both traefik and fail2ban containers use the same network: traefik_proxy

I have the following in the fail2ban logs, in DEBUG mode, is it correct:

2018-11-18 20:20:15,694 fail2ban.server         [1]: DEBUG     prefregex: '^<F-MLFID>(?:\\[\\])?\\s*(?:<[^.]+\\.[^.]+>\\s+)?(?:\\S+\\s+)?(?:kernel: \\[ *\\d+\\.\\d+\\]\\s+)?(?:@vserver_\\S+\\s+)?(?:(?:(?:\\[\\d+\\])?:\\s+[\\[\\(]?sshd(?:\\(\\S+\\))?[\\]\\)]?:?|[\\[\\(]?sshd(?:\\(\\S+\\))?[\\]\\)]?:?(?:\\[\\d+\\])?:?)\\s+)?(?:\\[ID \\d+ \\S+\\]\\s+)?</F-MLFID>(?:(?:error|fatal): (?:PAM: )?)?<F-CONTENT>.+</F-CONTENT>$'


2018-11-18 20:20:15,704 fail2ban.server         [1]: DEBUG     failregex: '^[aA]uthentication (?:failure|error|failed) for <F-USER>.*</F-USER> from <HOST>( via \\S+)?(?: (?:port \\d+|on \\S+|\\[preauth\\])){0,3}\\s*$'


2018-11-18 20:20:15,710 fail2ban.server         [1]: DEBUG     failregex: '^User not known to the underlying authentication module for <F-USER>.*</F-USER> from <HOST>(?: (?:port \\d+|on \\S+|\\[preauth\\])){0,3}\\s*$'


2018-11-18 20:20:15,715 fail2ban.server         [1]: DEBUG     failregex: '^Failed publickey for invalid user <F-USER>(?P<cond_user>\\S+)|(?:(?! from ).)*?</F-USER> from <HOST>(?: (?:port \\d+|on \\S+)){0,2}(?: ssh\\d*)?(?(cond_user): |(?:(?:(?! from ).)*)$)'


2018-11-18 20:20:15,721 fail2ban.server         [1]: DEBUG     failregex: '^Failed \\b(?!publickey)\\S+ for (?P<cond_inv>invalid user )?<F-USER>(?P<cond_user>\\S+)|(?(cond_inv)(?:(?! from ).)*?|[^:]+)</F-USER> from <HOST>(?: (?:port \\d+|on \\S+)){0,2}(?: ssh\\d*)?(?(cond_user): |(?:(?:(?! from ).)*)$)'


2018-11-18 20:20:15,728 fail2ban.server         [1]: DEBUG     failregex: '^<F-USER>ROOT</F-USER> LOGIN REFUSED FROM <HOST>'


2018-11-18 20:20:15,732 fail2ban.server         [1]: DEBUG     failregex: '^[iI](?:llegal|nvalid) user <F-USER>.*?</F-USER> from <HOST>(?: (?:port \\d+|on \\S+|\\[preauth\\])){0,3}\\s*$'


2018-11-18 20:20:15,737 fail2ban.server         [1]: DEBUG     failregex: '^User <F-USER>.+</F-USER> from <HOST> not allowed because not listed in AllowUsers(?: (?:port \\d+|on \\S+|\\[preauth\\])){0,3}\\s*$'


2018-11-18 20:20:15,742 fail2ban.server         [1]: DEBUG     failregex: '^User <F-USER>.+</F-USER> from <HOST> not allowed because listed in DenyUsers(?: (?:port \\d+|on \\S+|\\[preauth\\])){0,3}\\s*$'


2018-11-18 20:20:15,747 fail2ban.server         [1]: DEBUG     failregex: '^User <F-USER>.+</F-USER> from <HOST> not allowed because not in any group(?: (?:port \\d+|on \\S+|\\[preauth\\])){0,3}\\s*$'


2018-11-18 20:20:15,752 fail2ban.server         [1]: DEBUG     failregex: '^refused connect from \\S+ \\(<HOST>\\)'


2018-11-18 20:20:15,756 fail2ban.server         [1]: DEBUG     failregex: '^Received <F-MLFFORGET>disconnect</F-MLFFORGET> from <HOST>(?: (?:port \\d+|on \\S+)){0,2}:\\s*3: .*: Auth fail(?: (?:port \\d+|on \\S+|\\[preauth\\])){0,3}\\s*$'


2018-11-18 20:20:15,762 fail2ban.server         [1]: DEBUG     failregex: '^User <F-USER>.+</F-USER> from <HOST> not allowed because a group is listed in DenyGroups(?: (?:port \\d+|on \\S+|\\[preauth\\])){0,3}\\s*$'


2018-11-18 20:20:15,767 fail2ban.server         [1]: DEBUG     failregex: "^User <F-USER>.+</F-USER> from <HOST> not allowed because none of user's groups are listed in AllowGroups(?: (?:port \\d+|on \\S+|\\[preauth\\])){0,3}\\s*$"


2018-11-18 20:20:15,772 fail2ban.server         [1]: DEBUG     failregex: '^<F-NOFAIL>pam_[a-z]+\\(sshd:auth\\):\\s+authentication failure;</F-NOFAIL>(?:\\s+(?:(?:logname|e?uid|tty)=\\S*)){0,4}\\s+ruser=<F-ALT_USER>\\S*</F-ALT_USER>\\s+rhost=<HOST>(?:\\s+user=<F-USER>\\S*</F-USER>)?(?: (?:port \\d+|on \\S+|\\[preauth\\])){0,3}\\s*$'


2018-11-18 20:20:15,779 fail2ban.server         [1]: DEBUG     failregex: '^(error: )?maximum authentication attempts exceeded for <F-USER>.*</F-USER> from <HOST>(?: (?:port \\d+|on \\S+)){0,2}(?: ssh\\d*)?(?: (?:port \\d+|on \\S+|\\[preauth\\])){0,3}\\s*$'


2018-11-18 20:20:15,785 fail2ban.server         [1]: DEBUG     failregex: '^User <F-USER>.+</F-USER> not allowed because account is locked(?: (?:port \\d+|on \\S+|\\[preauth\\])){0,3}\\s*'


2018-11-18 20:20:15,788 fail2ban.server         [1]: DEBUG     failregex: '^<F-MLFFORGET>Disconnecting</F-MLFFORGET>: Too many authentication failures(?: for <F-USER>.+?</F-USER>)?(?: (?:port \\d+|on \\S+|\\[preauth\\])){0,3}\\s*$'


2018-11-18 20:20:15,790 fail2ban.server         [1]: DEBUG     failregex: '^<F-NOFAIL>Received <F-MLFFORGET>disconnect</F-MLFFORGET></F-NOFAIL> from <HOST>(?: (?:port \\d+|on \\S+)){0,2}:\\s*11:'


2018-11-18 20:20:15,795 fail2ban.server         [1]: DEBUG     failregex: '^<F-NOFAIL>Connection <F-MLFFORGET>closed</F-MLFFORGET></F-NOFAIL> by(?: authenticating user <F-USER>\\S+|.+?</F-USER>)? <HOST>(?: (?:port \\d+|on \\S+)){0,2}\\s*$'


2018-11-18 20:20:15,800 fail2ban.server         [1]: DEBUG     failregex: '^<F-MLFFORGET><F-NOFAIL>Accepted \\w+</F-NOFAIL></F-MLFFORGET> for <F-USER>\\S+</F-USER> from <HOST>(?:\\s|$)'


2018-11-18 20:20:15,805 fail2ban.server         [1]: DEBUG     failregex: '^Did not receive identification string from <HOST>'


2018-11-18 20:20:15,808 fail2ban.server         [1]: DEBUG     failregex: '^Connection <F-MLFFORGET>reset</F-MLFFORGET> by <HOST>'


2018-11-18 20:20:15,812 fail2ban.server         [1]: DEBUG     failregex: '^Connection <F-MLFFORGET>closed</F-MLFFORGET> by(?: authenticating user <F-USER>\\S+|.+?</F-USER>)? <HOST>(?: (?:port \\d+|on \\S+)){0,2}\\s+\\[preauth\\]\\s*$'


2018-11-18 20:20:15,817 fail2ban.server         [1]: DEBUG     failregex: '^<F-NOFAIL>SSH: Server;Ltype:</F-NOFAIL> (?:Authname|Version|Kex);Remote: <HOST>-\\d+;[A-Z]\\w+:'


2018-11-18 20:20:15,822 fail2ban.server         [1]: DEBUG     failregex: '^Read from socket failed: Connection <F-MLFFORGET>reset</F-MLFFORGET> by peer'


2018-11-18 20:20:15,823 fail2ban.server         [1]: DEBUG     failregex: '^Received <F-MLFFORGET>disconnect</F-MLFFORGET> from <HOST>(?: (?:port \\d+|on \\S+)){0,2}:\\s*14: No supported authentication methods available'


2018-11-18 20:20:15,828 fail2ban.server         [1]: DEBUG     failregex: '^Unable to negotiate with <HOST>(?: (?:port \\d+|on \\S+)){0,2}: no matching (?:(?:\\w+ (?!found\\b)){0,2}\\w+) found.'


2018-11-18 20:20:15,833 fail2ban.server         [1]: DEBUG     failregex: '^Unable to negotiate a (?:(?:\\w+ (?!found\\b)){0,2}\\w+)'


2018-11-18 20:20:15,834 fail2ban.server         [1]: DEBUG     failregex: '^no matching (?:(?:\\w+ (?!found\\b)){0,2}\\w+) found:'


2018-11-18 20:20:15,835 fail2ban.server         [1]: DEBUG     failregex: '^<F-NOFAIL>Connection from</F-NOFAIL> <HOST>'

[gurabli]

Here is my docker-compose, it works for SSH:

  fail2ban:
    image: crazymax/fail2ban:latest
    container_name: fail2ban
    networks:
      - traefik_proxy
    cap_add:
      - NET_ADMIN
      - NET_RAW
    environment:
      - TZ=${TZ}
      - F2B_MAX_RETRY=3
      - F2B_LOG_LEVEL=DEBUG
      - F2B_ACTION=%(action_)s
    volumes:
      - /var/log:/var/log:ro
      - ${USERDIR}/docker/fail2ban/data:/data

[onedr0p]

I am not sure if the docker network matters in this case for that, since this is iptables level ban. Are the traefik container and fail2bans container on the same server?

[gurabli]

Yes, same server. On my home server.

Edit: if you fail the set number of logins for traefik, what do you see in browser (after the IP is banned)?

[onedr0p]

This doesn't seem to be an issue with Traefik or fail2ban containers since the rules are getting added to iptables. I don't use iptables to ban (I send those IPs to cloudflare and let them handle it) it is hard for me to debug with you.

[crazy-max]

@gurabli Can you post the output of sudo iptables -L and docker info ?

[gurabli]

@crazy-max Sure
iptables -L Pastebin (didn't comment out the banned IP's, as they deserve to be banned as trying to brake into my system:) )

docker info: Pastebin (commented out my domain)

EDIT:
Maybe needed, my Traefik docker-compose: Pastebin

[onedr0p]

@gurabli what is the point in exposing the traefik dashboard to the web?

[crazy-max]

@onedr0p Yeah you right, he should not expose the dashboard. And --web is deprecated btw.

[gurabli]

@crazy-max Thanks, added F2B_IPTABLES_CHAIN=DOCKER-USER to fail2ban and removed --web from Traefik.
Still, it doesn't work :( I can still access Traefik dashboard after ban.

BTW: just using Trafik dashboard to test, it will not be exposed to web!

[onedr0p]

@crazy-max

And --web is deprecated btw.

Where do you see that? I cannot find anything in their changelogs

[crazy-max]

@onedr0p First see in the logs while using 1.6 and from the commande line :

--web                                         (Deprecated) Enable Web backend with default settings (default "false")
--web.address                                 (Deprecated) Web administration port (default ":8080")
--web.certfile                                (Deprecated) SSL certificate
--web.keyfile                                 (Deprecated) SSL certificate
--web.metrics                                 (Deprecated) Enable a metrics exporter

Now it's handled through API definition :

--api=true
--api.dashboard=true

[crazy-max]

I have created a quick example using Traefik. Feel free to open a PR if you have suggestions.