Do I need access.log to be turned on ? about nginx-ultimate-bad-bot-blocker

[mitchellkrogza]

It should work 100% Nginx does the blocking in real time from memory not from logs.

[mitchellkrogza]

I would however advise against turning off access logs. Without them you have no way of monitoring who is trying to do what to your site. Rather turn it on and modify the log rotate settings so that it rolls over the log daily, gzips the last days log and only keeps 1 - 3 months worth of logs.

[royashgit]

Thanks Mitchell, but I have this problem with logs if you could kindly help:

https://serverfault.com/questions/864664/nginx-no-access-to-log-files

[mitchellkrogza]

@neodjandre2 Please post the contents of your nginx.conf and vhost.conf files (/etc/nginx/sites-available/)

[royashgit]

user www-data;
worker_processes 2;
pid /run/nginx.pid;

events {
        worker_connections 1024;
        multi_accept on;
}

http {

        ##
        # Basic Settings
        ##

client_header_buffer_size 2k;
large_client_header_buffers 2 1k;

client_body_buffer_size 10M;
client_max_body_size 10M;

client_body_timeout 12;
client_header_timeout 12;
keepalive_timeout 15;
send_timeout 10;

        sendfile on;
        tcp_nopush on;
        tcp_nodelay on;
        types_hash_max_size 2048;
        server_tokens off;

        # server_names_hash_bucket_size 64;
        # server_name_in_redirect off;

        include /etc/nginx/mime.types;
        default_type application/octet-stream;

     ##
        # SSL Settings
        ##

        ssl_protocols TLSv1 TLSv1.1 TLSv1.2; # Dropping SSLv3, ref: POODLE
        ssl_prefer_server_ciphers on;

        ##
        # Logging Settings
        ##
        access_log off;
        error_log /var/log/nginx/error.log warn;

        ##
        # Gzip Settings
        ##

        gzip on;
        gzip_disable "msie6";

        # gzip_vary on;
         gzip_proxied any;
         gzip_comp_level 2;
        # gzip_buffers 16 8k;
        # gzip_http_version 1.1;
         gzip_types text/plain text/css application/json application/javascript text/xml application/xml application/xml+rss text/javascript;

        ##
        # Virtual Host Configs
        ##

        include /etc/nginx/conf.d/*.conf;
        include /etc/nginx/sites-enabled/*;

        ##
        # Security
        ##

        add_header Content-Security-Policy "default-src 'self' https: data: 'unsafe-inline' 'unsafe-eval';" always;
        add_header X-Xss-Protection "1; mode=block" always;

}


------------------------------------------------------------------


server {
    listen 80 default_server;
    listen [::]:80 default_server;
    server_name example.com www.example.com;
    return 301 https://$server_name$request_uri;
}

server {

    listen 443 ssl http2 default_server;
    listen [::]:443 ssl http2 default_server;
    include snippets/ssl-example.com.conf;
    include snippets/ssl-params.conf;
    large_client_header_buffers 4 32k;

    root /var/www/html;
    index index.php;

    error_page 401 403 404 /custom_404.html;
        location = /custom_404.html {
                root /usr/share/nginx/html;
                internal;
        }

    error_page 500 502 503 504 /custom_50x.html;
        location = /custom_50x.html {
                root /usr/share/nginx/html;
                internal;
        }

    location / { try_files $uri $uri/ /index.php$is_args$args; }

    location = /favicon.ico { log_not_found off; access_log off; }
    location = /robots.txt { log_not_found off; access_log off; allow all; }

    location ~ \.(ttf|ttc|otf|eot|woff|woff2|font.css|css|svg|svgz|js)$ {
        add_header Access-Control-Allow-Origin "*";
        log_not_found off;
        access_log off;
        expires max;
    }


 location ~* \.(gif|ico|jpeg|jpg|png|bmp|rss)$ {
        log_not_found off;
        access_log off;
        expires max;
    }

    location ~ /\.ht { access_log off; log_not_found off; deny all; }
    location ~ ~$ { access_log off; log_not_found off; deny all; }

    location ~ /.well-known { allow all; }

    location ~* ^/wp-content/uploads/.*.(html|htm|shtml|php)$ { types { } default_type text/plain; }

    location = /xmlrpc.php { deny all; access_log off; log_not_found off; }

    location ~ .(gif|png|jpe?g)$ {
     valid_referers none blocked example.com *.example.com;
     if ($invalid_referer) {
        return   403;
    }
 }

    location ~* .(pl|cgi|py|sh|lua)$ { return 444; }

    rewrite /wp-admin$ $scheme://$host$uri/ permanent;
    include rocket-nginx/rocket-nginx.conf;

    location ~ \.php$ {
        include snippets/fastcgi-php.conf;
        fastcgi_pass unix:/run/php/php7.0-fpm.sock;
    }

    location ~* /wp-includes/.*.php$ {deny all;access_log off;log_not_found off;}
    location ~* /wp-content/.*.php$ {deny all;access_log off;log_not_found off;}
    location ~* /(?:uploads|files)/.*.php$ {deny all;access_log off;log_not_found off;}

    location = /wp-config.php { deny all; }
    location ~* /debug\.log$ {deny  all;}

}

[mitchellkrogza]

Try this.

First set ownership of nginx logs folder

sudo chown -R www-data:adm /var/log/nginx/

Then in your vhost add this:

server {

    listen 443 ssl http2 default_server;
    listen [::]:443 ssl http2 default_server;
    include snippets/ssl-example.com.conf;
    include snippets/ssl-params.conf;
    large_client_header_buffers 4 32k;

    root /var/www/html;
    index index.php;
    #Add this section
    charset UTF-8;
    access_log /var/log/nginx/ssl-example.com-access.log;
    error_log /var/log/nginx/ssl-example.com-error.log;
    ### rest of your config

Then restart nginx and check /var/log/nginx/ for the access logs.

You need to specify log locations as above for all vhosts.

You can even add different logs for your port80 host if you want to monitor redirects etc.

server {
    listen 80 default_server;
    listen [::]:80 default_server;
    server_name example.com www.example.com;
    return 301 https://$server_name$request_uri;

    access_log /var/log/nginx/example.com-access.log;
    error_log /var/log/nginx/example.com-error.log;

}

[royashgit]

ok many thanks, will give this a go !

[royashgit]

do i need to delete this in the http block ?

 ##
    # Logging Settings
    ##
    access_log off;
    error_log /var/log/nginx/error.log warn;

[mitchellkrogza]

No, your Nginx.conf should always have:

	access_log /var/log/nginx/access.log;
	error_log /var/log/nginx/error.log warn;

These two logs can help you identify problems with Nginx itself and not a vhost.

[royashgit]

ok but if i turn the access_log on and then preload my wordpress cache using WP-Rocket, the whole server crashes! must have something to do with my permissions writing I think...

[royashgit]

or the bad bots... which is why i came here in the first place anyhow..

[royashgit]

bcs the bad bots flood my access.log all the time !

[mitchellkrogza]

That's the whole idea of the blocker it to block those things out BUT .... remember this, even though they get denied access to your site by the blocker they still appear in your logs but with a 444 error code. Then you use my fail2ban addon to perma-block repeat 444 and 403 offenders.

I run some big sites, some which generate big logs every day, has absolutely no impact on Nginx of Server performance whatsoever. My one Nginx servers runs 28 sites and at any point in time my processor usage for Nginx is 3-4% of CPU.

[mitchellkrogza]

As a note: Don't use wordpress plugins for caching. Use Nginx's built in fastcgi_cache it's superior to any other caching system and is blazingly fast.

[royashgit]

ok this is helpful stuff... I don't know how to use fastcgi_cache but I will do some research then... Although WP-Rocket does plenty of other staff such as minifaction, CDN etc..

[mitchellkrogza]

Need to learn your way around Nginx first but here's an example of a fastcgi_cache config which gets included within the server {} block of your vhost. Each vhost will have it's own cache settings defined by the name in this example - MYWEBSITECACHE1. you will also notice you can set if statements to prevent it cacheing certain things.

# FastCGI Caching
#################
set $skip_cache 0;
add_header X-Cache $upstream_cache_status;
    # POST requests and URLs with a query string should always go to PHP
	if ($request_method = POST) {
	set $skip_cache 1;
	}

	if ($query_string != "") {
	set $skip_cache 1;
	}

	# Don't cache URIs containing the following segments
	if ($request_uri ~* "/wp-admin/|/xmlrpc.php|wp-.*.php|/feed/|index.php|sitemap(_index)?.xml") {
	set $skip_cache 1;
	}

	# Bypass cache for wp-login redirect
	if ($request_uri ~* "/wp-login.php?redirect_to=|/wp-login.php?redirect_to=%2F|wp-login.php?loggedout=true") {
	set $skip_cache 1;
	}

	location ~ [^/]\.php(/|$) {
	fastcgi_split_path_info ^(.+?\.php)(/.*)$;
	if (!-f $document_root$fastcgi_script_name) {
		return 403;
	}
	# This is a robust solution for path info security issue and works with "cgi.fix_pathinfo = 1" in /etc/php.ini (default)
	include fastcgi.conf;
	fastcgi_pass unix:/run/php/php7.0-fpm.sock;
	fastcgi_index index.php;
	fastcgi_cache_methods GET HEAD;
	fastcgi_pass_header Set-Cookie;
	fastcgi_pass_header Cookie;
	fastcgi_ignore_headers Cache-Control Expires Set-Cookie;
    fastcgi_read_timeout 90s;
	fastcgi_pass_request_headers on;

	# MAKE SURE TO SET CACHE PURGE NAME TO THE SAME NAME BELOW
	fastcgi_cache MYWEBSITECACHE1;
	fastcgi_cache_min_uses 1;
	fastcgi_cache_valid 300s;
	fastcgi_cache_lock on;
	fastcgi_cache_lock_age 5s;
	fastcgi_cache_lock_timeout 5s;
	fastcgi_cache_revalidate on;
	fastcgi_no_cache $skip_cache;
	fastcgi_no_cache $cookie_nocache $arg_nocache$arg_comment;
	fastcgi_no_cache $http_pragma    $http_authorization;
	fastcgi_cache_bypass $skip_cache;
	fastcgi_cache_bypass $cookie_nocache $arg_nocache$arg_comment;
	fastcgi_cache_bypass $http_pragma    $http_authorization;
	fastcgi_temp_file_write_size 32k;
	fastcgi_buffer_size 32k;
    fastcgi_buffers 32 32k;
    }
    location ~ /purge(/.*) {
        fastcgi_cache_purge MYWEBSITECACHE1 "$scheme$request_method$host$1";
    }	

Then in your nginx.conf file you must define these caches.

	fastcgi_cache_path /var/cache/nginx/mwebsite levels=1:2 keys_zone=MYWEBSITECACHE1:10m inactive=60m;
	fastcgi_cache_path /var/cache/nginx/mywebsite1 levels=1:2 keys_zone=MYWEBSITECACHE2:20m inactive=60m;
	fastcgi_cache_use_stale error timeout invalid_header updating http_500;

And then you must create the folder /var/cache/nginx and set permissions on it.

sudo mkdir /var/cache/nginx

sudo chown -R www-data:www-data /var/cache/nginx

And of course restart nginx.

Then you can use this plugin to purge your wordpress sites cache for you:
https://wordpress.org/plugins/nginx-helper/

Or you can run a simple bash script like this.

#!/bin/bash
sudo service nginx stop
sudo rm -rf /var/cache/nginx/*
sudo service nginx start | mail -s "Nginx Purged" [email protected]
sudo chown -R www-data:www-data /var/cache/nginx
exit 0

Have fun !!! 😁

[mitchellkrogza]

@neodjandre2 Just for interest, here's how my main Nginx server looks over a week running 28 sites.

[royashgit]

wow, that looks great :) I should really focus on nginx cache!

[mitchellkrogza]

@neodjandre2 👍 once you learn your way around Nginx and how powerful it is you will never look at anything else. Especially when it comes to cacheing with fastcgi_cache. Nothing out there can beat it.

Also forget about minification of .js and .css files and using CDN's as they actually will not improve your speed or scores.

CDN's are inherently problematic and you are at the peril of their services (which do go down) so I refuse to use any of them.

The same goes for Cloudflare, I will never use it, simply will not ever hand over the control and handling of my DNS to a service that has had major problems already twice this year and last year too.

I run my own DNS, it's solid and works 24/7/365.

You can test fastcgi_cacheing on one of my busiest sites if you like https://www.environment.co.za click on a page, click back, click on it again and see how fast it loads and keep in mind that it a wordpress site that is heavy in plugins and also external assets like adwords. That site gets well over 500,000 visitors a month.

[royashgit]

yes this is blazing fast.. I really should focus on fastcgi..

[mitchellkrogza]

Awesome, learn your way around Nginx their docs are awesome (unlike Apache) and you will soon have web sites that kick *ss

[mitchellkrogza]

I worked on Apache for nearly 10 years, then switched to Nginx in one day and never looked back. Nothing can touch it and even Apache with all their "improvements" are slack to say the least. Their documentation SUCKS, and their performance cannot touch Nginx. Still to this day it is as easy as pie to completely flood an Apache server and bring it down and I have people trying to bring down my Nginx servers 24/7 and they do not and will not succeed.