Bad to create breaking changes when you have people using an auto-update script about nginx-ultimate-bad-bot-blocker

[itoffshore]

@mitchellkrogza - use the virtual

[itoffshore]

@mitchellkrogza - yes is why Alpine is the default flavor in docker now (& not ubuntu any more)

[mitchellkrogza]

Apologies for an Nginx EMERG behind your back, recent changes just had to be made. There are constant improvements being done all the time and @itoffshore has dedicated a lot of time to getting the update and setup scripts to be universal and work on all distro's.

The changes with all the include files I knew well in advance would cause issues for some people running auto update, it was very unfortunate but they just had to be implemented for helping users to be able to better customize the blocker to suit their own needs.

Assuring you we are doing our best to not break things behind your back.

[pjv]

Yes, I'm not saying don't make improvements. It's fantastic to make it better.

I'm saying before you make a breaking change that is definitely going to get pushed to people's production servers behind their backs (it did mine), you should set up a system so that the updates won't break their server.

It would be a lot of additional work to do. Given that you already have users out there using the auto update script, it would probably mean setting up a new version, with a new auto-update script that is capable of first updating itself before it updates any other files and then you could push whatever changes are necessary to that auto-update script to make sure that when it does the next update it won't make breaking changes (e.g., if in the last big update, the auto-update script had first updated itself, then it could have been changed to include tests for the existence of the newly required bots.d/*.conf files and create them if they were not found.)

Then you would have to keep the old system without the breaking updates running in parallel to the new version that has the self-updating auto-updates. That would keep people from breaking their production servers without knowing about it.

EDIT: Otherwise, just don't offer an auto-update script. If people create one on their own, then it's their own responsibility to make sure that they are not breaking their system. But if you offer the auto-update script, then I think it is YOUR responsibility to make sure you are not breaking their system.

[pjv]

And one more thing: if you do create a breaking change, like you did, it would be nice if you could also create a script that would fix it FAST, rather than making people who might be desperate to get their production server back online re-read a long page of documentation and try to figure out which parts are new vs. which parts they already have set up.

[pjv]

And finally: sorry to be so pissy. This is a great project and I am very grateful that you put it up here and make it available for everyone. I should have said that first!

[mitchellkrogza]

@pjv once again sincere apologies, the only way to help prevent this in future is to subscribe to the mailing list.

Also if you look in the builds section you would also quickly see notes when a major change has been introduced and what needs to be done. See the notes I added on https://github.com/mitchellkrogza/nginx-ultimate-bad-bot-blocker/releases/tag/2017-04-20.Build-340

I am so glad the blocker is helping you, I only aim to keep making it better and no breaking changes. I am hoping the mailing list will help with keeping users notified of any upcoming major changes. But I will make sure next time there is any changes coming that they are planned for well in advance so that things like the update script etc are also updated in advance to check for the existence of all required files when pulling updates and download them automatically during the update or else the update will not proceed and email a notification saying the update did not happen because of xxxxx .... I am sure @itoffshore is still going to be able to improve greatly on the update script and I will make sure he is always informed of changes well ahead of any.

For now ..... now more major changes planned that will break the blocker .... promise 😄

[itoffshore]

@pjv - all you had to do was run:

install-ngxblocker (to install the new files) - this was written specifically for the big update
setup-ngxblocker (to insert the additional new includes) - also revised for the big update

update-ngxblocker is only for updating globalblacklist.conf- also revised with command line switches

[pjv]

@itoffshore thanks, that's good to know. Would have been better to know that before i fixed things manually. I might be blind but I didn't see those simple instructions anywhere until you just put them in here. I was not sure if the install and setup scripts were idempotent and could be safely run again on a system that has already been installed and set up.

so those scripts will not over-write an existing file like whitelist-ips.conf that may have been customized by the user?

[itoffshore]

@pjv it only replaces missing files so won't overwrite anything

see check_if_updating() in install-ngxblocker

I'm the package maintainer for nginx-naxsi in Alpine Linux & test everything fully on my own servers before making a PR.

I did put some notes in the new PR's but the man pages & docs could probably do with updating.

[itoffshore]

@pjv - also when you run:

setup-ngxblocker
install-ngxblocker

without -x the scripts print what they will update / insert (but do not make any changes).

I put these features in so by default nothing unexpected occurs.

[pjv]

@itoffshore I just tried to run the new install script on another server (where the blocker is already installed) and got this error:

Error sourcing variables from: https://raw.githubusercontent.com/mitchellkrogza/nginx-ultimate-bad-bot-blocker/master/include_filelist.txt

This is on ubuntu 16.0.4

[itoffshore]

@pjv @mitchellkrogza - fixed for ubuntu in #40

[itoffshore]

Checking url: https://raw.githubusercontent.com/mitchellkrogza/nginx-ultimate-bad-bot-blocker/master/include_filelist.txt
Creating directory: /etc/nginx/conf.d
Creating directory: /etc/nginx/bots.d

REPO = https://raw.githubusercontent.com/mitchellkrogza/nginx-ultimate-bad-bot-blocker/master

Downloading [FROM]=>  [REPO]/conf.d/globalblacklist.conf            [TO]=>  /etc/nginx/conf.d/globalblacklist.conf...OK
Downloading [FROM]=>  [REPO]/conf.d/botblocker-nginx-settings.conf  [TO]=>  /etc/nginx/conf.d/botblocker-nginx-settings.conf...OK

REPO = https://raw.githubusercontent.com/mitchellkrogza/nginx-ultimate-bad-bot-blocker/master

Downloading [FROM]=>  [REPO]/bots.d/blockbots.conf              [TO]=>  /etc/nginx/bots.d/blockbots.conf...OK
Downloading [FROM]=>  [REPO]/bots.d/ddos.conf                   [TO]=>  /etc/nginx/bots.d/ddos.conf...OK
Downloading [FROM]=>  [REPO]/bots.d/whitelist-ips.conf          [TO]=>  /etc/nginx/bots.d/whitelist-ips.conf...OK
Downloading [FROM]=>  [REPO]/bots.d/whitelist-domains.conf      [TO]=>  /etc/nginx/bots.d/whitelist-domains.conf...OK
Downloading [FROM]=>  [REPO]/bots.d/blacklist-user-agents.conf  [TO]=>  /etc/nginx/bots.d/blacklist-user-agents.conf...OK
Downloading [FROM]=>  [REPO]/bots.d/blacklist-ips.conf          [TO]=>  /etc/nginx/bots.d/blacklist-ips.conf...OK
Downloading [FROM]=>  [REPO]/bots.d/bad-referrer-words.conf     [TO]=>  /etc/nginx/bots.d/bad-referrer-words.conf...OK
Downloading [FROM]=>  [REPO]/bots.d/custom-bad-referrers.conf   [TO]=>  /etc/nginx/bots.d/custom-bad-referrers.conf...OK
root@ubuntu:~# 

[pjv]

sudo ./install-ngxblocker
Checking url: https://raw.githubusercontent.com/mitchellkrogza/nginx-ultimate-bad-bot-blocker/master/include_filelist.txt
Spider mode enabled. Check if remote file exists.
--2017-04-27 14:08:32--  https://raw.githubusercontent.com/mitchellkrogza/nginx-ultimate-bad-bot-blocker/master/include_filelist.txt
Resolving raw.githubusercontent.com (raw.githubusercontent.com)... 151.101.48.133
Connecting to raw.githubusercontent.com (raw.githubusercontent.com)|151.101.48.133|:443... connected.
HTTP request sent, awaiting response... 200 OK
Length: 485 [text/plain]
Remote file exists.


** Dry Run ** | -x or --exec to download files

./install-ngxblocker: 132: local: /etc/nginx/bots.d: bad variable name

[mitchellkrogza]

Can confirm on Ubuntu 16.04

./install-ngxblocker

Checking url: https://raw.githubusercontent.com/mitchellkrogza/nginx-ultimate-bad-bot-blocker/master/include_filelist.txt
Spider mode enabled. Check if remote file exists.
--2017-04-27 16:21:11--  https://raw.githubusercontent.com/mitchellkrogza/nginx-ultimate-bad-bot-blocker/master/include_filelist.txt
Resolving raw.githubusercontent.com (raw.githubusercontent.com)... 151.101.0.133, 151.101.192.133, 151.101.64.133, ...
Connecting to raw.githubusercontent.com (raw.githubusercontent.com)|151.101.0.133|:443... connected.
HTTP request sent, awaiting response... 200 OK
Length: 485 [text/plain]
Remote file exists.


** Dry Run ** | -x or --exec to download files

./install-ngxblocker: 132: local: /etc/nginx/bots.d: bad variable name

and

./setup-ngxblocker

Checking url: https://raw.githubusercontent.com/mitchellkrogza/nginx-ultimate-bad-bot-blocker/master/include_filelist.txt
Spider mode enabled. Check if remote file exists.
--2017-04-27 16:22:17--  https://raw.githubusercontent.com/mitchellkrogza/nginx-ultimate-bad-bot-blocker/master/include_filelist.txt
Resolving raw.githubusercontent.com (raw.githubusercontent.com)... 151.101.0.133, 151.101.64.133, 151.101.128.133, ...
Connecting to raw.githubusercontent.com (raw.githubusercontent.com)|151.101.0.133|:443... connected.
HTTP request sent, awaiting response... 200 OK
Length: 485 [text/plain]
Remote file exists.

Error sourcing variables from: https://raw.githubusercontent.com/mitchellkrogza/nginx-ultimate-bad-bot-blocker/master/include_filelist.txt

[mitchellkrogza]

Once we get this sorted, I will try adding the testing of these scripts into the Travis tests as they run in an Ubuntu container. @pjv I'm sure we will have this sorted 100% shortly, thanks for your patience.

[mitchellkrogza]

@itoffshore what's best ISO version of Alpine to download and run as a VM? Standard / Extended / Virtual

[mitchellkrogza]

Great thanks Stuart, never checked out Alpine so am now doing just that. Utterly amazed at the size of the ISO for Virtual 26 Mb 😮 smallest linux ISO I've ever seen yet. Going to play now with it 😀

[itoffshore]

@pjv @mitchellkrogza - change the first line of install-ngxblocker to /bin/bash (from /bin/sh) to get it working temporarily until I find a fix that is compatible with all platforms.

[mitchellkrogza]

Confirmed that #42 fixes install-ngblocker. Thanks Stuart @itoffshore

setup-ngxblocker still gives error

./setup-ngxblocker
Checking url: https://raw.githubusercontent.com/mitchellkrogza/nginx-ultimate-bad-bot-blocker/master/include_filelist.txt
Spider mode enabled. Check if remote file exists.
--2017-04-27 18:13:26--  https://raw.githubusercontent.com/mitchellkrogza/nginx-ultimate-bad-bot-blocker/master/include_filelist.txt
Resolving raw.githubusercontent.com (raw.githubusercontent.com)... 151.101.0.133, 151.101.64.133, 151.101.128.133, ...
Connecting to raw.githubusercontent.com (raw.githubusercontent.com)|151.101.0.133|:443... connected.
HTTP request sent, awaiting response... 200 OK
Length: 485 [text/plain]
Remote file exists.

Error sourcing variables from: https://raw.githubusercontent.com/mitchellkrogza/nginx-ultimate-bad-bot-blocker/master/include_filelist.txt

[pjv]

Confirmed also on my box that #42 ran and installed the missing .conf files to bots.d.

Thanks @itoffshore.

[mitchellkrogza]

Tested install-ngxblocker on clean Ubuntu VM ..... 100% @itoffshore

[itoffshore]

will fix setup-ngxblocker in the next hour or 2

[mitchellkrogza]

@itoffshore you are a STAR so appreciate your valuable time and contributions

…

Sent from my iPhone

On 27 Apr 2017, at 6:28 PM, Stuart Cardall ***@***.***> wrote: will fix setup-ngxblocker in the next hour or 2 — You are receiving this because you modified the open/close state. Reply to this email directly, view it on GitHub, or mute the thread.

[itoffshore]

@pjv - these scripts should hardly change now they work.

the only file that needs to change in the future is the file they source: include_filelist.txt