include_tasks instead of import_tasks about smartserver

[HolgerHees]

I committed changes to the main branch.

I added a custom ansible config parser which allows a better default variables check. It gives also better error messages in case that something goes wrong. And it is faster. Specially as we now have some more default variables.

https://github.com/HolgerHees/smartserver/blob/master/action_plugins/include_config.py

And after some refactorings nearly everything could be switched 'on' and 'off', in theory :-) The deployment runs for me, with everything switched on (which is still the default). But feel free to switch everything off and report if it breaks something.

Some flags can be directly changed from 'yes' to 'no'. Others are depending on additional values. e.g. 'samba_enabled' can be directly changed from 'yes' to 'no', but 'pihole_enabled' is automatically changed to 'no' if there are no 'pihole_ip' is defined. For details, how it works, you can check. I also reorganized and documented it a bit.

https://github.com/HolgerHees/smartserver/blob/master/config/default.yml

To get an overview what feature you can change take a look at

https://github.com/HolgerHees/smartserver/blob/master/server.yml

I changed also all imports to includes, because it decreased the runtime a lot, in case that you deploy only one role. Before, ansible was compiling everything (all roles and all imports) again and again. Now the includes are only processed on a full run.

Before you start to deploy, you should wait until my CI Tests are done. So hopefully tomorrow.

----- UPDATE -----

I made a lot of variables now optional too, in case that their related service is disabled

20d5e0f

[HolgerHees]

A new commit, related to this topic with the following changes arrives.

5eaa289

1. Lot of cleanups and simplifications happens again.
2. Some additional services like backup_rsync oder speedtest can be disabled now
3. config/default.yml is better structured and documented
4. config/default.yml supports now dependencies. This means, if you disable a service where another service depends on, you will get an error early. e.g. you disable mysql, but another service which depends on mysql, is still active.
5. Better usage statics about variables. This means you can see:
   5.1 Names of custom variables, which are used in your own roles.
   5.2 Names of adjusted variables , which are overwriting default values.
   5.3 Names of default variables, which are not adjusted.
   5.4 Names of unused default variables (because the feature is disabled)
   5.5 Names of adjusted variables, which can be removed or cleaned, because the related feature is now disabled.

I will close this ticket now, but feel free to open new issues in case that you find some bugs.

[HolgerHees]

I'm still running OpenSuse 15.4, but I upgraded ansible to version "4.10.0" right now, which is based on ansible-core "2.11.12"

https://docs.ansible.com/ansible/latest/roadmap/COLLECTIONS_4.html

but I can't reproduce this error. I can't switch to version "5.10.0" which is based on ansible-core "2.12", because this requires python 3.8 and I stay with python 3.6

Is the recommended change fixing your case? If yes, I can made the change, but I can't verify for now.

b.t.w. I install ansible by using pip

[HolgerHees]

I added a branch which made this changes. Can you try if it works for you?

https://github.com/HolgerHees/smartserver/tree/switch-imports

btw. there are a lot of other changes from the last few days...

[HolgerHees]

As longer as I play around, I believe that this is not the reason. The breaking change from ansible version 2.8 should only affects listeners imported inside handlers.

Can you describe a little bit more detailed your case?

"The requested handler 'refresh prometheus' was not found" was happening inside a custom role from your setup or inside a role from smartserver?

Which role was it exactly? Maybe you can provide some code.

Have you run ansible with tags or have you a custom server.yml with different hosts?

From my understanding the "import role" statement inside the server.yml is compiled during startup and this includes the handlers too. Also if I skip e.g. prometheus by using a different tag, the prometheus handler is still available.

e.g. I can run ansible with --tags="camera_service" and I can trigger the "refresh prometheus" handler. I tested this also with ansible "4.10.0" => ansible-core "2.11.12" and it always works.

Before I will merge the new branch I will fully understand the problem and I want to be able to reproduce it locally.

[toffee]

I tried to setup a new server from scratch (but a slim version - without, for example, openhab, nextcloud, prometheus, grafana, wireguard - I commented some pre_tasks and tasks from server.yml). Fresh openSUSE 15.5, ansible installed with pip (without mentioning a specific version).

The first error I got was in hardware_smartd due to missing folders {{global_etc}}prometheus/rules/. I uncommented the line and the next error was the one from the original post (so, the hardware_smartd role cannot find the restart prometheus handler).

If I run again the ansible the error disappear (I suppose because the file was copied and no call for "notify") but appear with the next roles that call import_tasks: roles/prometheus/shared/add_config.yml (in my case was, if I remember correctly, the postfix).

The work around in my case was to run ansible multiple times. I guess, I can reproduce by deleting the files copied in {{global_etc}}prometheus/rules/ and {{global_etc}}prometheus/scrapes/{{name}}.yml and then I can confirm if your fix from branch is working.

[HolgerHees]

ok. then the reason is that you commented out prometheus, because in this case the handler is not included. For now I have no easy idea how to fix, my branch will not solve the problem.

A lot of services are registering a prometheus scrape and role (via add_config) for monitoring that everything runs fine. and there is no easy way to solve this. The same happens too for grafana, but this is not used so expensively, so maybe this works.

I would recommend to keep at least prometheus and just exclude nextcloud and openhab for now.

One possible solution could be a global flag like "prometheus_disable" which will script the prometheus role and all imports of add_configs, but it will increase the deployment complexity a lot. If you want to save some resources like memory I would recommend to disable libreoffice which will save 800MB and this will only be used by nextcloud.

[HolgerHees]

I will add some customizeable flags in the future like

feature_openhab: yes
feature_nextcloud: yes
feature_monitoring: yes
feature_influxdb: {feature_openhab | feature _system_service}
feature_xxx: yes

Etc which handle all of this in a proper way without the need of commenting out anything and break something.

I will check which features are easy to enable/disable in this way...

[toffee]

Looks great! Thank you! I will start checking this evening.

[toffee]

Excellent! I've integrated these changes and everything seems to work as expected. Thanks!