holgerhees / smartserver

SmartHome Server deployment setup

Smartserver

This project contains my complete automated IoT Server setup, deployable on opensuse, almalinux (rhel) and ubuntu.

For a detailed overview of the installed software and how it can be customized, see the following

• the main deployment server.yml file
• or the default configuration options
• or to see, how it is used, visit https://intranet-of-things.com/

For a detailed description about the main features and a lot more, check the wiki

Demo setup

To try this smartserver deployment, just 3 steps are needed. Install Vagrant and VirtualBox and run the deployment script.

A detailed explaination can be found in the How to start wiki section

Technology & Service stack

After running ansible, the following technology & service stack will be deployed.

In addition to well known services, some custom additions are deployed too.

1. a responsive Web UI, written in pure HTML, CSS and Javascript is deployed too. It is used to embedd webUI's from other services in a central place. It supports "user" and "admin" roles to show only nessecarry menu points to your family members.

2. "Private cloud" is a complete isolated network spanned through wireguard between several smartserver deployments to share backup folder, sensor messurements or other data with friends.

3. "Update service" gives you a webui to handle and manage your os updates and smartserver changes and getting notified if there are new updates available

4. "CI service" is running your smartserver deployment on suse, alma and ubuntu virtual boxes after every commit to vertify that your changes works fine

5. "Weather service" is fetching weather data from a public provider and make it available for my smarthome logic

6. "System Service" is the most complex service. It provides

   • A network scanner which builds and visualizes a network/cable structure of your home network. Data are comming from snmp aware router/switches, fritzbox devices and openwrt powered devices.

   

   • A network state checker, who is permanently monitoring your wan connection state and speed. The reachability of your cloud peers. The reachability of your backup folder etc.

   

   • A netflow and log analyser, who is able to block IP's automatically if they contact your server in a bad way. This includes also traffic like SYN requests which are normally not visible in log files.

   

smartserver's Issues

IndexError on roles/system_service/templates/opt/system_service_libs/lib/netflow/processor.py

The query

results = self.influxdb.query('SELECT "group","value" FROM "netflow_size" WHERE time >= now() - 358m AND "group"::tag != \'normal\'')

in my system returns an empty array. This leads to an exception (IndexError: list index out of range) here

Adding a check if the array is empty solve the issue. Proposed changes in this commit.

Thank you!

How about including Telegraf?

Telegraf broadens the number of entry points for Influxdb and makes collecting metrics much easier.

Need some way to handle self-signed certs in demo mode

Recent browser updates are not allowing self-signed certs to be accepted and there doesn't seem to be any good instructions on how to overcome this. Two possible approaches:

1. provide step-by-step instructions on how to configure the latest releases of Firefox/Chrome to accept self-signed certs from Smartserver.

or

2. provide a mode switch in Smartserver to allow it to operate without self-signed certificates.

I really hope this can be fixed, I'd like to start developing with Smartserver.

Documentation question/suggestion

I was using parts of the repo to help build my own system. But now I'd like to also use the staging/production mix and migrate my ansible roles into this.

It's not clear in https://github.com/HolgerHees/smartserver/wiki/Setup:-Create#create-your-own-setup how to:

• (from laptop) vagrant --config=demo --os=ubuntu up --provision works and provisions to the VM running locally.
• (from laptop) sudo ansible-playbook -i config/demo/server.ini server.yml will only run if I add the production IP to config/demo/server.ini.

Point 5: copying files is unclear - copy config/demo/ to the production server?

Perhaps an example where we:

1. mention adding production ip to server.ini,
2. run with (from laptop) ansible-playbook -i config/demo/server.ini --extra-vars "target=production" server.yml

At least this would be my thinking after playing around with your setup very briefly. I might be missing something so take all of this with a grain of salt.

Wrong check path when configuring grafana with individual dashboards

In file main.yml from grafana role in line is checked for wrong dashboards folder.

Should be similar with the path in this line.

Some suggestions

I love this project, just what I was looking for. A couple of suggestions:

It would be great to have some kind of tree/graph showing the various dependencies of the various subsystems. Then users could known which parts could be safely dropped or modified without side effects.

An overview of how Vagrant and Ansible interact to build out the system would help us understand how it all comes together.

Provide some simple FAQ/tutorial sections, something like:

• How to add a printer/device
• How to add a new category in the top heading

Role hardware_smartd does not support NVMe devices

In my system I have a NVMe drive:

# smartctl --scan
/dev/sda -d scsi # /dev/sda, SCSI device
/dev/sdb -d scsi # /dev/sdb, SCSI device
/dev/nvme0 -d nvme # /dev/nvme0, NVMe device

The command smartctl --scan | grep -oP "^[A-z/]+" returns (please see the missing 0 from /dev/nvme0):

/dev/sda
/dev/sdb
/dev/nvme

If I modify the comand like smartctl --scan | grep -oP "^[A-z0-9/]+" and change the hardware_smartd\templates\smartd.conf to guess the device type (eg. using -d auto) then the issue is fixed in my system.

These changes are in 1beaf45 - maybe you want to incorporate these changes.

system_update_check crash when system is recently updated

Regarding

smartserver/roles/update_daemon/templates/opt/update_daemon/plugins/os/opensuse.py

Line 14 in fb2da21

result = subprocess.run([ "/usr/bin/zypper ps -s" ], shell=True, check=False, stdout=subprocess.PIPE, stderr=subprocess.STDOUT, cwd=None )

On my system (recently updated) the output of command

/usr/bin/zypper ps -s

is:

No processes using deleted files found.

No core libraries or services have been updated since the last system boot.
Reboot is probably not necessary.

The following exception is thrown:

Traceback (most recent call last):
  File "/opt/update_daemon/system_update_check", line 31, in <module>
    repo = plugin.Repository()
  File "/opt/update_daemon/plugins/os/opensuse.py", line 19, in __init__
    self.needs_restart = m.group(0) is not None
AttributeError: 'NoneType' object has no attribute 'group'

For the moment I fixed this by this additional check:

self.needs_restart = False
if m is not None:
    self.needs_restart = m.group(0) is not None

but I'm not sure the logic is correct (needs_restart = False) as long the output says "Reboot is probably not necessary."

trafficblocker blocklist KeyError

Not sure if it's a defect or wrong configuration in my side, anyway I will let you know.

I'm getting the following error:

[ERROR] - [m_service.lib.influxdb:66] - Traceback (most recent call last):
  File "/etc/system_service/lib/influxdb.py", line 62, in run
    messurements += callback()
                    ^^^^^^^^^^
  File "/etc/system_service/lib/trafficwatcher/trafficblocker/trafficblocker.py", line 264, in getMessurements
    messurements.append("trafficblocker,extern_ip={},blocking_state={},blocking_reason={},blocking_list={},blocking_count={} value=\"{}\"".format(ip, data["state"], data["reason"], data["blocklist"], data["count"], data["last"]))
                                                                                                                                                                                     ~~~~^^^^^^^^^^^^^
KeyError: 'blocklist'

If I delete the /dataDisk/var/lib/system_service/trafficblocker.json file (version 3) the error is gone, but the file is empty for the moment. Previously, the file contains entries like (all missing blocklist):

"146.190.40.112": {
    "count": 1,
    "created": 1692759673.4107,
    "details": "scanning",
    "last": 1692759617.3242,
    "reason": "apache",
    "state": "blocked",
    "type": "unknown",
    "updated": 1692759673.4107
},

include_tasks instead of import_tasks

Hello,

By default on OpenSuse 15,5 the ansible version is 2.11.12 and since 2.8 there is a change that should be considered.

I receive

The requested handler 'refresh prometheus' was not found in either the main handlers list nor in the listening handlers list 

and I suppose that all import_tasks like import_tasks: roles/prometheus/shared/add_config.yml should be replaced by include_tasks: like include_tasks roles/prometheus/shared/add_config.yml.

In documentation you mentioned ansible 2.10.7 - I guess is affected by this change.

Edge case - image_proxy.php never ends

Hello Holger,

I've tried to use your recent changes to gallery and I think I found a edge case that may lead to image_proxy.php to never ends.

In my environment (the cameras are not configured to accept Basic auth or the format returned is not recognized by Imagick) seems that curl_exec of the camera snapshot url may throw an exception, and then in subsequent calls the apcu_delete( $url . ":fetch" ); never called and the loop never exit. The workaround is to restart the php container to clean the cache.

PHP logs looks like (with some of your debugging statements uncommented):

NOTICE: PHP message: fetch http://hostname/ipcamera/cam01/ipcamera.jpg
NOTICE: PHP message: 0.0033631324768066
NOTICE: PHP message: fetched size 282
NOTICE: PHP message: calculated size 282
NOTICE: PHP message: PHP Fatal error:  Uncaught ImagickException: no decode delegate for this image format `' @ error/blob.c/BlobToImage/363 in /dataDisk/htdocs/gallery/image_proxy.php:148
Stack trace:
#0 /dataDisk/htdocs/gallery/image_proxy.php(148): Imagick->readImageBlob()
#1 /dataDisk/htdocs/gallery/image_proxy.php(101): scaleImage()
#2 /dataDisk/htdocs/gallery/image_proxy.php(65): fetchUrl()
#3 /dataDisk/htdocs/gallery/image_proxy.php(14): getData()
#4 {main}
  thrown in /dataDisk/htdocs/gallery/image_proxy.php on line 148

Maybe you consider fixing this by handling such exception and cleanup the cache.