Low Database Performance about unifi

[goofball222]

I'm not personally seeing any performance issues using an external MongoDB in a separate container on the same host. It appears to be working multi-threaded/multi-connection based on the logs from the DB container:

2019-08-30T17:03:04.272+0000 I  NETWORK  [listener] connection accepted from X.X.X.X:57744 #9 (4 connections now open)
2019-08-30T17:03:14.972+0000 I  NETWORK  [listener] connection accepted from X.X.X.X:57748 #10 (5 connections now open)
2019-08-30T17:04:04.975+0000 I  NETWORK  [listener] connection accepted from X.X.X.X:57750 #11 (6 connections now open)
2019-09-01T22:35:26.998+0000 I  NETWORK  [listener] connection accepted from X.X.X.X:48290 #12 (7 connections now open)
2019-09-02T07:00:00.384+0000 I  NETWORK  [listener] connection accepted from X.X.X.X:43440 #13 (8 connections now open)
2019-09-02T07:00:00.521+0000 I  NETWORK  [listener] connection accepted from X.X.X.X:43442 #14 (9 connections now open)
2019-09-02T07:00:05.675+0000 I  NETWORK  [conn14] end connection X.X.X.X:43442 (8 connections now open)
2019-09-02T07:00:05.675+0000 I  NETWORK  [conn13] end connection X.X.X.X:43440 (7 connections now open)
2019-09-02T07:00:05.687+0000 I  NETWORK  [listener] connection accepted from X.X.X.X:43444 #15 (8 connections now open)
2019-09-02T07:00:05.738+0000 I  NETWORK  [listener] connection accepted from X.X.X.X:43446 #16 (9 connections now open)
2019-09-02T07:01:13.872+0000 I  NETWORK  [conn16] end connection X.X.X.X:43446 (8 connections now open)
2019-09-02T07:01:13.872+0000 I  NETWORK  [conn15] end connection X.X.X.X:43444 (7 connections now open)

I haven't tried it with cross-host mappings or anything like that, so not sure there. I believe there are folks on AWS and Azure using those hosted MongoDB products with UniFi without reported trouble.

The DB externalization stuff is all built into the UniFi software, so I don't really have any insight into it.

One possible problem I can think of would be the networking speeds or errors between the container host and the MongoDB host. If they're on the same LAN is everything running at least 1000mbit full duplex, no errors? Might try running iperf test between the two to see if something is going on there. If they're not on the same LAN it raises a whole slew of possible trouble-points & headaches...

[hemsleyk]

Yeah 1000mbps. The management platform coats everything between hosts in ipsec though so that might be it. Different LAN but same building. VMs are stored on SSD. May try some cross host transfer tests but they have historically run well.

What mongodb version and thread model are you running? I'm 3.4 on synchronous. Wondering if 3.6 with adaptive may be a fix?

Actual runtime performance is great. Just as smooth as local. It's only restores that are slow. Just over 100 devices on 20 sites with two weeks of 5 minute data so maybe I am torturing the controller a little. Backups are ~3GB.

[goofball222]

I've run with mongo 3.4 up through now 4.2.0 after forcing the UniFi DBs to 4.0+ compat mode, however that latter part is way outside of what UBNT supports. I don't recommend it for production.

My single-host setup ran fine on both mongo 3.6 and 4.0 with the UBNT supported mongo 3.6 compatibility mode. Everything is using the 3.2+ default WiredTiger DB storage. I haven't adjusted any other default mongodb config options, so whatever out of box threading model is what I'm running.

[hemsleyk]

So what I think is causing my DB to wipe every time the host reboots is the cert not being volumed correctly. Since its a self-signed I wasn't caring but I think when it has to remake the cert it invokes a drop of the entire DB. Which ironically doesn't take long, its reloading after that does ... voluming the cert will cause it to wipe again under the management platform I'm using so will test on synchronous and see what happens.

[goofball222]

Looking at entrypoint-functions.sh, I'm not sure where the cert function would cause the full UniFi DB to be dropped, it only makes changes against two files - ${DATADIR}/keystore and ${CERTDIR}/unificert.sha256.

I'd SWAG that the DB being wiped indicates the data volume isn't mapped somewhere persistent between container resets, or it doesn't have a valid system.properties file in it, or can't write one out. Out of the box a UniFi system has no system.properties file so it goes into setup mode. For this image there's a system.properties.default containing settings needed to function as a container. It doesn't have a uuid=(random-uuid), or a is_default=false entry. Without those present the controller software starts into setup mode and will pave over any databases it connects to. Once setup or restore is complete it writes those two values, and others out to the system.properties file under ${DATADIR}.

Declared volume paths in the container:
${CERTDIR} - /usr/lib/unifi/cert - not required, custom certificates storage
${DATADIR} - /usr/lib/unifi/data - required, without this mapped persistently to a known volume/location the software will reset to defaults. This also stores things like guest portal customizations and site-specific settings
${LOGDIR} - /usr/lib/unifi/logs - not required, controller logs storage