All Suricata alerts are of category "Unknown Classtype" on CentOS v8.2 and Ubuntu 18.04 about build-suricata

[philrz]

On my Ubuntu 18.04 VM, I thought to try unpacking the suricata-v5.0.3-brim18.linux-amd64.zip artifact into its own standalone directory, ran the suricataupdater, and then fed the same pcap through it. In this case the alerts were categorized.

$ zq -t 'event_type=alert | count() by alert.category' eve.json 
#0:record[alert:record[category:string],count:uint64]
0:[[Generic Protocol Command Decode;]263255;]
0:[[Attempted Administrator Privilege Gain;]12;]
0:[[Unknown Traffic;]43;]
0:[[Potentially Bad Traffic;]2;]
0:[[Misc activity;]2;]
0:[[Not Suspicious Traffic;]14;]

So maybe there's some problem there with the scripts that run the modified Suricata configs in the Brim app's data directory?

Attempting the same on my CentOS v8.2 VM, I saw the same cert problem that was reflected in the zqd log when Brim launched the suricataupdater:

$ ./suricataupdater 
24/11/2020 -- 21:32:35 - <Info> -- Loading /home/phil/suricata/var/lib/suricata/update.yaml
24/11/2020 -- 21:32:35 - <Info> -- Found Suricata version 5.0.3 at /home/phil/suricata/bin/suricata.
24/11/2020 -- 21:32:35 - <Info> -- Loading /home/phil/suricata/brim-conf.yaml
24/11/2020 -- 21:32:35 - <Info> -- No sources configured, will use Emerging Threats Open
24/11/2020 -- 21:32:35 - <Info> -- Checking https://rules.emergingthreats.net/open/suricata-5.0.3/emerging.rules.tar.gz.md5.
24/11/2020 -- 21:32:35 - <Warning> -- Failed to check remote checksum: <urlopen error [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed (_ssl.c:852)>
24/11/2020 -- 21:32:35 - <Info> -- Fetching https://rules.emergingthreats.net/open/suricata-5.0.3/emerging.rules.tar.gz.
24/11/2020 -- 21:32:35 - <Warning> -- Failed to fetch https://rules.emergingthreats.net/open/suricata-5.0.3/emerging.rules.tar.gz, will use latest cached version: <urlopen error [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed (_ssl.c:852)>
24/11/2020 -- 21:32:35 - <Info> -- Loading distribution rule file /home/phil/suricata/share/suricata/rules/app-layer-events.rules
24/11/2020 -- 21:32:35 - <Info> -- Loading distribution rule file /home/phil/suricata/share/suricata/rules/decoder-events.rules
24/11/2020 -- 21:32:35 - <Info> -- Loading distribution rule file /home/phil/suricata/share/suricata/rules/dhcp-events.rules
24/11/2020 -- 21:32:35 - <Info> -- Loading distribution rule file /home/phil/suricata/share/suricata/rules/dnp3-events.rules
24/11/2020 -- 21:32:35 - <Info> -- Loading distribution rule file /home/phil/suricata/share/suricata/rules/dns-events.rules
24/11/2020 -- 21:32:35 - <Info> -- Loading distribution rule file /home/phil/suricata/share/suricata/rules/files.rules
24/11/2020 -- 21:32:35 - <Info> -- Loading distribution rule file /home/phil/suricata/share/suricata/rules/http-events.rules
24/11/2020 -- 21:32:35 - <Info> -- Loading distribution rule file /home/phil/suricata/share/suricata/rules/ipsec-events.rules
24/11/2020 -- 21:32:35 - <Info> -- Loading distribution rule file /home/phil/suricata/share/suricata/rules/kerberos-events.rules
24/11/2020 -- 21:32:35 - <Info> -- Loading distribution rule file /home/phil/suricata/share/suricata/rules/modbus-events.rules
24/11/2020 -- 21:32:35 - <Info> -- Loading distribution rule file /home/phil/suricata/share/suricata/rules/nfs-events.rules
24/11/2020 -- 21:32:35 - <Info> -- Loading distribution rule file /home/phil/suricata/share/suricata/rules/ntp-events.rules
24/11/2020 -- 21:32:35 - <Info> -- Loading distribution rule file /home/phil/suricata/share/suricata/rules/smb-events.rules
24/11/2020 -- 21:32:35 - <Info> -- Loading distribution rule file /home/phil/suricata/share/suricata/rules/smtp-events.rules
24/11/2020 -- 21:32:35 - <Info> -- Loading distribution rule file /home/phil/suricata/share/suricata/rules/stream-events.rules
24/11/2020 -- 21:32:35 - <Info> -- Loading distribution rule file /home/phil/suricata/share/suricata/rules/tls-events.rules
24/11/2020 -- 21:32:35 - <Info> -- Ignoring file rules/emerging-deleted.rules
24/11/2020 -- 21:32:37 - <Info> -- Loaded 28525 rules.
24/11/2020 -- 21:32:37 - <Info> -- Disabled 0 rules.
24/11/2020 -- 21:32:37 - <Info> -- Enabled 0 rules.
24/11/2020 -- 21:32:37 - <Info> -- Modified 0 rules.
24/11/2020 -- 21:32:37 - <Info> -- Dropped 0 rules.
24/11/2020 -- 21:32:37 - <Info> -- Enabled 145 rules for flowbit dependencies.
24/11/2020 -- 21:32:37 - <Info> -- Backing up current rules.
24/11/2020 -- 21:32:39 - <Info> -- Writing rules to /home/phil/suricata/var/lib/suricata/rules/suricata.rules: total: 28525; enabled: 21276; added: 0; removed 0; modified: 14
24/11/2020 -- 21:32:39 - <Info> -- Writing /home/phil/suricata/var/lib/suricata/rules/classification.config
24/11/2020 -- 21:32:39 - <Info> -- Skipping test, disabled by configuration.
24/11/2020 -- 21:32:39 - <Info> -- Done.

Even before I ran the suricataupdater though, I now saw that the rules file as already large right as it came out of the ZIP, and I do get the categorized alerts after I run the pcap through.

$ zq -t 'event_type=alert | count() by alert.category' eve.json 
#0:record[alert:record[category:string],count:uint64]
0:[[Unknown Traffic;]43;]
0:[[Potentially Bad Traffic;]2;]
0:[[Misc activity;]2;]
0:[[Not Suspicious Traffic;]14;]
0:[[Generic Protocol Command Decode;]263255;]
0:[[Attempted Administrator Privilege Gain;]12;]

[philrz]

Ok, I figured out the root cause of one of the problems! I stared at the command line that suricata was being launched with and went to start examining the config files to see if any of them looked broken, only to find that the permissions wouldn't allow me to view them as a non-root user:

$ ls -l /usr/lib/brim/resources/app/zdeps/suricata/etc/suricata/classification.config
-rw------- 1 root root 4258 Nov 24 12:47 /usr/lib/brim/resources/app/zdeps/suricata/etc/suricata/classification.config

$ ls -l /usr/lib/brim/resources/app/zdeps/suricata/etc/suricata/reference.config
-rw------- 1 root root 1375 Nov 24 12:47 /usr/lib/brim/resources/app/zdeps/suricata/etc/suricata/reference.config

$ ls -l /usr/lib/brim/resources/app/zdeps/suricata/etc/suricata/threshold.config
-rw------- 1 root root 1644 Nov 24 12:47 /usr/lib/brim/resources/app/zdeps/suricata/etc/suricata/threshold.config

Once I gave those a chmod go+r, now after I run the pcap through Brim, I can see the categories for the generated alerts.

It looks like the suricata.yaml is also getting installed with 600 permissions, but I'm not sure if that matters. I didn't need to touch it to get the output shown above.

$ ls -l /usr/lib/brim/resources/app/zdeps/suricata/etc/suricata/suricata.yaml 
-rw------- 1 root root 70659 Nov 24 12:47 /usr/lib/brim/resources/app/zdeps/suricata/etc/suricata/suricata.yaml

[philrz]

I've verified the fix in #45 via test artifact brim_amd_rc-v0.20.0-suricata20.deb that's based on Brim commit 0481863.

Importing the same pcap as before in my Ubuntu 18.04 desktop, I can now see the alert categories where I couldn't before.

On my CentOS v8.2 desktop, we can see the categories have changed, but they're still all falling into one bucket.

Since the SSL cert problem is still present and I can see the rules file under $HOME/.config/Brim/suricata/rules are still at their small sizes:

$ ls -al
total 64
drwxrwx---. 2 phil phil    57 Nov 25 18:38 .
drwxrwxr-x. 4 phil phil    78 Nov 25 18:38 ..
-rw-rw-r--. 1 phil phil     0 Nov 25 18:38 classification.config
-rw-rw-r--. 1 phil phil 65096 Nov 25 18:38 suricata.rules

...I'm prepared to assume this is a side effect of not having all the Emerging Threats rules in play, hence will revisit when we have the fix in for that part. Thanks for the progress thus far, @henridf!

[henridf]

The SSL: CERTIFICATE_VERIFY_FAILED error happens because the pyinstaller-created suricata-update binary is hard-coded with the CA paths of the system where pyinstaller runs. (I'm assuming that this is somewhere in one of the many .so's that it packages).

These paths are returned by SSLContext.set_default_verify_paths as part of the python http client creation. So on any platform where the CA file is in a different location, verification fails.

[philrz]

Something's still not working right on CentOS v8.2. With the latest artifact:

$ unzip suricata-v5.0.3-brim24.linux-amd64.zip
...

$ cd suricata/
$ ./suricataupdater 
26/11/2020 -- 14:39:12 - <Info> -- Loading /home/phil/suricata/var/lib/suricata/update.yaml
26/11/2020 -- 14:39:12 - <Info> -- Found Suricata version 5.0.3 at /home/phil/suricata/bin/suricata.
26/11/2020 -- 14:39:12 - <Info> -- Loading /home/phil/suricata/brim-conf.yaml
26/11/2020 -- 14:39:12 - <Info> -- Disabling rules for protocol modbus
26/11/2020 -- 14:39:12 - <Info> -- Disabling rules for protocol dnp3
26/11/2020 -- 14:39:12 - <Info> -- Disabling rules for protocol enip
26/11/2020 -- 14:39:12 - <Info> -- No sources configured, will use Emerging Threats Open
26/11/2020 -- 14:39:12 - <Info> -- Checking https://rules.emergingthreats.net/open/suricata-5.0.3/emerging.rules.tar.gz.md5.
26/11/2020 -- 14:39:12 - <Warning> -- Failed to check remote checksum: <urlopen error [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed (_ssl.c:852)>
26/11/2020 -- 14:39:12 - <Info> -- Fetching https://rules.emergingthreats.net/open/suricata-5.0.3/emerging.rules.tar.gz.
26/11/2020 -- 14:39:12 - <Warning> -- Failed to fetch https://rules.emergingthreats.net/open/suricata-5.0.3/emerging.rules.tar.gz, will use latest cached version: <urlopen error [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed (_ssl.c:852)>
...

[philrz]

I've now verified the final fix on CentOS v8.2 using draft release artifact suricata-v5.0.3-brim26.linux-amd64.zip.

Here on a freshly-made CentOS v8.2 VM, I'm able to run the suricataupdater without complaint, run the wrccdc pcap through the suricatarunner, and get all the expected categories for the generated alerts.

$ ./suricataupdater 
29/11/2020 -- 15:22:23 - <Info> -- Loading /home/phil/home/runner/suricata/var/lib/suricata/update.yaml
29/11/2020 -- 15:22:23 - <Info> -- Found Suricata version 5.0.3 at /home/phil/home/runner/suricata/bin/suricata.
29/11/2020 -- 15:22:23 - <Info> -- Loading /home/phil/home/runner/suricata/brim-conf.yaml
29/11/2020 -- 15:22:23 - <Info> -- Disabling rules for protocol modbus
29/11/2020 -- 15:22:23 - <Info> -- Disabling rules for protocol dnp3
29/11/2020 -- 15:22:23 - <Info> -- Disabling rules for protocol enip
29/11/2020 -- 15:22:23 - <Info> -- No sources configured, will use Emerging Threats Open
29/11/2020 -- 15:22:23 - <Info> -- Checking https://rules.emergingthreats.net/open/suricata-5.0.3/emerging.rules.tar.gz.md5.
29/11/2020 -- 15:22:24 - <Info> -- Remote checksum has not changed. Not fetching.
29/11/2020 -- 15:22:24 - <Info> -- Loading distribution rule file /home/phil/home/runner/suricata/share/suricata/rules/app-layer-events.rules
29/11/2020 -- 15:22:24 - <Info> -- Loading distribution rule file /home/phil/home/runner/suricata/share/suricata/rules/decoder-events.rules
29/11/2020 -- 15:22:24 - <Info> -- Loading distribution rule file /home/phil/home/runner/suricata/share/suricata/rules/dhcp-events.rules
29/11/2020 -- 15:22:24 - <Info> -- Loading distribution rule file /home/phil/home/runner/suricata/share/suricata/rules/dnp3-events.rules
29/11/2020 -- 15:22:24 - <Info> -- Loading distribution rule file /home/phil/home/runner/suricata/share/suricata/rules/dns-events.rules
29/11/2020 -- 15:22:24 - <Info> -- Loading distribution rule file /home/phil/home/runner/suricata/share/suricata/rules/files.rules
29/11/2020 -- 15:22:24 - <Info> -- Loading distribution rule file /home/phil/home/runner/suricata/share/suricata/rules/http-events.rules
29/11/2020 -- 15:22:24 - <Info> -- Loading distribution rule file /home/phil/home/runner/suricata/share/suricata/rules/ipsec-events.rules
29/11/2020 -- 15:22:24 - <Info> -- Loading distribution rule file /home/phil/home/runner/suricata/share/suricata/rules/kerberos-events.rules
29/11/2020 -- 15:22:24 - <Info> -- Loading distribution rule file /home/phil/home/runner/suricata/share/suricata/rules/modbus-events.rules
29/11/2020 -- 15:22:24 - <Info> -- Loading distribution rule file /home/phil/home/runner/suricata/share/suricata/rules/nfs-events.rules
29/11/2020 -- 15:22:24 - <Info> -- Loading distribution rule file /home/phil/home/runner/suricata/share/suricata/rules/ntp-events.rules
29/11/2020 -- 15:22:24 - <Info> -- Loading distribution rule file /home/phil/home/runner/suricata/share/suricata/rules/smb-events.rules
29/11/2020 -- 15:22:24 - <Info> -- Loading distribution rule file /home/phil/home/runner/suricata/share/suricata/rules/smtp-events.rules
29/11/2020 -- 15:22:24 - <Info> -- Loading distribution rule file /home/phil/home/runner/suricata/share/suricata/rules/stream-events.rules
29/11/2020 -- 15:22:24 - <Info> -- Loading distribution rule file /home/phil/home/runner/suricata/share/suricata/rules/tls-events.rules
29/11/2020 -- 15:22:24 - <Info> -- Ignoring file rules/emerging-deleted.rules
29/11/2020 -- 15:22:25 - <Info> -- Loaded 28569 rules.
29/11/2020 -- 15:22:25 - <Info> -- Disabled 14 rules.
29/11/2020 -- 15:22:25 - <Info> -- Enabled 0 rules.
29/11/2020 -- 15:22:25 - <Info> -- Modified 0 rules.
29/11/2020 -- 15:22:25 - <Info> -- Dropped 0 rules.
29/11/2020 -- 15:22:25 - <Info> -- Enabled 145 rules for flowbit dependencies.
29/11/2020 -- 15:22:25 - <Info> -- Backing up current rules.
29/11/2020 -- 15:22:27 - <Info> -- Writing rules to /home/phil/home/runner/suricata/var/lib/suricata/rules/suricata.rules: total: 28569; enabled: 21186; added: 0; removed 0; modified: 0
29/11/2020 -- 15:22:27 - <Info> -- Writing /home/phil/home/runner/suricata/var/lib/suricata/rules/classification.config
29/11/2020 -- 15:22:27 - <Info> -- No changes detected, exiting.

$ cat ~/wrccdc.pcap | ./suricatarunner 
29/11/2020 -- 15:22:39 - <Notice> - This is Suricata version 5.0.3 RELEASE running in USER mode
29/11/2020 -- 15:22:39 - <Info> - CPUs/cores online: 4
29/11/2020 -- 15:22:39 - <Info> - No 'host-mode': suricata is in IDS mode, using default setting 'sniffer-only'
29/11/2020 -- 15:22:39 - <Info> - eve-log output device (regular) initialized: eve.json
29/11/2020 -- 15:22:39 - <Info> - 1 rule files processed. 21186 rules successfully loaded, 0 rules failed
29/11/2020 -- 15:22:39 - <Info> - Threshold config parsed: 0 rule(s) found
29/11/2020 -- 15:22:40 - <Info> - 21189 signatures processed. 1534 are IP-only rules, 3808 are inspecting packet payload, 15618 inspect application layer, 103 are decoder event only
29/11/2020 -- 15:22:45 - <Notice> - all 1 packet processing threads, 2 management threads initialized, engine started.
29/11/2020 -- 15:22:45 - <Info> - Starting file run for -
29/11/2020 -- 15:22:45 - <Info> - No packets with invalid checksum, assuming checksum offloading is NOT used
29/11/2020 -- 15:22:54 - <Info> - Flow emergency mode over, back to normal... unsetting FLOW_EMERGENCY bit (ts.tv_sec: 1521835199, ts.tv_usec:533645) flow_spare_q status(): 45% flows at the queue
29/11/2020 -- 15:22:55 - <Info> - Flow emergency mode over, back to normal... unsetting FLOW_EMERGENCY bit (ts.tv_sec: 1521835200, ts.tv_usec:512592) flow_spare_q status(): 32% flows at the queue
29/11/2020 -- 15:22:56 - <Info> - Flow emergency mode over, back to normal... unsetting FLOW_EMERGENCY bit (ts.tv_sec: 1521835202, ts.tv_usec:237823) flow_spare_q status(): 33% flows at the queue
29/11/2020 -- 15:22:57 - <Info> - Flow emergency mode over, back to normal... unsetting FLOW_EMERGENCY bit (ts.tv_sec: 1521835204, ts.tv_usec:142696) flow_spare_q status(): 32% flows at the queue
29/11/2020 -- 15:22:58 - <Info> - Flow emergency mode over, back to normal... unsetting FLOW_EMERGENCY bit (ts.tv_sec: 1521835205, ts.tv_usec:954171) flow_spare_q status(): 70% flows at the queue
29/11/2020 -- 15:22:59 - <Info> - Flow emergency mode over, back to normal... unsetting FLOW_EMERGENCY bit (ts.tv_sec: 1521835207, ts.tv_usec:363024) flow_spare_q status(): 45% flows at the queue
29/11/2020 -- 15:23:00 - <Info> - Flow emergency mode over, back to normal... unsetting FLOW_EMERGENCY bit (ts.tv_sec: 1521835208, ts.tv_usec:398921) flow_spare_q status(): 44% flows at the queue
29/11/2020 -- 15:23:01 - <Info> - Flow emergency mode over, back to normal... unsetting FLOW_EMERGENCY bit (ts.tv_sec: 1521835209, ts.tv_usec:85686) flow_spare_q status(): 40% flows at the queue
29/11/2020 -- 15:23:03 - <Info> - Flow emergency mode over, back to normal... unsetting FLOW_EMERGENCY bit (ts.tv_sec: 1521835210, ts.tv_usec:285238) flow_spare_q status(): 51% flows at the queue
29/11/2020 -- 15:23:05 - <Info> - Flow emergency mode over, back to normal... unsetting FLOW_EMERGENCY bit (ts.tv_sec: 1521835211, ts.tv_usec:345437) flow_spare_q status(): 39% flows at the queue
29/11/2020 -- 15:23:06 - <Info> - Flow emergency mode over, back to normal... unsetting FLOW_EMERGENCY bit (ts.tv_sec: 1521835212, ts.tv_usec:131433) flow_spare_q status(): 49% flows at the queue
29/11/2020 -- 15:23:08 - <Info> - Flow emergency mode over, back to normal... unsetting FLOW_EMERGENCY bit (ts.tv_sec: 1521835213, ts.tv_usec:529673) flow_spare_q status(): 31% flows at the queue
29/11/2020 -- 15:23:09 - <Info> - Flow emergency mode over, back to normal... unsetting FLOW_EMERGENCY bit (ts.tv_sec: 1521835214, ts.tv_usec:325613) flow_spare_q status(): 48% flows at the queue
29/11/2020 -- 15:23:09 - <Info> - pcap file - end of file reached (pcap err code 0)
29/11/2020 -- 15:23:09 - <Notice> - Signal Received.  Stopping engine.
29/11/2020 -- 15:23:09 - <Info> - time elapsed 23.312s
29/11/2020 -- 15:23:09 - <Notice> - Pcap-file module read 1 files, 1650919 packets, 473586644 bytes
29/11/2020 -- 15:23:09 - <Info> - Alerts: 0
29/11/2020 -- 15:23:09 - <Info> - cleaning up signature grouping structure... complete

$ zq -t 'event_type=alert | count() by alert.category' eve.json
#0:record[alert:record[category:string],count:uint64]
0:[[Not Suspicious Traffic;]14;]
0:[[Generic Protocol Command Decode;]263255;]
0:[[Attempted Administrator Privilege Gain;]12;]
0:[[Unknown Traffic;]43;]
0:[[Potentially Bad Traffic;]2;]
0:[[Misc activity;]2;]

Thanks @henridf!