brimdata / build-suricata Goto Github PK

View Code? Open in Web Editor NEW

7.0 7.0 2.0 103 KB

Build Suricata for packaging with Brim

Shell 39.23% Go 60.77%

build-suricata's People

Contributors

Stargazers

Watchers

Forkers

liyanvip johnny-hui

build-suricata's Issues

Submit "-r -" change to upstream Suricata

brimdata/suricata@b4d6ca7 (allows passing "-" to suricata to indicate "read from stdin") seems generally useful. We should submit it as a PR for consideration to upstream.

Move linux build to libpcap 1.9.1

#3 (review)

Community ID not populated on Windows

Testing with artifact Brim-Setup-rc-v0.20.0-suricata18.exe on Windows, we see Suricata alerts are being generated, but the community_id field is consistently null. I think we already knew this, but I hadn't yet opened an official issue to track it.

Excess "runner" paths in zip artifact bundle

When verifying #28 I found the artifacts seemed to build ok, but the paths inside the ZIP files have some excess length that reveal their origins on GitHub Actions runners. Since this would not make them a drop-in replacement for the artifacts we were generating in the storage bucket phase, I assume we'll want to adjust this.

$ unzip -l suricata-v5.0.3-brim26.darwin-amd64.zip | head -5
Archive:  suricata-v5.0.3-brim26.darwin-amd64.zip
  Length      Date    Time    Name
---------  ---------- -----   ----
        0  11-29-2020 11:21   Users/runner/suricata/
      897  11-29-2020 11:21   Users/runner/suricata/suricatarunner

$ unzip -l suricata-v5.0.3-brim26.linux-amd64.zip | head -5
Archive:  suricata-v5.0.3-brim26.linux-amd64.zip
  Length      Date    Time    Name
---------  ---------- -----   ----
        0  11-29-2020 11:19   home/runner/suricata/
    13994  11-29-2020 11:19   home/runner/suricata/brim-conf.yaml

$ unzip -l suricata-v5.0.3-brim26.windows-amd64.zip | head -5
Archive:  suricata-v5.0.3-brim26.windows-amd64.zip
  Length      Date    Time    Name
---------  ---------- -----   ----
        0  11-29-2020 11:33   home/runneradmin/suricata/
        0  11-29-2020 11:33   home/runneradmin/suricata/bin/

Move release packages to Github

Releases are currently stored in a public Google Cloud bucket. We should move them to be on the GitHub Releases page for this repo (like we do for zeek).

Build Suricata binary for Ubuntu

Suricata outputs [ERRCODE: SC_ERR_INVALID_SIGNATURE(39)] errors

Suricata currently outputs errors about protocol modbus being disabled. Logs are still processed and output ok, but we should fix the configuration so it doesn't.

0/11/2020 -- 18:46:01 - <Notice> - This is Suricata version 5.0.3 RELEASE running in USER mode
20/11/2020 -- 18:46:01 - <Info> - CPUs/cores online: 8
20/11/2020 -- 18:46:01 - <Info> - No 'host-mode': suricata is in IDS mode, using default setting 'sniffer-only'
20/11/2020 -- 18:46:01 - <Info> - eve-log output device (regular) initialized: eve.json
20/11/2020 -- 18:46:01 - <Error> - [ERRCODE: SC_ERR_INVALID_SIGNATURE(39)] - protocol dnp3 is disabled
20/11/2020 -- 18:46:01 - <Error> - [ERRCODE: SC_ERR_INVALID_SIGNATURE(39)] - error parsing signature "alert dnp3 any any -> any any (msg:"SURICATA DNP3 Request flood detected";        app-layer-event:dnp3.flooded; classtype:protocol-command-decode; sid:2270000; rev:2;)" from file /Users/henridf/work/brim/zq/bin/suricata-v5.0.3-brim13/var/lib/suricata/rules/suricata.rules at line 123
20/11/2020 -- 18:46:01 - <Error> - [ERRCODE: SC_ERR_INVALID_SIGNATURE(39)] - protocol dnp3 is disabled
20/11/2020 -- 18:46:01 - <Error> - [ERRCODE: SC_ERR_INVALID_SIGNATURE(39)] - error parsing signature "alert dnp3 any any -> any any (msg:"SURICATA DNP3 Length too small";        app-layer-event:dnp3.len_too_small; classtype:protocol-command-decode; sid:2270001; rev:3;)" from file /Users/henridf/work/brim/zq/bin/suricata-v5.0.3-brim13/var/lib/suricata/rules/suricata.rules at line 124
20/11/2020 -- 18:46:01 - <Error> - [ERRCODE: SC_ERR_INVALID_SIGNATURE(39)] - protocol dnp3 is disabled
20/11/2020 -- 18:46:01 - <Error> - [ERRCODE: SC_ERR_INVALID_SIGNATURE(39)] - error parsing signature "alert dnp3 any any -> any any (msg:"SURICATA DNP3 Bad link CRC";        app-layer-event:dnp3.bad_link_crc; classtype:protocol-command-decode; sid:2270002; rev:2;)" from file /Users/henridf/work/brim/zq/bin/suricata-v5.0.3-brim13/var/lib/suricata/rules/suricata.rules at line 125
20/11/2020 -- 18:46:01 - <Error> - [ERRCODE: SC_ERR_INVALID_SIGNATURE(39)] - protocol dnp3 is disabled
20/11/2020 -- 18:46:01 - <Error> - [ERRCODE: SC_ERR_INVALID_SIGNATURE(39)] - error parsing signature "alert dnp3 any any -> any any (msg:"SURICATA DNP3 Bad transport CRC";        app-layer-event:dnp3.bad_transport_crc; classtype:protocol-command-decode; sid:2270003; rev:2;)" from file /Users/henridf/work/brim/zq/bin/suricata-v5.0.3-brim13/var/lib/suricata/rules/suricata.rules at line 126
20/11/2020 -- 18:46:01 - <Error> - [ERRCODE: SC_ERR_INVALID_SIGNATURE(39)] - protocol dnp3 is disabled
20/11/2020 -- 18:46:01 - <Error> - [ERRCODE: SC_ERR_INVALID_SIGNATURE(39)] - error parsing signature "alert dnp3 any any -> any any (msg:"SURICATA DNP3 Unknown object";        app-layer-event:dnp3.unknown_object; classtype:protocol-command-decode; sid:2270004; rev:2;)" from file /Users/henridf/work/brim/zq/bin/suricata-v5.0.3-brim13/var/lib/suricata/rules/suricata.rules at line 127
20/11/2020 -- 18:46:01 - <Error> - [ERRCODE: SC_ERR_INVALID_SIGNATURE(39)] - protocol modbus is disabled
20/11/2020 -- 18:46:01 - <Error> - [ERRCODE: SC_ERR_INVALID_SIGNATURE(39)] - error parsing signature "alert modbus any any -> any any (msg:"SURICATA Modbus invalid Protocol version"; app-layer-event:modbus.invalid_protocol_id; classtype:protocol-command-decode; sid:2250001; rev:2;)" from file /Users/henridf/work/brim/zq/bin/suricata-v5.0.3-brim13/var/lib/suricata/rules/suricata.rules at line 224
20/11/2020 -- 18:46:01 - <Error> - [ERRCODE: SC_ERR_INVALID_SIGNATURE(39)] - protocol modbus is disabled
20/11/2020 -- 18:46:01 - <Error> - [ERRCODE: SC_ERR_INVALID_SIGNATURE(39)] - error parsing signature "alert modbus any any -> any any (msg:"SURICATA Modbus unsolicited response"; app-layer-event:modbus.unsolicited_response; classtype:protocol-command-decode; sid:2250002; rev:2;)" from file /Users/henridf/work/brim/zq/bin/suricata-v5.0.3-brim13/var/lib/suricata/rules/suricata.rules at line 225
20/11/2020 -- 18:46:01 - <Error> - [ERRCODE: SC_ERR_INVALID_SIGNATURE(39)] - protocol modbus is disabled
20/11/2020 -- 18:46:01 - <Error> - [ERRCODE: SC_ERR_INVALID_SIGNATURE(39)] - error parsing signature "alert modbus any any -> any any (msg:"SURICATA Modbus invalid Length"; app-layer-event:modbus.invalid_length; classtype:protocol-command-decode; sid:2250003; rev:2;)" from file /Users/henridf/work/brim/zq/bin/suricata-v5.0.3-brim13/var/lib/suricata/rules/suricata.rules at line 226
20/11/2020 -- 18:46:01 - <Error> - [ERRCODE: SC_ERR_INVALID_SIGNATURE(39)] - protocol modbus is disabled
20/11/2020 -- 18:46:01 - <Error> - [ERRCODE: SC_ERR_INVALID_SIGNATURE(39)] - error parsing signature "alert modbus any any -> any any (msg:"SURICATA Modbus invalid Unit Identifier"; app-layer-event:modbus.invalid_unit_identifier; classtype:protocol-command-decode; sid:2250004; rev:2;)" from file /Users/henridf/work/brim/zq/bin/suricata-v5.0.3-brim13/var/lib/suricata/rules/suricata.rules at line 227
20/11/2020 -- 18:46:01 - <Error> - [ERRCODE: SC_ERR_INVALID_SIGNATURE(39)] - protocol modbus is disabled
20/11/2020 -- 18:46:01 - <Error> - [ERRCODE: SC_ERR_INVALID_SIGNATURE(39)] - error parsing signature "alert modbus any any -> any any (msg:"SURICATA Modbus invalid Function code"; app-layer-event:modbus.invalid_function_code; classtype:protocol-command-decode; sid:2250005; rev:2;)" from file /Users/henridf/work/brim/zq/bin/suricata-v5.0.3-brim13/var/lib/suricata/rules/suricata.rules at line 228
20/11/2020 -- 18:46:01 - <Error> - [ERRCODE: SC_ERR_INVALID_SIGNATURE(39)] - protocol modbus is disabled
20/11/2020 -- 18:46:01 - <Error> - [ERRCODE: SC_ERR_INVALID_SIGNATURE(39)] - error parsing signature "alert modbus any any -> any any (msg:"SURICATA Modbus invalid Value"; app-layer-event:modbus.invalid_value; classtype:protocol-command-decode; sid:2250006; rev:2;)" from file /Users/henridf/work/brim/zq/bin/suricata-v5.0.3-brim13/var/lib/suricata/rules/suricata.rules at line 229
20/11/2020 -- 18:46:01 - <Error> - [ERRCODE: SC_ERR_INVALID_SIGNATURE(39)] - protocol modbus is disabled
20/11/2020 -- 18:46:01 - <Error> - [ERRCODE: SC_ERR_INVALID_SIGNATURE(39)] - error parsing signature "alert modbus any any -> any any (msg:"SURICATA Modbus Exception code invalid"; flow:to_client; app-layer-event:modbus.invalid_exception_code; classtype:protocol-command-decode; sid:2250007; rev:2;)" from file /Users/henridf/work/brim/zq/bin/suricata-v5.0.3-brim13/var/lib/suricata/rules/suricata.rules at line 230
20/11/2020 -- 18:46:01 - <Error> - [ERRCODE: SC_ERR_INVALID_SIGNATURE(39)] - protocol modbus is disabled
20/11/2020 -- 18:46:01 - <Error> - [ERRCODE: SC_ERR_INVALID_SIGNATURE(39)] - error parsing signature "alert modbus any any -> any any (msg:"SURICATA Modbus Data mismatch"; flow:to_client; app-layer-event:modbus.value_mismatch; classtype:protocol-command-decode; sid:2250008; rev:2;)" from file /Users/henridf/work/brim/zq/bin/suricata-v5.0.3-brim13/var/lib/suricata/rules/suricata.rules at line 231
20/11/2020 -- 18:46:01 - <Error> - [ERRCODE: SC_ERR_INVALID_SIGNATURE(39)] - protocol modbus is disabled
20/11/2020 -- 18:46:01 - <Error> - [ERRCODE: SC_ERR_INVALID_SIGNATURE(39)] - error parsing signature "alert modbus any any -> any any (msg:"SURICATA Modbus Request flood detected"; flow:to_server; app-layer-event:modbus.flooded; classtype:protocol-command-decode; sid:2250009; rev:2;)" from file /Users/henridf/work/brim/zq/bin/suricata-v5.0.3-brim13/var/lib/suricata/rules/suricata.rules at line 232
20/11/2020 -- 18:46:02 - <Info> - 1 rule files processed. 21336 rules successfully loaded, 14 rules failed
20/11/2020 -- 18:46:02 - <Info> - Threshold config parsed: 0 rule(s) found
20/11/2020 -- 18:46:02 - <Info> - 21339 signatures processed. 1453 are IP-only rules, 4007 are inspecting packet payload, 15650 inspect application layer, 103 are decoder event only
20/11/2020 -- 18:46:09 - <Notice> - all 1 packet processing threads, 2 management threads initialized, engine started.
20/11/2020 -- 18:46:09 - <Info> - Starting file run for -
20/11/2020 -- 18:46:09 - <Info> - No packets with invalid checksum, assuming checksum offloading is NOT used
20/11/2020 -- 18:46:09 - <Info> - pcap file - end of file reached (pcap err code 0)
20/11/2020 -- 18:46:09 - <Notice> - Signal Received.  Stopping engine.
20/11/2020 -- 18:46:09 - <Info> - time elapsed 0.056s
20/11/2020 -- 18:46:09 - <Notice> - Pcap-file module read 1 files, 2000 packets, 705670 bytes
20/11/2020 -- 18:46:09 - <Info> - Alerts: 0
20/11/2020 -- 18:46:09 - <Info> - cleaning up signature grouping structure... complete

Default checksum handling results in lack of alerts

Repro is with suricata-v5.0.3-brimpre2.

Consider the attached leak.pcap.gz, which I took by running tcpdump -i ens4 -w leak.pcap on a Linux VM on Google Cloud. While it was being gathered, I ran ifconfig curl.co, which is my favorite easy way to trigger a legit Suricata alert, since it flags it as an "Attempted Information Leak".

When run through our Suricata artifact, we see the SC_ERR_INVALID_CHECKSUM error message shown below, and the generated alerts are exclusively just grumping about this.

$ gzcat leak.pcap.gz | suricata/suricatarunner
13/1/2021 -- 17:48:09 - <Notice> - This is Suricata version 5.0.3 RELEASE running in USER mode
13/1/2021 -- 17:48:09 - <Info> - CPUs/cores online: 12
13/1/2021 -- 17:48:09 - <Info> - No 'host-mode': suricata is in IDS mode, using default setting 'sniffer-only'
13/1/2021 -- 17:48:09 - <Info> - eve-log output device (regular) initialized: eve.json
13/1/2021 -- 17:48:10 - <Info> - 1 rule files processed. 21244 rules successfully loaded, 0 rules failed
13/1/2021 -- 17:48:10 - <Info> - Threshold config parsed: 0 rule(s) found
13/1/2021 -- 17:48:10 - <Info> - 21247 signatures processed. 1594 are IP-only rules, 3803 are inspecting packet payload, 15621 inspect application layer, 103 are decoder event only
13/1/2021 -- 17:48:15 - <Notice> - all 1 packet processing threads, 2 management threads initialized, engine started.
13/1/2021 -- 17:48:15 - <Info> - Starting file run for -
13/1/2021 -- 17:48:15 - <Info> - pcap file - end of file reached (pcap err code 0)
13/1/2021 -- 17:48:15 - <Notice> - Signal Received.  Stopping engine.
13/1/2021 -- 17:48:15 - <Info> - time elapsed 0.020s
13/1/2021 -- 17:48:15 - <Warning> - [ERRCODE: SC_ERR_INVALID_CHECKSUM(11)] - 1/2th of packets have an invalid checksum, consider setting pcap-file.checksum-checks variable to no or use '-k none' option on command line.
13/1/2021 -- 17:48:15 - <Notice> - Pcap-file module read 1 files, 77 packets, 7615 bytes
13/1/2021 -- 17:48:15 - <Info> - Alerts: 0
13/1/2021 -- 17:48:15 - <Info> - cleaning up signature grouping structure... complete

$ zq -f ndjson 'count() by alert' eve.json | jq .
{
  "alert": {
    "action": "allowed",
    "category": "Generic Protocol Command Decode",
    "gid": 1,
    "rev": 2,
    "severity": 3,
    "signature": "SURICATA TCPv4 invalid checksum",
    "signature_id": 2200074
  },
  "count": 29
}
{
  "alert": {
    "action": "allowed",
    "category": "Generic Protocol Command Decode",
    "gid": 1,
    "rev": 2,
    "severity": 3,
    "signature": "SURICATA UDPv4 invalid checksum",
    "signature_id": 2200075
  },
  "count": 2
}

However, if I follow its advice and append the following config to suricata/brim-conf.yaml:

pcap-file:
  checksum-checks: no

The problem goes away.

$ gzcat leak.pcap.gz | suricata/suricatarunner
13/1/2021 -- 17:49:59 - <Notice> - This is Suricata version 5.0.3 RELEASE running in USER mode
13/1/2021 -- 17:49:59 - <Info> - CPUs/cores online: 12
13/1/2021 -- 17:49:59 - <Info> - No 'host-mode': suricata is in IDS mode, using default setting 'sniffer-only'
13/1/2021 -- 17:49:59 - <Info> - eve-log output device (regular) initialized: eve.json
13/1/2021 -- 17:50:00 - <Info> - 1 rule files processed. 21244 rules successfully loaded, 0 rules failed
13/1/2021 -- 17:50:00 - <Info> - Threshold config parsed: 0 rule(s) found
13/1/2021 -- 17:50:00 - <Info> - 21247 signatures processed. 1594 are IP-only rules, 3803 are inspecting packet payload, 15621 inspect application layer, 103 are decoder event only
13/1/2021 -- 17:50:04 - <Notice> - all 1 packet processing threads, 2 management threads initialized, engine started.
13/1/2021 -- 17:50:04 - <Info> - Starting file run for -
13/1/2021 -- 17:50:04 - <Info> - pcap file - end of file reached (pcap err code 0)
13/1/2021 -- 17:50:04 - <Notice> - Signal Received.  Stopping engine.
13/1/2021 -- 17:50:04 - <Info> - time elapsed 0.020s
13/1/2021 -- 17:50:04 - <Notice> - Pcap-file module read 1 files, 77 packets, 7615 bytes
13/1/2021 -- 17:50:04 - <Info> - Alerts: 0
13/1/2021 -- 17:50:05 - <Info> - cleaning up signature grouping structure... complete

$ zq -f ndjson 'count() by alert' eve.json | jq .
{
  "alert": {
    "action": "allowed",
    "category": "Attempted Information Leak",
    "gid": 1,
    "metadata": {
      "created_at": [
        "2011_06_14"
      ],
      "updated_at": [
        "2020_04_22"
      ]
    },
    "rev": 5,
    "severity": 2,
    "signature": "ET POLICY curl User-Agent Outbound",
    "signature_id": 2013028
  },
  "count": 1
}

We went through basically the equivalent problem on the Zeek side some time ago, which led us to include -C ("ignore checksums") in the command line options used by the runner when invoking Zeek, so doing a similar fix for Suricata seems a reasonable thing to do.

Build static binary for Linux

The Linux build from #5 depends on a number of packages being installed. We'd rather have a static binary (or at least "mostly static" in that it should be ok to dynamically link e.g. to libc as our zeek executable does).

Windows suricataupdater.exe failure: pyyaml is required

I'd spotted this in a previous test artifact I'd created while working on #44, but I've now reproduced it with the draft release artifact suricata-v5.0.3-brim26.windows-amd64.zip as well. I just unpacked it and then:

C:\Users\Phil\Downloads\suricata-v5.0.3-brim26.windows-amd64\home\runneradmin\suricata>.\suricataupdater.exe
error: pyyaml is required
2020/11/29 12:42:53 launchSuricata failed exit status 1

Solus Linux failures

A community user reported:

Hi, I need some help with suricata and brim. Looks like its not working on my installation
I'm on Solus Linux and had to copy a magic.mgc to /usr/share/file/ and /usr/share/misc/magic.mgc from ubuntu system to make it work
I thing this could be the problem
Because without this magic.mgc file:

can those magic files be part of the app instead ?

Indeed, it looks like we take this approach of bundling the magic file on macOS bundling today, so I expect we could do the same thing in Linux now that we know there's some distros that lack the file in the common location.

In addition to the magic file, additional testing has also revealed a problem with SSL certs, as attempts to run the suricataupdater on Solus failed to download the Emerging Threats rule set due to error SSL: CERTIFICATE_VERIFY_FAILED. We'd seen the same on CentOS, and to address that we added the following to the suricataupdater script:

ca_path="$(openssl version -d | cut -d ' ' -f 2)"
ca_path="${ca_path//\"}"

SSL_CERT_FILE="$ca_path/cert.pem" ...

However, on Solus Linux, that cert.pem file is not present in that location and my web searches did not turn up any hits on Solus Linux packages that include it. The problem seems to be specific to how Suricata Update works, though, because tools like curl and wget on Solus have no problem downloading the rules from the same URL that's failing with Suricata update.

We confirmed that we could make it work by manually copying over a cert.pem from a CentOS system, and a community user was able to make it work by pointing at a specific cert:

SSL_CERT_FILE=/etc/ssl/certs/DigiCert_Global_Root_CA.pem ./usr/lib/brim/resources/app/zdeps/suricata/bin/suricata-update

Finally, if we make this all smooth, then there'd still be the higher-level question of a Brim installer that works on Solus, as the package formats we currently create like .deb and .rpm are not supported. The community user that reported this issue was working around that problem by manually unpacking the .deb:

Download the .deb file;
Unpack the .deb file with ar x brim_amd64.deb, which usually has data.tar.xz, control.tar.xz and debian-binary (text file with package containing a number / version).
All I need is to extract data.tar.xz and run brim from the current usr/bin folder

As we went on to discuss, one of the approaches described in brimdata/zui#685 might be the way to go.

Windows build

Tidy up suricata package

The suricata packages bundle stuff that isn't needed. Clean this up.

Community ID not being generated on macOS/Linux

As discussed in brimdata/zui#1207, an issue related to the nss libraries is currently preventing the Community ID field from being populated in alert events on macOS.

I discovered a similar problem on CentOS 8 using a fresh-launched VM.

$ cat wrccdc.2018-03-23.010014000000000.pcap | ./suricatarunner 
/home/phil/suricata/bin/suricata: error while loading shared libraries: libnss3.so: cannot open shared object file: No such file or directory

FWIW I had a different failure on CentOS 7, but this is so ancient that perhaps we shouldn't pursue it. The Brim support statement sets expectations that we're not actively testing on versions older than CentOS 8.

$ cat wrccdc.pcap | ./suricatarunner 
/home/phil/suricata/bin/suricata: /lib64/libc.so.6: version `GLIBC_2.22' not found (required by /home/phil/suricata/bin/suricata)
/home/phil/suricata/bin/suricata: /lib64/libc.so.6: version `GLIBC_2.27' not found (required by /home/phil/suricata/bin/suricata)
/home/phil/suricata/bin/suricata: /lib64/libc.so.6: version `GLIBC_2.18' not found (required by /home/phil/suricata/bin/suricata)
/home/phil/suricata/bin/suricata: /lib64/libc.so.6: version `GLIBC_2.25' not found (required by /home/phil/suricata/bin/suricata)

However, it fails similarly on Fedora 21, which is currently the "floor" on our support statement for Linux.

$ cat ../wrccdc.pcap | ./suricatarunner 
/home/phil/suricata/bin/suricata: /lib64/libc.so.6: version `GLIBC_2.22' not found (required by /home/phil/suricata/bin/suricata)
/home/phil/suricata/bin/suricata: /lib64/libc.so.6: version `GLIBC_2.27' not found (required by /home/phil/suricata/bin/suricata)
/home/phil/suricata/bin/suricata: /lib64/libc.so.6: version `GLIBC_2.25' not found (required by /home/phil/suricata/bin/suricata)

how to add this to Brim MacOS version ?

Suricata timestamps are +2 hours relative to their corresponding Zeek events

I've thus far only been able to repro this problem on my personal Thinkpad running Windows 10. I'd love to assume that my laptop is some unique corner case we'll never see in the wild, however I'd like to understand the root cause so we don't have the risk of other users running into the same problem.

Here's a repro with the Brim-Setup-rc-v0.20.0-suricata18.exe artifact and a walk through of why it creates trouble. I start by importing the https://archive.wrccdc.org/pcaps/2018/wrccdc.2018-03-23.010014000000000.pcap.gz test pcap (after uncompressing it, of course). My understanding is that in the pcap import case, we've traditionally set the default time pickers based on the begin/end timestamps we find in the overall pcap when we're indexing it, since the timestamps for all the Zeek events are expected to fall into this range as well. Here's the state of the app after that import finishes:

However, because of the +2 hour timestamp problem, a search for Suricata alerts out of the gate shows nothing.

However, if I select "Whole Space" from the time picker, the end time gets advanced by +2 and we see the Suricata alerts showing up in that time range.

Since we don't yet have Community ID on Windows (#43) it's a little tricky to pair up the alert with the corresponding Zeek event, but for this pairing it's possible since the alert is complaining about DNS contact to .biz and indeed we can find the corresponding Zeek event that summarized the same query:

My plan is to first do a bit of work with a "regular" Suricata-for-Windows and compare its output with what happens here to see if the timestamp problem is ever present on my laptop or if it follows the Suricata build.

All Suricata alerts are of category "Unknown Classtype" on CentOS v8.2 and Ubuntu 18.04

While verifying in #30 that Suricata now runs on CentOS v8.2 via the brim_x86_64_rc-v0.20.0-suricata18.rpm artifact , I noticed that all the alerts were coming up in the same category "Unknown Classtype". A closer look at the zqd-core.log made me wonder if the root cause was a failure to download the Emerging Threats rules, as there's a SSL: CERTIFICATE_VERIFY_FAILED error in here, and I could see the suricata.rules file was way smaller than I'm accustomed to seeing on platforms like macOS where I've successfully seen alerts of differing categories.

{"level":"info","ts":1606257650.3495624,"msg":"Suricata updater stdout","stdout":"24/11/2020 -- 17:40:50 - <Info> -- Loading /home/phil/.config/Brim/suricata/update.yaml\n24/11/2020 -- 17:40:50 - <Info> -- Found Suricata version 5.0.3 at /usr/lib/brim/resources/app/zdeps/suricata/bin/suricata.\n24/11/2020 -- 17:40:50 - <Info> -- Loading /usr/lib/brim/resources/app/zdeps/suricata/brim-conf.yaml\n24/11/2020 -- 17:40:50 - <Info> -- No sources configured, will use Emerging Threats Open\n24/11/2020 -- 17:40:50 - <Info> -- Fetching https://rules.emergingthreats.net/open/suricata-5.0.3/emerging.rules.tar.gz.\n24/11/2020 -- 17:40:50 - <Error> -- Failed to fetch https://rules.emergingthreats.net/open/suricata-5.0.3/emerging.rules.tar.gz: <urlopen error [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed (_ssl.c:852)>\n24/11/2020 -- 17:40:50 - <Info> -- Loading distribution rule file /usr/lib/brim/resources/app/zdeps/suricata/share/suricata/rules/app-layer-events.rules\n24/11/2020 -- 17:40:50 - <Info> -- Loading distribution rule file /usr/lib/brim/resources/app/zdeps/suricata/share/suricata/rules/decoder-events.rules\n24/11/2020 -- 17:40:50 - <Info> -- Loading distribution rule file /usr/lib/brim/resources/app/zdeps/suricata/share/suricata/rules/dhcp-events.rules\n24/11/2020 -- 17:40:50 - <Info> -- Loading distribution rule file /usr/lib/brim/resources/app/zdeps/suricata/share/suricata/rules/dnp3-events.rules\n24/11/2020 -- 17:40:50 - <Info> -- Loading distribution rule file /usr/lib/brim/resources/app/zdeps/suricata/share/suricata/rules/dns-events.rules\n24/11/2020 -- 17:40:50 - <Info> -- Loading distribution rule file /usr/lib/brim/resources/app/zdeps/suricata/share/suricata/rules/files.rules\n24/11/2020 -- 17:40:50 - <Info> -- Loading distribution rule file /usr/lib/brim/resources/app/zdeps/suricata/share/suricata/rules/http-events.rules\n24/11/2020 -- 17:40:50 - <Info> -- Loading distribution rule file /usr/lib/brim/resources/app/zdeps/suricata/share/suricata/rules/ipsec-events.rules\n24/11/2020 -- 17:40:50 - <Info> -- Loading distribution rule file /usr/lib/brim/resources/app/zdeps/suricata/share/suricata/rules/kerberos-events.rules\n24/11/2020 -- 17:40:50 - <Info> -- Loading distribution rule file /usr/lib/brim/resources/app/zdeps/suricata/share/suricata/rules/modbus-events.rules\n24/11/2020 -- 17:40:50 - <Info> -- Loading distribution rule file /usr/lib/brim/resources/app/zdeps/suricata/share/suricata/rules/nfs-events.rules\n24/11/2020 -- 17:40:50 - <Info> -- Loading distribution rule file /usr/lib/brim/resources/app/zdeps/suricata/share/suricata/rules/ntp-events.rules\n24/11/2020 -- 17:40:50 - <Info> -- Loading distribution rule file /usr/lib/brim/resources/app/zdeps/suricata/share/suricata/rules/smb-events.rules\n24/11/2020 -- 17:40:50 - <Info> -- Loading distribution rule file /usr/lib/brim/resources/app/zdeps/suricata/share/suricata/rules/smtp-events.rules\n24/11/2020 -- 17:40:50 - <Info> -- Loading distribution rule file /usr/lib/brim/resources/app/zdeps/suricata/share/suricata/rules/stream-events.rules\n24/11/2020 -- 17:40:50 - <Info> -- Loading distribution rule file /usr/lib/brim/resources/app/zdeps/suricata/share/suricata/rules/tls-events.rules\n24/11/2020 -- 17:40:50 - <Info> -- Loaded 344 rules.\n24/11/2020 -- 17:40:50 - <Info> -- Disabled 0 rules.\n24/11/2020 -- 17:40:50 - <Info> -- Enabled 0 rules.\n24/11/2020 -- 17:40:50 - <Info> -- Modified 0 rules.\n24/11/2020 -- 17:40:50 - <Info> -- Dropped 0 rules.\n24/11/2020 -- 17:40:50 - <Info> -- Enabled 0 rules for flowbit dependencies.\n24/11/2020 -- 17:40:50 - <Info> -- Creating directory /home/phil/.config/Brim/suricata/rules.\n24/11/2020 -- 17:40:50 - <Info> -- Backing up current rules.\n24/11/2020 -- 17:40:50 - <Info> -- Writing rules to /home/phil/.config/Brim/suricata/rules/suricata.rules: total: 344; enabled: 302; added: 344; removed 0; modified: 0\n24/11/2020 -- 17:40:50 - <Info> -- Writing /home/phil/.config/Brim/suricata/rules/classification.config\n24/11/2020 -- 17:40:50 - <Info> -- Skipping test, disabled by configuration.\n24/11/2020 -- 17:40:50 - <Info> -- Done.\n"}

Here's proof that the suricata.rules remained at its small default size:

$ ls -l $HOME/.config/Brim/suricata/rules
total 64
-rw-rw-r--. 1 phil phil     0 Nov 24 18:25 classification.config
-rw-rw-r--. 1 phil phil 65096 Nov 24 18:25 suricata.rules

On the same VM, using other tools, I was able to download from the URL it's complaining about.

$ wget https://rules.emergingthreats.net/open/suricata-5.0.3/emerging.rules.tar.gz
--2020-11-24 18:31:05--  https://rules.emergingthreats.net/open/suricata-5.0.3/emerging.rules.tar.gz
Resolving rules.emergingthreats.net (rules.emergingthreats.net)... 96.43.137.99
Connecting to rules.emergingthreats.net (rules.emergingthreats.net)|96.43.137.99|:443... connected.
HTTP request sent, awaiting response... 200 OK
Length: 2812166 (2.7M) [application/x-gzip]
Saving to: ‘emerging.rules.tar.gz’

emerging.rules.tar.gz                                  100%[===========================================================================================================================>]   2.68M  4.81MB/s    in 0.6s    

2020-11-24 18:31:06 (4.81 MB/s) - ‘emerging.rules.tar.gz’ saved [2812166/2812166]

$ curl -o foo https://rules.emergingthreats.net/open/suricata-5.0.3/emerging.rules.tar.gz
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100 2746k  100 2746k    0     0  3357k      0 --:--:-- --:--:-- --:--:-- 3357k

Is there something we need to do with our Suricata updater to tell it to use the same CA config as the other tools?

Strangely, I see the same "Unknown Classtype" problem on Ubuntu 18.04 when I test with the brim_amd_rc-v0.20.0-suricata18.deb artifact, but there I didn't see any SSL: CERTIFICATE_VERIFY_FAILED error. FWIW, here's the equivalent line from the zqd-core.log when the updater ran:

{"level":"info","ts":1606260835.9722102,"msg":"Suricata updater stdout","stdout":"24/11/2020 -- 15:33:03 - <Info> -- Loading /home/phil/.config/Brim/suricata/update.yaml\n24/11/2020 -- 15:33:03 - <Info> -- Found Suricata version 5.0.3 at /usr/lib/brim/resources/app/zdeps/suricata/bin/suricata.\n24/11/2020 -- 15:33:03 - <Info> -- Loading /usr/lib/brim/resources/app/zdeps/suricata/brim-conf.yaml\n24/11/2020 -- 15:33:03 - <Info> -- No sources configured, will use Emerging Threats Open\n24/11/2020 -- 15:33:03 - <Info> -- Fetching https://rules.emergingthreats.net/open/suricata-5.0.3/emerging.rules.tar.gz.\n24/11/2020 -- 15:33:45 - <Info> -- Done.\n24/11/2020 -- 15:33:45 - <Info> -- Loading distribution rule file /usr/lib/brim/resources/app/zdeps/suricata/share/suricata/rules/app-layer-events.rules\n24/11/2020 -- 15:33:45 - <Info> -- Loading distribution rule file /usr/lib/brim/resources/app/zdeps/suricata/share/suricata/rules/decoder-events.rules\n24/11/2020 -- 15:33:45 - <Info> -- Loading distribution rule file /usr/lib/brim/resources/app/zdeps/suricata/share/suricata/rules/dhcp-events.rules\n24/11/2020 -- 15:33:45 - <Info> -- Loading distribution rule file /usr/lib/brim/resources/app/zdeps/suricata/share/suricata/rules/dnp3-events.rules\n24/11/2020 -- 15:33:45 - <Info> -- Loading distribution rule file /usr/lib/brim/resources/app/zdeps/suricata/share/suricata/rules/dns-events.rules\n24/11/2020 -- 15:33:45 - <Info> -- Loading distribution rule file /usr/lib/brim/resources/app/zdeps/suricata/share/suricata/rules/files.rules\n24/11/2020 -- 15:33:45 - <Info> -- Loading distribution rule file /usr/lib/brim/resources/app/zdeps/suricata/share/suricata/rules/http-events.rules\n24/11/2020 -- 15:33:45 - <Info> -- Loading distribution rule file /usr/lib/brim/resources/app/zdeps/suricata/share/suricata/rules/ipsec-events.rules\n24/11/2020 -- 15:33:45 - <Info> -- Loading distribution rule file /usr/lib/brim/resources/app/zdeps/suricata/share/suricata/rules/kerberos-events.rules\n24/11/2020 -- 15:33:45 - <Info> -- Loading distribution rule file /usr/lib/brim/resources/app/zdeps/suricata/share/suricata/rules/modbus-events.rules\n24/11/2020 -- 15:33:45 - <Info> -- Loading distribution rule file /usr/lib/brim/resources/app/zdeps/suricata/share/suricata/rules/nfs-events.rules\n24/11/2020 -- 15:33:45 - <Info> -- Loading distribution rule file /usr/lib/brim/resources/app/zdeps/suricata/share/suricata/rules/ntp-events.rules\n24/11/2020 -- 15:33:45 - <Info> -- Loading distribution rule file /usr/lib/brim/resources/app/zdeps/suricata/share/suricata/rules/smb-events.rules\n24/11/2020 -- 15:33:45 - <Info> -- Loading distribution rule file /usr/lib/brim/resources/app/zdeps/suricata/share/suricata/rules/smtp-events.rules\n24/11/2020 -- 15:33:45 - <Info> -- Loading distribution rule file /usr/lib/brim/resources/app/zdeps/suricata/share/suricata/rules/stream-events.rules\n24/11/2020 -- 15:33:45 - <Info> -- Loading distribution rule file /usr/lib/brim/resources/app/zdeps/suricata/share/suricata/rules/tls-events.rules\n24/11/2020 -- 15:33:45 - <Info> -- Ignoring file rules/emerging-deleted.rules\n24/11/2020 -- 15:33:50 - <Info> -- Loaded 28534 rules.\n24/11/2020 -- 15:33:50 - <Info> -- Disabled 0 rules.\n24/11/2020 -- 15:33:50 - <Info> -- Enabled 0 rules.\n24/11/2020 -- 15:33:50 - <Info> -- Modified 0 rules.\n24/11/2020 -- 15:33:50 - <Info> -- Dropped 0 rules.\n24/11/2020 -- 15:33:51 - <Info> -- Enabled 145 rules for flowbit dependencies.\n24/11/2020 -- 15:33:51 - <Info> -- Backing up current rules.\n24/11/2020 -- 15:33:55 - <Info> -- Writing rules to /home/phil/.config/Brim/suricata/rules/suricata.rules: total: 28534; enabled: 21285; added: 9; removed 0; modified: 1564\n24/11/2020 -- 15:33:55 - <Info> -- Writing /home/phil/.config/Brim/suricata/rules/classification.config\n24/11/2020 -- 15:33:55 - <Info> -- Skipping test, disabled by configuration.\n24/11/2020 -- 15:33:55 - <Info> -- Done.\n"}

And here's the proof that the suricata.rules did get increase in size as we'd expect:

$ ls -l $HOME/.config/Brim/suricata/rules
total 15556
-rw-rw-r-- 1 phil phil     3207 Nov 24 15:36 classification.config
-rw-rw-r-- 1 phil phil 15925061 Nov 24 15:36 suricata.rules

Maybe we have a problem using the rules on both Linux variants, and the problem downloading the rules on just CentOS?

I was hoping to compare the outputs from the actual Suricata runs during pcap import so I could see how they differed between these Linux systems and my Mac, but if those logs are going somewhere, I can't find them.

Windows build with libpcap

Per @philrz 's comment: #17 (review)

suricatarunner.exe failure on Windows Server 2019

I tried testing manually on a scratch Windows Server 2019 Datacenter VM on Google Cloud using the brim12 artifact. This was the error I witnessed:

C:\Users\phil\Desktop\foo\suricata>type \users\phil\Desktop\NTLM-wenchao.pcap | .\suricatarunner.exe
18/11/2020 -- 15:52:04 - <Info> - Running as service: no
18/11/2020 -- 15:52:04 - <Error> - [ERRCODE: SC_ERR_CONF_YAML_ERROR(242)] - Failed to parse configuration file at line 158: could not find expected directive name

2020/11/18 15:52:04 launchSuricata failed exit status 1
The process tried to write to a nonexistent pipe.

These are the final lines of brim-conf-run.yaml, with the middle one being the line 158 (at least, that's what Notepad++ labels as line 158):

rule-files:
  - C:\Users\phil\Desktop\foo\suricata\var\lib\suricata\rules\suricata.rules
%!(EXTRA string=C:\Users\phil\Desktop\foo\suricata)

Build Suricata binary for macOS

Build a suricata binary for macOS that can be downloaded into zq/bin as a make dependency like we do for Zeek. The resulting script will likely end up as a GitHub action in a new repository that will be created for the suricata build scripts.

Disable "escape-slash" config

Per brimdata/zed#1687 (comment), @henridf has discovered that we can make the Community ID values in the EVE JSON log line up with the ones from Zeek if we change the suricata.yaml escape-slash config to "no" (https://suricata.readthedocs.io/en/suricata-6.0.0/output/eve/eve-json-output.html#json-flags).