zeek / cmake

CMake scripts used in Zeek

The Zeek Network Security Monitor

A powerful framework for network traffic analysis and security monitoring.

Key Features — Documentation — Getting Started — Development — License

Follow us on Twitter at @zeekurity.

Key Features

• In-depth Analysis Zeek ships with analyzers for many protocols, enabling high-level semantic analysis at the application layer.

• Adaptable and Flexible Zeek's domain-specific scripting language enables site-specific monitoring policies and means that it is not restricted to any particular detection approach.

• Efficient Zeek targets high-performance networks and is used operationally at a variety of large sites.

• Highly Stateful Zeek keeps extensive application-layer state about the network it monitors and provides a high-level archive of a network's activity.

Getting Started

The best place to find information about getting started with Zeek is our web site www.zeek.org, specifically the documentation section there. On the web site you can also find downloads for stable releases, tutorials on getting Zeek set up, and many other useful resources.

You can find release notes in NEWS, and a complete record of all changes in CHANGES.

To work with the most recent code from the development branch of Zeek, clone the master git repository:

git clone --recursive https://github.com/zeek/zeek

With all dependencies in place, build and install:

./configure && make && sudo make install

Write your first Zeek script:

# File "hello.zeek"

event zeek_init()
    {
    print "Hello World!";
    }

And run it:

zeek hello.zeek

For learning more about the Zeek scripting language, try.zeek.org is a great resource.

Development

Zeek is developed on GitHub by its community. We welcome contributions. Working on an open source project like Zeek can be an incredibly rewarding experience and, packet by packet, makes the Internet a little safer. Today, as a result of countless contributions, Zeek is used operationally around the world by major companies and educational and scientific institutions alike for securing their cyber infrastructure.

If you're interested in getting involved, we collect feature requests and issues on GitHub here and you might find these to be a good place to get started. More information on Zeek's development can be found here, and information about its community and mailing lists (which are fairly active) can be found here.

License

Zeek comes with a BSD license, allowing for free use with virtually no restrictions. You can find it here.

Tooling

We use the following tooling to help discover issues to fix, amongst a number of others.

• Clang-Tidy
• Coverity
• PVS-Studio - static analyzer for C, C++, C#, and Java code.

cmake's Issues

BifCl.cmake needs Zeek::BifCl triggered when touching CMakeLists.txt

When developing a plugin with a ./configure && make workflow, in Zeek 5.2, after modifying/touching CMakeLists.txt (e.g adding more sources or changing flags), then re-running make would trigger a cmake re-run and continue the build usually succeeding. With latest master, modifying the CMakeLists.txt now hard-fails the re-run of cmake:

In a zeek-community-id checkout:

$ ./configure
$ cd build
$ make    # success
$ touch ../CMakeLists.txt
$ make    # success expected

With Zeek 6.0-rc1, the last step fails as follows:

$ make
CMake Error at /opt/zeek-dev/share/zeek/cmake/BifCl.cmake:2 (message):
  BifCl.cmake needs Zeek::BifCl
Call Stack (most recent call first):
  /opt/zeek-dev/share/zeek/cmake/ZeekPlugin.cmake:66 (include)
  CMakeLists.txt:22 (include)


-- Configuring incomplete, errors occurred!
See also "/home/awelzel/corelight-oss/zeek-community-id/build/CMakeFiles/CMakeOutput.log".
See also "/home/awelzel/corelight-oss/zeek-community-id/build/CMakeFiles/CMakeError.log".
make: *** [Makefile:401: cmake_check_build_system] Error 1

The workaround is to remove the build directory and re-run ./configure, but IMO the 5.2.2 behavior was nicer as it just worked. Above error is also not actionable for someone running into it.

@Neverlord - I see there's some caching if bootstrapping ran and that seems to cause the issue:

cmake/ZeekPlugin.cmake

Lines 59 to 64 in 4e41cdd

	# Make sure BifCl and BinPAC are available.
	if (NOT ZEEK_PLUGIN_INTERNAL_BUILD AND NOT ZEEK_PLUGIN_HAD_BOOTSTRAPPING)
	zeek_plugin_bootstrapping()
	# Remember that we have called the function to not call it again.
	set(ZEEK_PLUGIN_HAD_BOOTSTRAPPING ON CACHE BOOL "Plugin bootstrapping has completed." FORCE)
	endif ()

Plugin-level scripts install via symlink back into the source tree

I think this is an old optimization aimed to simplify life for developers, but nowadays it means we create a symlink back into zkg's internal state, which is risky and complicates/prevents things like zeek/package-manager#140.

An example:

$ pwd
/home/christian/inst/opt/zeek/lib64/zeek/plugins/packages/zeek-community-id
$ ls -la scripts
lrwxrwxrwx. 1 christian christian 82 Oct 20 14:46 scripts -> /home/christian/inst/opt/zeek/var/lib/zkg/clones/package/zeek-community-id/scripts/

zeek_add_plugin: No zeek_plugin_scripts() equivalent?

I was looking at converting the init-plugin script's CMakeLists.txt to the new world order:

https://github.com/zeek/zeek-aux/blob/master/plugin-support/skeleton/CMakeLists.txt

It seems for zeek_plugin_scripts() there's no corresponding parameter for zeek_add_plugin() - was that on purpose?

zeek_plugin_scripts() was added not too long ago with fb332b8. Maybe I missed some discussion that this isn't wanted anymore, but potentially confuses users converting to the new approach what to do with existing zeek_plugin_scripts().

Installing scripts is broken for Spicy analyzers.

Looks like SPICY_SCRIPTS_OUTPUT_DIR_INSTALL is never set.

Packaging Plugins creates RPMs with File Conflicts

Trying to build an RPM of a plugin package will generate an RPM with paths that will conflict with official Zeek RPM's. This was done on CentOS 7.9, using Zeek 4.0.1 (I did this a little while back). As an example, I'm using the af-packet plugin (https://github.com/J-Gras/zeek-af_packet-plugin), but this issue seems to apply to anything using the standard CMake/CPack setup provided by the Zeek skeletons/templates. (System was using devtoolset-7 scl and cmake 3.17.5)

After installing the LTS RPM's, I grabbed the af_packet source and built it using make package to invoke CPack's auto-package building capability.

Here's the list of files/paths in RPM generated for the AF_Packet Plugin using that process:

Generated by CPack RPM (no Changelog file were provided)

drwxr-xr-x    2 root    root                        0 Nov  2 10:08 /opt/zeek
drwxr-xr-x    2 root    root                        0 Nov  2 10:08 /opt/zeek/lib
drwxr-xr-x    2 root    root                        0 Nov  2 10:08 /opt/zeek/lib/zeek
drwxr-xr-x    2 root    root                        0 Nov  2 10:08 /opt/zeek/lib/zeek/plugins
drwxr-xr-x    2 root    root                        0 Nov  2 10:08 /opt/zeek/lib/zeek/plugins/Zeek_AF_Packet
-rw-rw-r--    1 root    root                      819 Apr 14  2021 /opt/zeek/lib/zeek/plugins/Zeek_AF_Packet/COPYING
-rw-rw-r--    1 root    root                     4247 Apr 14  2021 /opt/zeek/lib/zeek/plugins/Zeek_AF_Packet/README
-rw-rw-r--    1 root    root                        6 Apr 14  2021 /opt/zeek/lib/zeek/plugins/Zeek_AF_Packet/VERSION
-rw-r--r--    1 root    root                       16 Apr 28  2021 /opt/zeek/lib/zeek/plugins/Zeek_AF_Packet/__bro_plugin__
drwxr-xr-x    2 root    root                        0 Nov  2 10:08 /opt/zeek/lib/zeek/plugins/Zeek_AF_Packet/lib
-rwxr-xr-x    1 root    root                  1912688 Apr 28  2021 /opt/zeek/lib/zeek/plugins/Zeek_AF_Packet/lib/Zeek-AF_Packet.linux-x86_64.so
drwxr-xr-x    2 root    root                        0 Nov  2 10:08 /opt/zeek/lib/zeek/plugins/Zeek_AF_Packet/lib/bif
-rw-r--r--    1 root    root                       27 Apr 28  2021 /opt/zeek/lib/zeek/plugins/Zeek_AF_Packet/lib/bif/__load__.zeek
-rw-r--r--    1 root    root                      306 Apr 28  2021 /opt/zeek/lib/zeek/plugins/Zeek_AF_Packet/lib/bif/af_packet.bif.zeek
drwxr-xr-x    2 root    root                        0 Nov  2 10:08 /opt/zeek/lib/zeek/plugins/Zeek_AF_Packet/scripts
-rw-rw-r--    1 root    root                       73 Apr 14  2021 /opt/zeek/lib/zeek/plugins/Zeek_AF_Packet/scripts/__load__.zeek
-rw-rw-r--    1 root    root                      589 Apr 14  2021 /opt/zeek/lib/zeek/plugins/Zeek_AF_Packet/scripts/init.zeek
drwxr-xr-x    2 root    root                        0 Nov  2 10:08 /opt/zeek/lib/zeek/plugins/Zeek_AF_Packet/zeekctl
-rw-rw-r--    1 root    root                     1179 Apr 14  2021 /opt/zeek/lib/zeek/plugins/Zeek_AF_Packet/zeekctl/af_packet.py
drwxr-xr-x    2 root    root                        0 Nov  2 10:08 /usr/local

If you try to install the RPM, you get the following error:

Transaction check error:
  file /opt/zeek from install of zeekpluginaf_packet-2.1.2-1.x86_64 conflicts with file from package zeek-lts-core-4.0.1-1.1.x86_64
  file /opt/zeek/lib/zeek from install of zeekpluginaf_packet-2.1.2-1.x86_64 conflicts with file from package zeek-lts-core-4.0.1-1.1.x86_64
  file /opt/zeek/lib from install of zeekpluginaf_packet-2.1.2-1.x86_64 conflicts with file from package zeek-lts-core-4.0.1-1.1.x86_64
  file /opt/zeek/lib/zeek/plugins from install of zeekpluginaf_packet-2.1.2-1.x86_64 conflicts with file from package zeek-lts-core-4.0.1-1.1.x86_64
  file /opt/zeek from install of zeekpluginaf_packet-2.1.2-1.x86_64 conflicts with file from package zeekctl-lts-4.0.1-1.1.x86_64
  file /opt/zeek/lib/zeek from install of zeekpluginaf_packet-2.1.2-1.x86_64 conflicts with file from package zeekctl-lts-4.0.1-1.1.x86_64
  file /opt/zeek/lib from install of zeekpluginaf_packet-2.1.2-1.x86_64 conflicts with file from package zeekctl-lts-4.0.1-1.1.x86_64
  file /opt/zeek from install of zeekpluginaf_packet-2.1.2-1.x86_64 conflicts with file from package zeek-lts-libcaf-devel-4.0.1-1.1.x86_64
  file /opt/zeek/lib from install of zeekpluginaf_packet-2.1.2-1.x86_64 conflicts with file from package zeek-lts-libcaf-devel-4.0.1-1.1.x86_64
  file /opt/zeek from install of zeekpluginaf_packet-2.1.2-1.x86_64 conflicts with file from package zeek-lts-btest-4.0.1-1.1.x86_64
  file /opt/zeek/lib/zeek from install of zeekpluginaf_packet-2.1.2-1.x86_64 conflicts with file from package zeek-lts-btest-4.0.1-1.1.x86_64
  file /opt/zeek/lib from install of zeekpluginaf_packet-2.1.2-1.x86_64 conflicts with file from package zeek-lts-btest-4.0.1-1.1.x86_64
  file /opt/zeek from install of zeekpluginaf_packet-2.1.2-1.x86_64 conflicts with file from package zeek-lts-zkg-4.0.1-1.1.x86_64
  file /opt/zeek/lib/zeek from install of zeekpluginaf_packet-2.1.2-1.x86_64 conflicts with file from package zeek-lts-zkg-4.0.1-1.1.x86_64
  file /opt/zeek/lib from install of zeekpluginaf_packet-2.1.2-1.x86_64 conflicts with file from package zeek-lts-zkg-4.0.1-1.1.x86_64
  file /opt/zeek/lib/zeek/plugins from install of zeekpluginaf_packet-2.1.2-1.x86_64 conflicts with file from package zeek-lts-zkg-4.0.1-1.1.x86_64
  file /opt/zeek from install of zeekpluginaf_packet-2.1.2-1.x86_64 conflicts with file from package libbroker-lts-devel-4.0.1-1.1.x86_64
  file /opt/zeek/lib from install of zeekpluginaf_packet-2.1.2-1.x86_64 conflicts with file from package libbroker-lts-devel-4.0.1-1.1.x86_64
  file /opt/zeek from install of zeekpluginaf_packet-2.1.2-1.x86_64 conflicts with file from package zeek-lts-devel-4.0.1-1.1.x86_64
  file /opt/zeek/lib from install of zeekpluginaf_packet-2.1.2-1.x86_64 conflicts with file from package zeek-lts-devel-4.0.1-1.1.x86_64

Error Summary
-------------

I made the following mod/patch to ConfigurePackaging.cmake (line numbers might be off based on the version I was using):

diff --git a/ConfigurePackaging.cmake b/ConfigurePackaging.cmake
index f18fc2e..5b433ff 100644
--- a/ConfigurePackaging.cmake
+++ b/ConfigurePackaging.cmake
@@ -170,7 +170,16 @@ macro(SetPackageMetadata)
         set(CPACK_RPM_PACKAGE_GROUP "Applications/System")
     endif ()
  
-    set(CPACK_RPM_EXCLUDE_FROM_AUTO_FILELIST_ADDITION /opt /var /var/opt)
+    set(CPACK_RPM_EXCLUDE_FROM_AUTO_FILELIST_ADDITION
+        /opt
+        /var
+        /var/opt
+        ${CMAKE_INSTALL_PREFIX}
+        ${BRO_CONFIG_PREFIX}
+        ${BRO_CONFIG_PREFIX}/lib
+        ${BRO_CONFIG_PREFIX}/lib/zeek
+        ${BRO_CONFIG_PLUGIN_DIR}
+    )
 endmacro(SetPackageMetadata)
  
 # Sets pre and post install scripts for PackageMaker packages.

This results in the RPM with the following list of files/paths:

drwxr-xr-x    2 root    root                        0 Apr 28  2021 /opt/zeek/lib/zeek/plugins/Zeek_AF_Packet
-rw-rw-r--    1 root    root                      819 Apr 14  2021 /opt/zeek/lib/zeek/plugins/Zeek_AF_Packet/COPYING
-rw-rw-r--    1 root    root                     4247 Apr 14  2021 /opt/zeek/lib/zeek/plugins/Zeek_AF_Packet/README
-rw-rw-r--    1 root    root                        6 Apr 14  2021 /opt/zeek/lib/zeek/plugins/Zeek_AF_Packet/VERSION
-rw-r--r--    1 root    root                       16 Apr 28  2021 /opt/zeek/lib/zeek/plugins/Zeek_AF_Packet/__bro_plugin__
drwxr-xr-x    2 root    root                        0 Apr 28  2021 /opt/zeek/lib/zeek/plugins/Zeek_AF_Packet/lib
-rwxr-xr-x    1 root    root                  1912688 Apr 28  2021 /opt/zeek/lib/zeek/plugins/Zeek_AF_Packet/lib/Zeek-AF_Packet.linux-x86_64.so
drwxr-xr-x    2 root    root                        0 Apr 28  2021 /opt/zeek/lib/zeek/plugins/Zeek_AF_Packet/lib/bif
-rw-r--r--    1 root    root                       27 Apr 28  2021 /opt/zeek/lib/zeek/plugins/Zeek_AF_Packet/lib/bif/__load__.zeek
-rw-r--r--    1 root    root                      306 Apr 28  2021 /opt/zeek/lib/zeek/plugins/Zeek_AF_Packet/lib/bif/af_packet.bif.zeek
drwxr-xr-x    2 root    root                        0 Apr 28  2021 /opt/zeek/lib/zeek/plugins/Zeek_AF_Packet/scripts
-rw-rw-r--    1 root    root                       73 Apr 14  2021 /opt/zeek/lib/zeek/plugins/Zeek_AF_Packet/scripts/__load__.zeek
-rw-rw-r--    1 root    root                      589 Apr 14  2021 /opt/zeek/lib/zeek/plugins/Zeek_AF_Packet/scripts/init.zeek
drwxr-xr-x    2 root    root                        0 Apr 28  2021 /opt/zeek/lib/zeek/plugins/Zeek_AF_Packet/zeekctl
-rw-rw-r--    1 root    root                     1179 Apr 14  2021 /opt/zeek/lib/zeek/plugins/Zeek_AF_Packet/zeekctl/af_packet.py

This RPM installs just fine on top of a Zeek RPM install. Although, my patch works, given that ConfigurePackaging shows up in different projects, I don't know if it's a 'correct' solution, there might be some CPack variable I don't know about or some other parameter I'm missing.

Also, I understand zkg is the way forward, but zkg is exclusively source focused requiring you to build source plugins on the target system (unless things have changed, bundle just tars up the source repos).

scripts not added through zeek_plugin_script are still added to packaging

zeek_plugin_script was created to help with dependency issues when rebuilding.

The issue was if you only changed a script, and did a new build ( without clean ) you would not get the changes in the new package/ install.

While zeek_plugin_script lets you add scripts, that are then tracked now as dependencies to solve the above behavior, it is not completely correct, or perhaps not intuitively correct.

The way zeek_plugin_script should currently be understood is "Scripts added through zeek_plugin_script will be tracked for changes such that any changes to them alone can trigger a repackaging of all scripts"

It is perhaps unintuitive that zeek_plugin_script does not define the complete script set ( in the same way that you would have to add a .cc file to having it built ) but instead is a trigger where even undefined scripts will be packaged.

This is because the script to create the package is not aware of these definitions and just takes the whole script directory.

It would be more correct if the zeek_plugin_script -> _plugin_scripts var was honored by the packaging script IF PRESENT and only those scripts where included in the package.

Thus making the descriptions:

• zeek_plugin_script is optional, but when used defines the complete set of scripts that will be packaged with the plugin, and whose changes will be enough to cause a re-packaging in and of themselves.
• If it is not used at all then all scripts in the scripts directory will be packaged, but changes will not be tracked without a make clean.

Consider installing Spicy grammars for bundled Spicy parsers

With more Spicy grammars becoming part of Zeek we should consider installing Spicy grammars (.spicy files), similar to how we install C++ headers now. This should allow users to depend on bundled grammars in their parsers if needed, e.g., zeek/zeek#3234 could allow users to depend on the then bundled ASN.1 grammar.

One approach to make this work would be to modify our CMake spicy_add_analyzer macro so it installs Spicy sources passed in SOURCES.

Required compiler versions are out-of-date from Zeek documentation

https://docs.zeek.org/en/master/install.html#required-dependencies states that we require GCC >= 8 and Clang >= 9. RequireCXX17.cmake requires GCC >= 7 and Clang >= 4, and should be updated to match the what our required dependencies are.

The Apple clang version can be updated to 11.4 as well, which is equivalent to Clang 9.0 according to https://en.wikipedia.org/wiki/Xcode#Xcode_11.0_-_14.x_(since_SwiftUI_framework)_2.

cmake policy warning CMP0110

I've started to see the following in fresh configure runs on the Zeek master, likely triggered here. I have cmake 3.22.2.

-- Configuring done
CMake Warning (dev) in src/logging/writers/ascii/CMakeLists.txt:
  Policy CMP0110 is not set: add_test() supports arbitrary characters in test
  names.  Run "cmake --help-policy CMP0110" for policy details.  Use the
  cmake_policy command to set the policy and suppress this warning.

  The following name given to add_test() is invalid if CMP0110 is not set or
  set to OLD:

    `writers.ascii prefix_basename_with´

This warning is for project developers.  Use -Wno-dev to suppress it.

-- Generating done

Remove vendored cmake files?

We have a number of cmake files that we copy from the CMake distribution into our local cmake setup. Is there any reason to continue including these ourselves vs using them out of the distribution?

The following all exist in my local CMake installation, at the least:

• FindBISON.cmake
• FindOpenSSL.cmake
• FindPackageHandleStandardArgs.cmake
• FindPackageMessage.cmake
• SelectLibraryConfigurations.cmake