zeek / package-template

A Zeek package template for use with the zkg package manager

Zeek Package Template

This is the default template for the Zeek package manager. If your zkg supports the create command, you can use this template to bootstrap new Zeek packages.

Features

By default, the template provides a plain Zeek package with a functional btest setup. You can add the following optional features:

• plugin adds plugin support to the new package. It includes a minimal, functional plugin that Zeek loads and shows in its -N output, with a testcase.

  The plugin's Zeek and C++ sources reside in the package's plugin folder. This is a departure from past plugin layouts that helps avoid subtle script-loading problems we've occasionally encountered in the past. You'll find the plugin-level Zeek scripts (such as __preload__.zeek) in plugin/scripts, and the package-level ones (where you'll define log streams, handle runtime events, etc) directly in the toplevel scripts folder.

• spicy-file-analyzer/spicy-packet-analyzer/spicy-protocol-analyzer each add a Spicy analyzer to the package: either a file analyzer, or a packet analyzer, or a protocol analyzer, respectively. They all expect to receive one common user variable: analyzer specifies the name of the analyzer (e.g., HTTP). Packet and file analyzers further rely on the name of a top-level Spicy unit where to start parsing their format. Protocol analyzers expect two separate units instead, one for each direction (unit_orig and unit_resp; these may be the same), as well as a protocol to specify whether it's a TCP- or UDP-based protocol that's to be parsed. For all analyzers, make sure to read through the generated package for remaining TODOs. These features and plugin are mutually exclusive. For Zeek 5.0 and newer, the resulting package will work immediately; for older Zeek versions please ensure you have the spicy-plugin Zeek package installed.

• license lets you choose a license for your package. Available choices include the Apache 2.0, BSD 2- and 3-clause, MIT, and Mozilla 2.0 licenses. You're free to use others; these are just the ones most commonly used for Zeek packages. The resulting license gets placed into COPYING at the package's toplevel.

• github-ci adds two Github Action workflows. The first tests the package across our triplet of supported binary packages (the latest nightly Zeek build, the latest release, and the latest LTS release) for pushes and pull requests. The second is a daily test of the newest package version against the Zeek nightly build. Both rely on our Github action for testing Zeek packages.

All packages require Zeek 4 or newer, and Zeek 5 is recommended.

Example

To create scripting-only Zeek package with a 3-clause BSD license:

$ zkg create --features license --packagedir newpackage
"package-template" requires a "name" value (the name of the package, e.g. "FooBar"):
name: FooBar
"package-template" requires a "author" value (your name and email address):
author: My Name <[email protected]>
"package-template" requires a "license" value (one of apache, bsd-2, bsd-3, mit, mpl-2):
license: bsd-3

$ cd newpackage
$ ls
COPYING  README  scripts/  testing/  zkg.meta

Status

Note that zkg's template support is a beta feature and some functionality is still undergoing changes. Early feedback, feature requests, and bug reports are all very welcome.

package-template's Issues

BiF help has gone missing in the plugin feature

There's supposed to be a pointer here: https://github.com/zeek/package-template/blob/master/features/plugin/plugin/src/%40SLUG%40.bif#L9

Remove random.seed from templates?

Given Zeek's deterministic option:

$ zeek --help 2>&1 | grep -- -D
    -D|--deterministic              | initialize random seeds to zero

Does it make sense to remove the random.seed file from the templates and use -D instead? How can the option be passed automatically? It wasn't clear to me how the RANDOM_SEED var is used.

Incorrect error message when combing `plugin` and `spicy-file-analyzer` features

And while I'm here. Dropping spicy-protocol-analyzer from the list of selected features doesn't seem to have any effect. Zkg is still complaining of a mutual exclusion conflict for spicy-protocol-analyzer even though it was not selected. Perhaps it was pulled in through a dependency?

% zkg create --packagedir foobar --features license plugin spicy-file-analyzer spicy-packet-analyzer
"package-template" requires a "name" value (the name of the package, e.g. "FooBar" or "spicy-http"):
name: FooBar
"package-template" requires a "namespace" value (a namespace for the package, e.g. "MyOrg"):
namespace: MyOrg
"package-template" requires a "analyzer" value (name of the Spicy analyzer, which typically corresponds to the protocol/format being parsed (e.g. "HTTP", "PNG")):
analyzer: HTTP
"package-template" requires a "unit" value (name of the top-level Spicy parsing unit for the file/packet format (e.g. "File" or "Packet")):
unit: File
"package-template" requires a "author" value (your name and email address):
author: [email protected]
"package-template" requires a "license" value (one of apache, bsd-2, bsd-3, mit, mpl-2):
license: bsd-3
error: template input invalid, the "plugin" and "spicy-protocol-analyzer" features are mutually exclusive

Originally posted by @pauldokas in zeek/package-manager#167 (comment)

Include path issue for BiFs

For the plugin feature, including a header from the plugin sources doesn't work in BiF files:

module SomePlugin;

%%{
#include "MyPluginHeader.h"
%%}

@awelzel found this to be an include path issue: BiFs seems to be looking at src while the sources are in plugin/src.

RFC: Remove spicy:: prefix from .evt files

IIUC, the spicy:: prefix causes the analyzer tag to be prefixed with SPICY_ which in turn ends up in the conn.log service field as spicy_<analyzer>. This causes the "tech" used to write an analyzer to leak through into the logs.

Should we remove the spicy:: prefix from the template?

package-template/features/spicy-protocol-analyzer/analyzer/@[email protected]@ALT-one-unit@

Line 9 in cd045d0

protocol analyzer spicy::@ANALYZER@ over @PROTOCOL_UPPER@:

Reference zeek/zeek#2651

spicy-protocol-analyzer: Conn::RemovalHook rather than connection_state_remove

The current template provides a skeleton connection_state_remove() handler for final log writing. As far as I understand Zeek base uses removal hooks throughout so that only hooks for appropriate connections are executed and not for every connection.

We should update the template to adhere to this.

package-template/features/spicy-protocol-analyzer/scripts/main.zeek@ALT-one-unit@

Lines 74 to 79 in e87ad1b

	event connection_state_remove(c: connection) &priority=-5
	{
	# TODO: For UDP protocols, you may want to do this after every request
	# and/or reply.
	emit_log(c);
	}

Don't touch `ZEEK_PLUGIN_PATH` if a package has no plugin

We've historically long had an unconditional ZEEK_PLUGIN_PATH line in the tests' btest.cfg, but it shouldn't be there unless the package has a plugin feature.

See zeek/btest#98 for related discussion.

Add top-level Makefile for plugin feature

I'm fairly used to build and install plugins via ./configure && make && make install, but the plugin feature doesn't seem to create a top-level Makefile.

Was that somehow on purpose or could we consider adding one?

We need CI

The package template needs CI coverage — the combination of features, flags, moving Zeek versions etc is too fragile to assume it just works. @rsmmr got bitten by this recently, and I once nearly pushed a breaking change to master because I didn't try out an offending combination of settings.

This relates to zeek/action-zkg-install#5: we should verify that the CI feature actually runs successfully.

`spicy-protocol-analyzer` doesn't build.

Using the template to create a new protocol analyzer does not work as expected:

$ zkg --user create --template https://github.com/zeek/package-template --version v2.0.0 --features spicy-protocol-analyzer --packagedir spicy-template-test
...
$ mkdir build && cd build && cmake ..
$ make
[ 50%] Compiling Template analyzer
error: PCH file uses an older PCH format that is no longer supported
1 error generated.
error: PCH file uses an older PCH format that is no longer supported
1 error generated.
error: PCH file uses an older PCH format that is no longer supported
1 error generated.
error: PCH file uses an older PCH format that is no longer supported
1 error generated.
error: PCH file uses an older PCH format that is no longer supported
1 error generated.
[error] <Spicy Plugin for Zeek>: JIT compilation failed
make[2]: *** [spicy-modules/template.hlto] Error 1
make[1]: *** [analyzer/CMakeFiles/Template.dir/all] Error 2
make: *** [all] Error 2

Migrate Spicy templates to register ports dynamically

Right now we generate code where ports are specified in EVT files. This makes it hard for users to change ports after the fact (e.g., impossible to disable analyzer for hardcoded ports). We should change the template to instead declare a redef'able script ports constant and on zeek_init call Analyzer::register_for_ports; the EVT port setting should then go.

Don't add toplevel package directory to `ZEEK_PLUGIN_DIR` in `get-zeek-env`

Since packages with plugins know their build directory, they should add it, not the package's toplevel, to ZEEK_PLUGIN_DIR. Being more specific here will avoid Zeek traversing into unexpected and possibly transient directories.

See zeek/btest#98 for related discussion.