alexprivalov / regex-performance

Performance comparison of regular expression engines.

Regex Performance

Introduction

Regular expressions are commonly used in pattern search algorithms. This tool is based on the work of John Maddock (See his own regex comparison here) and the sljit project (See their regex comparison here).

Requirements

dependency	version
Cmake	>=3.0
Ragel	6.9
Python	>=3.0
Boost (*2)	>=1.57
Pcap	>=0.8
Autoconf	2.69 (*)
Automake	1.15 (*)
Autopoint	0.19.7 (*)
Gettext	0.19.7 (*)
Libtool	2.4.6 (*)
Git	2.11.0 (*)

(*) Tested with named version only. Older versions may work too. (*2) Needs boost-regex to be installed as a component

Supported engines

The following regex engines are supported and covered by the tool:

• CTRE
• Hyperscan
• Oniguruma
• RE2
• Tre
• PCRE2
• Rust regex crate
• Regress regex crate
• Boost regex
• C++ Standard regex
• YARA

The engines are built from their sources. In the case an installed engine should be used, the corresponding cmake variable INCLUDE_<name> has to be set to system. The configuration script tries to locate the library and uses the library for linking the benchmark. The same variable can be set to disabled to exclude an engine.

The configuration script distinguishes between nightly and other Rust toolchains to enable the SIMD-feature which is currently available in the nightly built only. The SIMD-feature improves the throughput of the regex crate for defined expressions.

Necessary packages

Install Rust:

curl https://sh.rustup.rs -sSf | sh

Other necessary packages:

sudo apt-get install cmake
sudo apt-get install libboost-all-dev
sudo apt-get install ragel
sudo apt-get install autoconf automake libtool autopoint

Building the tool

The different engines have different requirements which are not described here. Please see the related project documentations.

In the case all depencies are fulfilled, just configure and build the cmake based project:

mkdir build && cd build
cmake ..
make

The make command will build all engines and the test tool regex_perf.

To build the test tool or a library only, call make with corresponding target, i.e.:

make regex_perf

Usage

The test tool calls each engine with a defined set of different regular expression on a given file. The repository contains a ~16Mbyte large text file (3200.txt) which can be used for measuring.

./src/regex_perf -f ../3200.txt

By default, the tool repeats each test 5 times and prints the best time of each test. The overall time to process each regular expression is measured and accounted. The scoring algorithhm distributes the fastest engine 5 points, the second fastest 4 points and so on. The score points help to limit the impact of a slow regular expression eninge test in comparision to the absolut time value.

You can specify a file to write the test results per expression and engine:

./src/regex_perf -f ../3200.txt -o ../results.csv

The test tool writes the results in a csv-compatible format.

Oleksandr Chupryna's modifications

I've added 2 keys - '-t' - test data and '-e' - test pattern(s). Patterns can be specified separated by comma. Key '-m1' means run hyperscan_mutli. It is required either specify custom patterns with '-e' option or path to file that contains line by line regexes with '-i' option.

'-n' option specifies quantity of rounds(5 is default)

Examples:

./src/regex_perf -f data_to_be_grepped.txt -i regexes.txt ./src/regex_perf -t "my test data" -e "tes,ata" -n1 -m1 ./src/regex_perf -t "my test data" -e "\wdata\b" -n1 -m1 ./src/regex_perf -f data_to_be_grepped.txt -e "\wdata\b"

Spreadsheet generator

We included a spreadsheet generator for easy visualization of the results. Once you have ran the results and obtained the results.csv file, you can create a spreadsheet with (assuming you are still in the build directory)

python3 ../genspreadsheet.py results.csv

It will save an Excel spreadsheet with the name regex-results-YYYYMMDD-HHMMSS.xlsx in the current directory.

Compiling with clang + libc++

Unfortunately it is not possible to run both standard C++ from GCC/stdlibc++ and clang+libc++ at the same time, it is just the way that cmake selects a single compiler.

To run with clang+libc++ use the following recipe:

mkdir build && cd build
cmake \
    -DCMAKE_BUILD_TYPE=Release \
    -DCMAKE_EXE_LINKER_FLAGS="-lc++abi -lc++"  \
    -DCMAKE_CXX_COMPILER=/usr/local/bin/clang++ \
    -DCMAKE_C_COMPILER=/usr/local/bin/clang \
    -DCMAKE_CXX_FLAGS_INIT="-std=c++20 -stdlib=libc++ -march=native -mtune=native" \
    -G Ninja ..

Results

These results were obtained in an AMD Threadripper 3960X (Zen2) at 3.8 GHz running Ubuntu 20.04.5 LTS.