tudinfse / fex

Fex: A Software Systems Evaluator

Fex

Fex is a software evaluation framework. It is:

• extensible: can be easily extended with custom experiment types,
• practical: supports composition of different benchmark suites and real-world applications,
• reproducible: it is built on container technology to guarantee the same software stack across platforms.

Fex provides an interface for unified building, running, and processing results of evaluation experiments. Out of the box, it supports the following workloads:

• Benchmark suites: Parsec 3.0, Phoenix, Splash 3
• Applications: SQLite, PostgreSQL, Memcached, Nginx, Apache

Installing Fex

Fex does not require installation and only has to be downloaded:

git clone https://github.com/tudinfse/fex.git

Fex has been mainly tested in combination with Docker and it is recommended to do all experiments in it. For that, build the corresponding Docker image:

make

Trying it out

First, run the container:

make run

Inside the container, run one of the experiments:

./fex.py install -n phoenix
./fex.py run -n phoenix -t gcc_native -m 2 --num_runs 1

This will install all dependencies of Phoenix benchmark suite, compile the benchmarks with GCC, and run them on 2 threads.

The results of benchmark runs are aggregated in a log file, saved under /data/results/phoenix/. This directory is also mounted into your host machine, under your_project_dir/data/results/phoenix:

vim your_project_dir/data/results/phoenix/raw.csv

Using Fex

Full Fex documentation is on our wiki page.

fex's Issues

Broken timeout

The timeout feature (--timeout flag) does not work all the time. In some cases, the timeout is longer than expected, in others it does not work at all. What causes it? I don't know, we'd have to figure it out.