tobert / effio

Tools for crunching data from fio, the Flexible IO Tester

effio

COMING SOON

Tools for running suites of tests with fio, capturing the output, then generating reports.

This tool establishes and automates a few conventions for managing tests with the goal of making those tests manageable and repeatable.

Warning

effio only supports fio 2.1.9. Later versions write one logfile per job which is not supported for merging yet.

Usage

./effio make -id 2014-05-07 \
           -dev ./conf/machines/brak.tobert.org.json \
           -fio ./conf/fio_disk_latency \
           -path ./out

Subcommands

effio make -id <string> -path <dir> -dev <file.json> -fio <dir>

All arguments are required. This command takes a set of fio configuration and a JSON file defining the devices to be tested and produces a new tree of files containing all of the data required to run the suite. The tests are a cartesian product of all fio configs x all devices in the json.

Fio configuration files are run through text/template with data from the device and other derived strings available.

• -id string a unique id for the generated suite, should be a suitable directory name
• -path dir the suite will be written under this dir with id as the first entry
• -dev file.json a file populated with device information, documented below
• -fio dir a directory containing fio configuration files

The directory structure will look something like this, given -id 'ID', the device json below, and one fio test in the -fio dir, dir/rand_512b_write_iops.fio.

path/
    ID/
        suite.json    # a dump of all information related to the suite
        rand_512b_write_iops-samsung_840_pro_256/
          config.fio  # the fio configuration file
          test.json   # a dump of all data used to generate this test
          output.json # json output from fio --output-format=json
          run.sh      # the exact command used to run fio

Device JSON Format

The JSON device file is an array of devices and some relevant data. e.g.

Check out the example file

[
   {
     "name":       "samsung_840_pro_256",
     "device":     "/dev/disk/by-id/ata-Samsung_SSD_840_PRO_Series_S1ATNEAD541857W",
     "mountpoint": "/mnt/sda",
     "filesystem": "ext4",
     "brand":      "Samsung",
     "series":     "840 PRO",
     "capacity":   256060514304,
     "rotational": false,
     "transport":  "SATA",
     "hba":        "AHCI",
     "media":      "MLC",
     "blocksize":  512
   }
]

Field	Description
name	manually assigned, will be used in file names!
device	always use the /dev/disk/by-id/ path
mountpoint	location where the filesystem is mounted
filesystem	ext4, xfs, zfs, btrfs, ntfs-3g
brand	Samsung, Fusion IO, I/O Switch Tech, Seagate, Western Digital, etc.
series	"840 PRO",
capacity	blockdev --getsize64 /dev/sda
rotational	false for SSD, true for HDD, true if device contains any HDD
transport	SATA, SAS, PCIe, MDRAID, iSCSI, virtio
hba	ioMemory, AHCI, SAS3004, USB3, mixed (for MDRAID)
media	MLC, Iron (for HDDs), TLC, SLC, Hybrid (SSHD)
blocksize	blockdev --getpbsz /dev/sda

effio's Issues

[Feature Request] support for fio 2.1.14

Hi,

Please add support for fio 2.1.14

Thanks,
Kiran.

[Feature Request] Filesystem microbench

Please add support for Filesystem microbenchmark as shown in below link

https://gist.github.com/brendangregg/7270ff9698c70d9e7496

also

http://dtrace.org/blogs/brendan/2014/01/10/benchmarking-the-cloud/