tomc603 / scriba Goto Github PK

Allow the user to specify file should be pre-filled. This will prevent the OS from short-cutting the read path when a file is created with fallocate() but has not been written to fully.

Ignore the input of the same path multiple times. The user should specify file numbers, or create sub directories in a path if their goal is actually to execute N times the operations against a single path.

This will change the output file names.

Output for read/write performance data is only displaying a single reader/writer for each file rather than all of them. The summing logic for each path appears to be correct though. But the summing logic for grand totals is suspect.

Output should list values for each reader/writer routine for a given file path. Then a sum for the parent path specified on the scriba CLI. Finally a grand total for all parent paths should be given.

Theoretical example

Reader Performance:
/test/path/one/scriba.0.data:
  12.99 MiB/sec
  11.87 MiB/sec
  13.72 MiB/sec
/test/path/one/scriba.1.data:
  17.01 MiB/sec
  12.82 MiB/sec
  15.75 MiB/sec
...
/test/path/one: 38.58 MiB/sec
/test/path/two: 45.58 MiB/sec
Read Total: 225.48 MiB/sec

Writer Performance:
...

Output total figures summed by base path, summing all activity from each file within a path.

IO stat collection sucks right now, collecting latency values for each read and write, and storing all values in a struct. At no point during operation is this data written to disk, so very high IO/s systems with limited RAM resources can run into an OOM.

Ideas for addressing this:

• Pass running latency data in batches through a channel to a writer routine.
• Write latency data to one tracking file per reader or writer process.
• Bucket latency values, and average them over a time duration.
• A combination of the above.

Bandwidth throughput data also isn't necessary to collect any more, given the data is gathered in main() now.