hpides / perma-bench Goto Github PK

View Code? Open in Web Editor NEW

17.0 4.0 2.0 4.45 MB

A benchmarking suite to evaluate the performance of persistent memory access (PerMA-Bench @ VLDB '22)

License: MIT License

CMake 2.90% C++ 85.32% Shell 0.18% Python 11.09% CSS 0.51%

perma-bench's People

Contributors

Stargazers

Watchers

Forkers

yzouchen mweisgut

perma-bench's Issues

Allow user to specify --no-numa when running with numactl directly

TearDown in Destructor of Benchmark

Unmap pmem file in destructor of benchmark

Check constructor with default values for benchmarks

Fix HTML/plot layout

There is some overlay in the plots. Also, the plots should be cleaner in the HTML pages. Maybe we should add a table layout.

fix overlap
add KB/MB/GB to large numbers
separate latency and bandwidth visually
fix y scale for large values to avoid 1e6 etc.

Fix result path arg

If the path does not start with ./ or /, it is empty.

Mixed Workloads in Parallel

add something like parallel_benchmark as top level entry in config file to run multiple different workloads in parallel

Use HdrHistogram for latency measurements

Implement SetUp for Benchmark

Create IOOperations
Map files read & write

Run visualizations

Change path to path to JSON and not to a directory or accept more than one JSON in a result folder.

Result Array with RAW option

Add quartiles to output
Add raw option for output
Adapt config output

Check cache flush between runs

Array
WBINVD

Generate random data

One memory mapped file per benchmark in yaml config

Non-AVX operations for write and read

Fix config output

The parameters:

persist _instruction
data_instruction
exec_mode
random_distribution

only output there enum integer. Instead, the input string should be used as an output as well.

Introduce different distribution for read write relation

Data format benchmark

{  "benchmark_name": "bm_name",
   "config": <config>,
   "results": [
       { "type": "write",
         "latency": x, // (in nanoseconds)
         "bandwidht":  x, // (in GB/s)
         "data_size": x, // (in GB)
         "start_ts": x, // (timestamp)
         "end_ts": x, // (timestamp)
         "thread_id": x //
       },
       { "type": "read",
         "latency": x, // (in nanoseconds)
         "bandwidht":  x, // (in GB/s)
         "data_size": x, // (in GB)
         "start_ts": x, // (timestamp)
         "end_ts": x, // (timestamp)
         "thread_id": x //
       },
       { "type": "pause",
         "length": x, // (in microseconds)
         "start_ts": x, // (timestamp)
         "end_ts": x, // (timestamp)
         "thread_id": x //
       }
     ]
}

Define Config Struct per Benchmark

Use struct as constructor arg
Use struct as member variable
Implement convert function for each benchmark

Asserts for parameter

Add assertions for our assumptions like relation of number of threads to number of partitions or read and write ratio.

Add cli parser

Single header parser: https://github.com/CLIUtils/CLI11

Refactor camelCase to snake_case

Investigate SegFault in BenchmarkTest RunMultiThread tests

Current status: tests are commented out

NUMA None config

Add NUMA none config in which threads can run on every CPU

Mixed Workload NUMA (multiple sockets)

Idea: give the user the option to differentiate between near or far benchmarks. Threads are scheduled on the same numa node where the data is in memory for near benchmarks. They are scheduled on different numa nodes if possible.

Benchmarksuite

Main -(filename)-> BMSuite

Create benchmark
Setup benchmark
Benchmark run
Benchmark getResult
Visualization

Run on simulators

See if we can setup a simulator to work on, e.g. quartz.

Create Project Website

We should create a GitHub Pages site for this project.

The website should contain

documentation
all our results (not just the ones form the paper)
paper, presentation, etc.

Add NUMA thread pinning

Dump intermediate result jsons

If we wait until the end, all results are lost if a single BM fails. We should find a way to print intermediate results.

Implement different persist instructions

Error handling in benchmark threads

How do we handle errors in ::run_in_thread()?

Only one read/write ratio option

Instead of having read_ratio and write_ratio, only have one variable. then we can add this as a matrix arg

Enable mixed patterns

Integrate mixed patterns such as:

Random Read & Sequential Write
Random Write & Sequential Read

Handle far nodes when they are given by user

This is still open after #102.

Define result folder structure

User passes path as program argument via -r or --result-directory or default is current_path.

Map for BenchmarkOptions

Use Map in resolveBenchmarkOption instead of multiple ifs.

Loop unrolling for better performance

Setup and tear down method for benchmarks

Switch flush operations to generalized version

Use three items: None, NoCache, Cache
These need to be implemented system-specific but should be hidden from the user.

Fix standard deviation

This is the result for one benchmark:
{ "bandwidth": { "read": 43.37 }, "config": { ... }, "duration": { "avg": 212.00540293040294, "lower_quartile": 205, "max": 203647, "median": 215, "min": 69, "percentile_90": 234, "percentile_95": 241, "percentile_99": 284, "percentile_999": 370, "percentile_9999": 545, "std_dev": 805.4285555848616, "upper_quartile": 225 } }

std_dev value is 805.xx which is clearly to high.

Fix annoying test warning

We constantly get:

perma-bench/test/utils_test.cpp:14: warning: the use of 'tmpnam' is dangerous, better use 'mkstemp'.

We should fix this sometime.