arepetti / netmicrobenchmark Goto Github PK

Advanced statistical analysis may be performed with Microsoft Excel. This OutputRenderer should not use Statisticsbut simply dump all measures in Benchmarkinto a spreadsheet document.

Better statistical analysis (especially if using a huge number of measures) may detect aleatory performance in some algorithms (for example because of dependencies on unstable resources like I/O or network connection) like this:

public class Benchmark {
    public void DownloadAllFiles() {
        foreach (var fileToDownload in GetFilesToDownload()) {
            DownloadFileWithHttp(fileToDownload.LocalPath, fileToDownload.ServerUrl);
        }
    }

    public void DownloadAllFiles() {
        Parallel.ForEach(GetFilesToDownload(), fileToDownload =>  {
            DownloadFileWithHttp(fileToDownload.LocalPath, fileToDownload.ServerUrl);
        });
    }

    // ...
}

Note that highly dispersed measures may also be caused by wrong benchmark setup...

Benchmarks may be performed directly inside Visual Studio (searching for classes by convention or popping up a dialog where user can select which classes he wants to benchmark). Add-in should be available with nuget.

Currently to execute a benchmark you need Visual Studio (where you write and compile code). A small utility where you can write benchmark code (without Intellisense but with syntax highlighter) and run/view test results may be useful.

Same utility may also benchmark a set of given already compiled assemblies eventually selecting specific methods to test. It may work also simply generating benchmark code to invoke those methods - if public - because for this kind of tests we.assume that function call overhead is almost negligible.

Command line support would also be a nice feature (for example as a simple msbuild support).

Current implementation is for VS2k12 however some desiderable features are available in VS2k15, when codebase is stable (to have a reliable version compatible with older compilers) will be available then this minor porting should be done.

Currently the only output render is HtmlOutputRendererwhich renders to HTML files. Sometimes a simple text file is enough (possibly to be used directly as output in console applications).

Note that this renderer is needed (and a simple plain text template cannot be used) because HtmlOutputRenderer also performs HTML escaping (which we don't want in plain text files).
Note that this is not true (and this issue may be closed) if we switch to Razor as template engine (instead of Mustache).