jpenuchot / ctbench Goto Github PK
View Code? Open in Web Editor NEWCompiler-assisted variable size benchmarking for the study of C++ metaprogram compile times.
Home Page: https://jpenuchot.github.io/ctbench-docs/
License: MIT License
Compiler-assisted variable size benchmarking for the study of C++ metaprogram compile times.
Home Page: https://jpenuchot.github.io/ctbench-docs/
License: MIT License
Just figured that some configs I wrote in https://github.com/JPenuchot/rule-of-cheese/
are generic enough to be provided as examples. I'd like them to be baked in a default config directory, and maybe implement a lookup directory list for configurations.
Compiler execution time measurement through ttw
seems to be broken, the time values it reports aren't coherent with Clang's time trace profiler. More works needs to be done to measure compiler execution more precisely, and Hyperfine seems to have that figured out so it's worth having a look and reproduce their measurement method in ttw
.
Hey, please find my review of your tool below:
Regarding the manuscript I don't have major comments, other than that there are many typos/grammatical errors, especially in the summary.
I found the installation of the package a bit tedious:
./vcpkg/vcpkg install sciplot fmt
on an ARM. I think it would be easiest to provide a Dockerfile that does the exact installation with all dependencies in the example folder (and have a GitHub action that tests on Mac M1).I think providing install instructions using conda would be helpful and also make it straight-forward to use for Mac users. I think all required dependencies are on there.
I haven't done much C++ development in the last year but seeing how much traction Meson is gaining, I find it unusual that it isn't provided as alternative build system. Would that be easy to offer? Also, is the CMake 3.25 dependency really necessary?
I think providing a self-contained example with exact instructions how the visualisations are created is helpful. I have probably overlooked something, but it's not 100% clear how you made them from a quick look.
Create a CONTRIBUTING file with how-to's on contributing new code.
Cheers,
Simon
nlohmann::json has a nice interface but performance doesn't scale for what we're doing (processing hundreds of megabytes of JSON trace events).
Running ctbench-grapher-plot
through perf shows that nlohmann::json-related calls are the ones that account for most overhead.
Turning graph saving off makes ctbench-grapher-plot
execution a lot shorter (a several seconds vs several minutes). Generating graphs using ROOT instead of Sciplot might be a good way to cut graph generation times by a significant factor.
Having an up-to-date documentation built upon the main
branch would be a nice plus. Having it running outside of Github would be even nicer.
The compare_by plotter saves plots with filenames generated using the key itself. This becomes problematic when the symbols grow large enough for filenames to be too long.
The issue is known and currently has no other solution than generating smaller symbols at the moment, eg. benchmark driver functions with names that do not depend on benchmark instantiation size.
Solutions for this problem are welcome. I'm open to having different name shortening strategies, generating index files to help find plot files, and others. I will address this issue if it becomes really problematic.
I'm on Ubuntu 22.04, which is not that old, and finding that cmake 3.22.1 is too old to meet your CMakeLists.txt requirements. Can these be relaxed?
Furthermore, CMakeLists.txt requires 3.25 while CMakePresets.json requires 3.23. Can you make these consistent?
Took note from @marcorubini's idea of using a multi tree visualization strategy to display multiple clang traces on a single graph. At the moment this is only a reminder issue for later.
Seems to be comparable to Clang's time trace, so adapting ctbench to it might be pretty easy.
https://devblogs.microsoft.com/cppblog/introducing-c-build-insights/
I own zero Windows machine at the moment, so any help is welcome.
Thanks @JolyLoic for pinging me about that :)
To fix for later:
boost
is missingsciplot
should be specified as sciplot-git
(the regular sciplot
package uses catch2-v2
, which can't be installed with the new version)If I edit CMakeLists.txt
to read:
cmake_minimum_required(VERSION 3.22)
and then run
cmake -DCMAKE_BUILD_TYPE=RelWithDebInfo -GNinja ..
ninja
I see
/z/ctbench/grapher/lib/grapher/plotters/debug.cpp:1:10: fatal error: llvm/Support/raw_ostream.h: No such file or directory
1 | #include <llvm/Support/raw_ostream.h>
| ^~~~~~~~~~~~~~~~~~~~~~~~~~~~
/z/ctbench/grapher/grapher-utils.cpp:1:10: fatal error: llvm/Support/CommandLine.h: No such file or directory
1 | #include <llvm/Support/CommandLine.h>
| ^~~~~~~~~~~~~~~~~~~~~~~~~~~~
Is it possible to remove this LLVM dependency in favour of using the C++ STL instead? If not, is it possible to make the LLVM linking mechanism more robust?
Maybe: add action for tests compiled with GCC?
I'm not a CMake expert but maybe someone would be able to help making ctbench installable and usable with find_package
. The end goal would be to end up with something that's easy to handle with makepkg (AUR), conan, and vcpkg.
Is it possible to set sciplot up so that it can be pulled from a submodule or similar without requiring a system-wide install?
One way to easily add (limited) support for GCC would be to measure compiler execution time instead of relying on internal profiling data. This could be useful to compare metaprogram performance scaling across a variety of compilers.
The main issues are:
And so far my favorite solutions are:
Adding a predicate that matches events with a given parent. The matching could be done by name, by pointer, or simply by using other predicates to make it modular.
The main sub-issue for this one will be to keep track of event arborescence.
I'm having a hard time finding a good way to organize the documentation into something that's clear and concise. I'm currently not satisfied by the way JSON configuration values are documented as I feel like users have to dig way too much just to find JSON configuration information.
Additionally, the docs aren't clear about what's internal or not.
I'm leaving this issue here as long as I'm not satisfied with way the docs are presented, any suggestion or critics are welcome.
Some GCC flags worth checking out:
-ftime-report
-ftime-report-details
-time[=file]
It'd be handy if you made a note in the readme that running
sudo apt install nlohmann-json3-dev
installs a necessary prerequisite. Perhaps there are others as well?
ctbench
only supports variable size benchmarks, support for single size benchmarks for A/B comparisons would be nice too. It would require work on everything from the CMake API to grapher core data structures, and possibly the CLI too.
Trying to use cmake 3.22.1 to run
cmake --preset release
per the installation instructions returns
CMake Error: Could not read presets from /z/ctbench: Unrecognized "version" field
A declarative, efficient, and flexible JavaScript library for building user interfaces.
๐ Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. ๐๐๐
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google โค๏ธ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.