Comments (8)
I proposed criteria in fortran-lang/webpage#61. At the high level:
Our benchmarks should show, that Fortran is an excellent choice for this mission:
- enable scientists, engineers, and other domain experts to write programs that naturally express the mathematics and algorithms employed, are portable across HPC systems, remain viable over decades of use, and extract a high percentage of performance from the underlying hardware.
At a lower level, we might want to consider to have several sections of benchmarks:
- Fortran only (Fortran compilers comparison)
- Languages comparison (the best is probably to show various Fortran and C++ and other languages' compilers); inline assembly and intrinsics are not allowed
- Assembly section (where any code in any form is allowed)
The "Languages comparison" section can also have subsections, like
- all optimizations on (the default section)
-ffast-math
not allowed (that's essentially the Julia benchmarks page), which has its merits, but I would argue most Fortran users care more about the "all optimizations on" section
In general, I would say that instead of rejecting some benchmarks (for one reason or another), let's rather have different sections and explain why a given benchmark (say with -ffast-math
enabled) is not a good candidate for a section where -ffast-math
is not allowed, but it's a great candidate for a section with "all optimizations on". That way different people and communities might be interested in different sections (for example I expect the Julia community might be particularly interested in the -ffast-math
is not allowed section, and most Fortran users I expect might be interested in the "all optimizations on" section).
from benchmarks.
@LKedward good point, I agree. I think we should simply collect as much data as we can. How we then visualize and present the data and organize it into sections is a tough problem and we should be flexible and I expect we will iterate on things as we gain experience.
I do think we should have some recommendations what users should use for their codes, and recommend that these are good defaults, but if they care more about performance, to test different options and see for their particular code.
from benchmarks.
I agree. Run benchmarks for all compilers and all optimization levels, while keeping note that inter-compiler comparison may be meaningful only for no-optimization case and maybe the maximum optimization, but I'm not sure about the latter.
from benchmarks.
I do think we should have some recommendations what users should use for their codes, and recommend that these are good defaults, but if they care more about performance, to test different options and see for their particular code.
I agree, this is important.
Run benchmarks for all compilers and all optimization levels, while keeping note that inter-compiler comparison may be meaningful only for no-optimization case and maybe the maximum optimization, but I'm not sure about the latter.
For inter-compiler comparison it may be best to simply compare the best result from all levels.
from benchmarks.
I agree, this is a good structure.
For the Fortran only comparison, it would be good to have several levels of optimisation to be able to see what is gained at each level:
- O0
- O1
- O2
- O3
- O3+ffast-math
from benchmarks.
Remember that different Fortran compilers have different optimization options. For GFortran, I use the following options to get maximum performance: -O3 -march=native -ffast-math -funroll-loops
. More modern GFortran also has -Ofast
. Furthermore, I should enable things like allocating arrays on stack.
So we can have various sections that you proposed, but it might be hard to do that across compilers. But at the very least, we need to have a section of "the best options for the given compiler and benchmark (?)". I put a question mark next to benchmark, as I don't know if we should allow specific options that are benchmark dependent, or just have options for the whole section.
Update: thinking about this, I think I would like a section where we figure out the best optimizations options that we recommend for each Fortran compiler, and use those options for all benchmarks in the section. The idea is that users would then simply use the same options for their code and could expect the performance from the benchmarks.
from benchmarks.
Yes, the list of optimisation levels will be compiler-dependent - hence why we need a flexible drive framework.
By running many optimisation levels we can identify the best set for each benchmark - this is what happens in the Julia repo.
This can also identify any cases of over-aggressive optimisation
from benchmarks.
we figure out the best optimizations options that we recommend for each Fortran compiler, and use those options for all benchmarks in the section.
The problem I see with this is that the 'best optimisation options' will be code-dependent.
If we run a selection of different options for each compiler, we can automatically determine the best set for each benchmark case - this informs the user which options are suitable for which codes.
from benchmarks.
Related Issues (20)
- Add summation of sin example
- Add "benchmarks" from Julia
- Add PRK kernels
- Add examples from benchmarksgame
- Add FFT examples
- Add example for M(i,j) = exp(i_*k*sqrt(a(i)**2 + a(j)**2))
- Add Recaman's sequence
- Random number generation
- Should we rename the repository from `benchmarks` to `idioms`? HOT 6
- Ising model HOT 4
- Himeno benchmark
- Renaming default branch HOT 1
- precision problem for optimized.f90 HOT 1
- add topics
- TeaLeaf benchmark HOT 1
- Automatic benchmarks via Github workflow HOT 1
- USM3D CFD benchmark
- LULESH HOT 3
- HPCG Benchmark
- Arrays vs loops
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from benchmarks.