quepas / compiler-benchmark-suites

A list of benchmark suites used in the research related to compilers, program performance, scientific computations etc.

Compiler-benchmark-suites

A list of benchmark suites (and some loose kernels) used in the research related to compilers, program performance, scientific computations etc.

Benchmark suites

Benchmark	Lang	Platform	Published in	Content	Additional info
AnghaBench	C	CPU	[DaSilva2021]	~1 million benchmarks	Obtained from open-source repositories; for analysis of predictive compilers
BEEBS	C	Embedded	[Pallister2013]	10 benchmarks	For analysis of energy consumption on embedded platforms
Berkeley Multimedia Workload	C	CPU (MMX/SSE/...)	[Slingerland2002]	14 applications	Architectural support for multimedia applications
Embench™	C	Embedded	[Patterson2019]	19 benchmarks	Modern benchmarks for performance analysis built on top of selected BEEBS benchmarks
GridNBP 3.1	Fortran, Java (v3.0)	CPU	[VanDerWijngaart2002]	4 benchmarks	Evaluation of the computational grids, their communications and distributed tasks.
LCALS v1.0.2	C	CPU	[Hornung2013]	32 loops	For analysis of compiler optimisations; modern extension to Livermore loops; more information
Livermore loops (LFK)	FORTRAN	CPU	[McMahon1986]	24 loops	For performance analysis
Livermore loops	C	CPU	1992	24 loops	Port to C
Mälardalen WCET Benchmarks	C	CPU	[Gustafsson2010]	35 benchmarks	For Worst-Case Execution Time (WCET) analysis
MediaBench	C	CPU	[Lee1997]	19 applications	For analysis of Instruction-Level Parallelism (ILP) in compilers; source code
MediaBench II	C	CPU	[Fritts2005, Fritts2009]	11 applications	General analysis of compilers on video/media related application
MiBench	C	Embedded	[Guthaus2001]	6 benchmarks	For performance analysis
NAS Parallel Benchmarks 3.4.2	C, Fortran, MPI, OpenMP, Java (v3.0)	CPU	[Bailey1991]	5 kernels, 3 programs, 4 other benchmarks	Evaluation of the performance of parallel supercomputers; more information
PARKBENCH 2.1.1	Fortran, MPI, PVM	CPU	[Hockney1994]	10 measurement codes, 7 kernels, 3 application	For performance analysis of parallel architectures; more information
PARSEC 3.0	C, OpenMP, pthreads	CPU	[Bienia2008]	13 programs	For research on parallelization; other versions
PolyBench/ACC	C, CUDA, OpenCL, OpenACC, HMPP, OpenMP	GPU, accelerators	[Grauer-Gray2012]	29 kernels	Updated PolyBench/GPU; GitHub
PolyBench/C 4.2	C	CPU	2010	30 kernels	For the analysis of performance and compiler optimisations (especially related to polyhedra compilation); more information; SourceForge
PolyBench/Fortran 1.0	FORTRAN	CPU	2011	30 kernels	Port to FORTRAN; more information
PolyBench/GPU 1.0	C, CUDA, OpenCL, OpenHMPP	CPU, GPU	[Grauer-Gray2012]	15 kernels	Port to heterogeneous architectures; more information
Rodinia v3.1	C, OpenMP, OpenCL, CUDA	CPU, GPU	[Che2009]	23 applications (2 under evaluations)	Performance analysis of heteregenous architectures
SPLASH	C	CPU	[Singh1992]	4 kernels, 8 programs	For research on parallelization; source code unavailable
SPLASH-2	C	CPU	[Woo1995]	4 kernels, 8 programs	For research on parallelization; original source code is unavailable, several modifications: staceyson/splash2, liuyix/splash2_benchmark, andysan/splash2
SPLASH-3	C	CPU	[Sakalis2016]	4 kernels, 8 programs	For research on parallelization
TSVC	FORTRAN	CPU	[Callahan1988]	135 loops	For testing automatic vectorizing compilers; Single precision version
TSVC	C	CPU	[Maleki2011]	151 loops	Extended port to C
TSVC 2	C	CPU	2015	151 loops	Update to TSVC in C
UTDSP	C	CPU	1992	6 loops, 12 programs	For testing compilers on Digitial Signal Processing (DSP) applications; more information
VersaBench	C	Custom CPUs	[Rabbah2004]	15 programs	For the performance analysis of new flexible CPU architectures

Full names

• BEEBS — Bristol/Embecosm Embedded Benchmark Suite
• LCALS — Livermore Compiler Analysis Loop Suite
• LFK — Livermore FORTRAN Kernels
• NPB — NAS Parallel Benchmarks
• PARKBENCH — PARallel Kernels and BENCHmarks
• PARSEC — The Princeton Application Repository for Shared-Memory Computers
• PolyBench — The Polyhedral Benchmark Suite
• SPLASH — Stanford Parallel Applications for Shared-Memory
• TSVC — Test Suite for Vectorizing Compilers
• VersaBench — A Benchmark Suite for Versatile Architectures

Collections of benchmarks

• [email protected]
• Benchmarks by Nikolaos Kavvadias
• Benchmarks by lac-dcc Research Group - 260 benchmarks, 36 test suites

Collections of loose kernels

• RRZE-HPC/Kerncraft
• ucb-bar/opencl-kernels

References

• [Bailey1991] D. H. Bailey et al., "The NAS parallel benchmarks summary and preliminary results," Supercomputing '91: Proceedings of the 1991 ACM/IEEE Conference on Supercomputing, 1991, pp. 158-165, doi: https://doi.org/10.1145/125826.125925.
• [Bienia2008] Christian Bienia, Sanjeev Kumar, Jaswinder Pal Singh, and Kai Li. 2008. "The PARSEC benchmark suite: characterization and architectural implications". In Proceedings of the 17th international conference on Parallel architectures and compilation techniques (PACT '08). Association for Computing Machinery, New York, NY, USA, 72–81. doi:https://doi.org/10.1145/1454115.1454128
• [Callahan1988] Callahan, D., Dongarra, J., & Levine, D. (1988). "Vectorizing compilers: a test suite and results". In Proceedings. SUPERCOMPUTING ’88 (pp. 98–105). IEEE Comput. Soc. Press. https://doi.org/10.1109/SUPERC.1988.44642
• [Che2009] S. Che et al., "Rodinia: A benchmark suite for heterogeneous computing," 2009 IEEE International Symposium on Workload Characterization (IISWC), 2009, pp. 44-54, doi: https://doi.org/10.1109/IISWC.2009.5306797
• [DaSilva2021] A. F. da Silva, B. C. Kind, J. W. de Souza Magalhães, J. N. Rocha, B. C. Ferreira Guimarães and F. M. Quinão Pereira, "ANGHABENCH: A Suite with One Million Compilable C Benchmarks for Code-Size Reduction," 2021 IEEE/ACM International Symposium on Code Generation and Optimization (CGO), 2021, pp. 378-390, doi: https://doi.org/10.1109/CGO51591.2021.9370322
• [Fritts2005] Jason E. Fritts, Frederick W. Steiling, Joseph A. Tucek, "MediaBench II video: expediting the next generation of video systems research," Proc. SPIE 5683, Embedded Processors for Multimedia and Communications II, (8 March 2005); https://doi.org/10.1117/12.587955
• [Fritts2009] Jason E. Fritts, Frederick W. Steiling, Joseph A. Tucek, and Wayne Wolf. 2009. MediaBench II video: Expediting the next generation of video systems research. Microprocess. Microsyst. 33, 4 (June, 2009), 301–318. https://doi.org/10.1016/j.micpro.2009.02.010
• [Guthaus2001] M. R. Guthaus, J. S. Ringenberg, D. Ernst, T. M. Austin, T. Mudge and R. B. Brown, "MiBench: A free, commercially representative embedded benchmark suite," Proceedings of the Fourth Annual IEEE International Workshop on Workload Characterization. WWC-4 (Cat. No.01EX538), Austin, TX, USA, 2001, pp. 3-14, doi:https://doi.org/10.1109/WWC.2001.990739
• [Grauer-Gray2012] Scott Grauer-Gray, Lifan Xu, Robert Searles, Sudhee Ayalasomayajula, and John Cavazos. "Auto-tuning a High-Level Language Targeted to GPU Codes". Proceedings of Innovative Parallel Computing (InPar '12), 2012. https://doi.org/10.1109/InPar.2012.6339595
• [Gustafsson2010] Jan Gustafsson, Adam Betts, Andreas Ermedahl, and Björn Lisper. "The Mälardalen WCET benchmarks: Past, present and future". 10th International Workshop on Worst-Case Execution Time Analysis, WCET 2010, July 6, 2010, Brussels, Belgium. https://doi.org/10.4230/OASIcs.WCET.2010.136
• [Hockney1994] Roger Hockney, and Michael Berry. "Public International Benchmarks for Parallel Computers". Technical report. February 7, 1994. https://netlib.org/parkbench/parkbench.ps
• [Hornung2013] Richard D. Hornung and Jeffrey A. Keasler, “A Case for Improved C++ Compiler Support to Enable Performance Portability in Large Physics Simulation Codes”, LLNL-TR-635681 (2013). https://computing.llnl.gov/projects/co-design/compilersupportwhitepaper_tr-635681_cover.pdf
• [Lee1997] Chunho Lee, M. Potkonjak and W. H. Mangione-Smith, "MediaBench: a tool for evaluating and synthesizing multimedia and communications systems," Proceedings of 30th Annual International Symposium on Microarchitecture, 1997, pp. 330-335, doi:https://doi.org/10.1109/MICRO.1997.645830
• [Maleki2011] Maleki, S., Gao, Y., Garzarán, M. J., Wong, T., & Padua, D. A. (2011). "An evaluation of vectorizing compilers". Parallel Architectures and Compilation Techniques - Conference Proceedings, PACT, 7, 372–382. https://doi.org/10.1109/PACT.2011.68
• [McMahon1986] F. H. McMahon. "Livermore fortran kernels: A computer test of numerical performance range". Technical Report UCRL-53745, Lawrence Livermore National Laboratory, Livermore, CA, December 1986; COMMENT: no PDF on the web.
• [Patterson2019] David Patterson, Jeremy Bennett, Palmer Dabbelt, Cesare Garlati, G. S. Madhusudan and Trevor Mudge. "Embench™: An Evolving Benchmark Suite for Embedded IoT Computers from an Academic-Industrial Cooperative: Towards the Long Overdue and Deserved Demise of Dhrystone" (2019). https://riscv.org/wp-content/uploads/2019/06/9.25-Embench-RISC-V-Workshop-Patterson-v3.pdf
• [Pallister2013] James Pallister, Simon Hollis, Jeremy Bennett (2013). "BEEBS: Open Benchmarks for Energy Measurements on Embedded Platforms" http://arxiv.org/abs/1308.5174.
• [Rabbah2004] Rodric M. Rabbah, Ian Bratt, Krste Asanovic, and Anant Agarwal, "Versatility and VersaBench: A New Metric and a Benchmark Suite for Flexible Architectures", MIT CSAIL Technical Memo, MIT-LCS-TM-646, June 2004, http://catfish.csail.mit.edu/~rabbah/versabench/MIT-LCS-TM-646.pdf
• [Sakalis2016] C. Sakalis, C. Leonardsson, S. Kaxiras and A. Ros, "Splash-3: A properly synchronized benchmark suite for contemporary research", 2016 IEEE International Symposium on Performance Analysis of Systems and Software (ISPASS), 2016, pp. 101-111, doi: https://www.doi.org/10.1109/ISPASS.2016.7482078.
• [Singh1992] Jaswinder Pal Singh, Wolf-Dietrich Weber, and Anoop Gupta. 1992. "SPLASH: Stanford parallel applications for shared-memory". SIGARCH Comput. Archit. News 20, 1 (March 1992), 5–44. DOI:https://doi.org/10.1145/130823.130824
• [Slingerland2002] Slingerland, N., Smith, A. Design and characterization of the Berkeley multimedia workload. Multimedia Systems 8, 315–327 (2002). https://doi.org/10.1007/s005300200052
• [VanDerWijngaart2002] Rob F. Van der Wijngaart et al. "NAS Grid Benchmarks Version 1.0". NASA Technical Report NAS-02-005. July, 2002. https://www.nas.nasa.gov/assets/pdf/techreports/2002/nas-02-005.pdf
• [Woo1995] S. C. Woo, M. Ohara, E. Torrie, J. P. Singh and A. Gupta, "The SPLASH-2 programs: characterization and methodological considerations," Proceedings 22nd Annual International Symposium on Computer Architecture, 1995, pp. 24-36, doi: https://www.doi.org/10.1109/ISCA.1995.524546.