Awesome-Database-Testing
This is a curitated list of resources on database testing.
What covered: papers, blogs, notes, tools and etc. for whoever wants to test a database mangement system (DBMS).
What is not: using SQLs, learning database basic knowledge, implementing a database system.
Note: this is NOT an exhaustive list of materials, you may want to find more accroding to references of each item.
High-level Views
Here we put materials with general discussions.
Research Papers
NOTE: We only incldue peer-reviewed and pulished papers here, the same applies to other entries. You may want to find the authors' free-version of the papers on their personal page.
-
Understanding the query optimization Query Optimization in Database Systems [1984]
-
Understanding the query optimization - a more recent view An Overview of Query Optimization [1998]
-
What are the problems from the indutrial view Testing SQL Server's Query Optimizer : Challenges , Techniques and Experiences
-
Optimizer Evaluation How good are query optimizers, really?
Others
- What are the problems from the academic view Is Query Optimization a “Solved” Problem?
Input Generation
Research Papers
- Query generation with template substitution Generating Thousand Benchmark Queries in Seconds
- Language for customized data generator Flexible Database Generators
- RAGS Microsoft SQL Server Massive Stochastic Testing of SQL
- Generate data regarding the query constraints QAGen: Generating Query-Aware Test Databases
- How could symbolic execution help Qex: Symbolic SQL Query Explorer
- Data generation as a search problem Search-based test data generation for SQL queries
Tools
- Randgen MySQL version (not maintained) doc, MariaDB version, PingCAP version
- pquery - stress testing MySQL
- Sysbench - MySQL benchmarks
- SQLsmith - fuzzing style tool
Engine Correctness
Here we mainly refer to the case where the engine returns wrong query result.
Research Papers
What You Really Want to Know About
- RAGS with system-differential testing Massive Stochastic Testing of SQL
- Data generation with contraints sovling Query-Aware Test Generation Using a Relational Constraint Solver
- Differential testing for the optimizing rules A framework for testing query transformation rules
- Criteria on integrtity testing The Effectiveness of Test Coverage Criteria for Relational Database Schema Integrity Constraints
What You Want to Know About
- Parallelism on the old machines Quickly generating billion-record synthetic databases
- Verified database system Toward a Verified Relational Database Management System
- Reply the transactions Debugging transactions and tracking their provenance with reenactment
What's New
- Search-based test data generation for SQL queries [2018]
- Automated verification of query equivalence using satisfiability modulo theories [2019]
- Detecting Optimization Bugs in Database Engines via Non-Optimizing Reference Engine Construction [2020]
- Testing query execution engines with mutations [2020]
Tools
- SQLancer@ETH_ZURICH
- go-sqlancer@PingCAP
Performance
Here we focus on detecting and debugging performance issues.
Research Papers
What You Really Want to Know About
- Very first system Efficient testing of high performance transaction processing systems
- Is the exact estimation possible Exact Cardinality Query Optimization for Optimizer Testing
- How to identify issues on plan generation On the stability of plan costs and the costs of plan stability
- What's wrong with the models Testing cardinality estimation models in SQL server
What You Want to Know About
- Generate and compare the plans Testing the accuracy of query optimizers
What's New
Tools
- TPC Benchmarking
- APOLLO@gatech
React
Typescript
Django
Laravel
Facebook
Microsoft
Google
Alibaba
D3
Tencent