Our demo paper "Unshackling Database Benchmarking from Synthetic Workloads" has been accepted at the 39th IEEE International Conference on Data Engineering (ICDE 2023)!
The Problem
Introducing new (learned) features into a DBMS requires considerable experimentation and benchmarking to avoid regressions in production workloads. Using standard benchmarks such as TPC-H and TPC-DS is common practice, but unfortunately, these do not represent the complexity of real production workloads.
Our Solution
We propose a technique that generates a synthetic dataset from query logs and metadata — without touching the original data. This approach enables:
- Realistic workload characteristics
- Privacy-preserving benchmarking
- Better evaluation of learned database components
Practical Impact
This tool addresses a critical gap in database research: the ability to benchmark systems on workloads that actually represent production complexity, rather than artificial benchmarks that learned systems can easily overfit to.
This is joint work in collaboration with Microsoft Research and MIT: Parimarjan Negi (MIT), Mohammad Alizadeh (MIT), Tim Kraska (MIT), Jyoti Leeka (Microsoft Research), Anja Gruenheid (Microsoft Research), and Matteo Interlandi (Microsoft Research).