"Learned" systems can automatically adapt to new workloads, data, or hardware without extensive human tuning, potentially reducing analytics costs significantly. However, traditional benchmarks like TPC or YCSB prove insufficient because they use stable workloads and data distributions that these systems can easily overfit to.
Companies are often reluctant to incorporate learned techniques in mainstream systems due to a lack of evidence of their effectiveness under real-world conditions.
Key Proposals
Our work suggests new benchmarking approaches should:
- Avoid fixed workloads and data distributions — learned systems can overfit to static benchmarks
- Measure adaptability through descriptive statistics and outliers rather than averages
- Incorporate model training costs and savings into results
- Better characterize performance under varying conditions
Standard cost-per-performance metrics fail to account for essential trade-offs related to the training cost of models and the elimination of manual database tuning.
We are implementing these concepts in a new benchmarking suite designed specifically for evaluating learned database components.