Our workshop paper "Towards a Benchmark for Learned Systems" has been accepted at SMDB (Self-Managing Database Systems Workshop)!
Due to COVID-19, SMDB 2021 will be held online as part of ICDE 2021 (originally planned for Chania, Crete, Greece). I will present the work via recorded video.
Abstract
The paper initiates discussion on benchmarking data management systems incorporating machine learning. Traditional benchmarks like TPC and YCSB prove inadequate for evaluating learned systems because they assess performance under stable conditions, whereas learned systems continuously adapt to changing workloads and environments.
We propose new metrics better suited to evaluate these systems while ensuring comparable results across diverse deployments.
Key Challenge
Standard cost-per-performance metrics fail to account for essential trade-offs related to the training cost of models and the elimination of manual database tuning.
Our work addresses this gap by proposing benchmarking methodologies that capture the true value proposition of learned components in database systems.