Is the TPC-DS Performance Testing Framework Still Relevant in Today's Emerging Architectures?

I had an interesting question from a customer the other day: "Is the performance testing framework TPC-DS still relevant with today's emerging architectures?"

For those who haven't come across it, the DS version of the great TPC benchmarks (TPC-DS), created in 2006, specifically tests relational Data Warehouse implementations. Today, there are many new architectural approaches like Data Mesh, Data Fabric, and pre-aggregation (e.g., the excellent Apache Pinot) that take different approaches. Companies are transitioning to these to satisfy a more varied set of analytics requirements.

So, do we need a new set of benchmarks that test business outcomes rather than specific engineering aspects?

One example could be something akin to Robot Wars, where different designs are tested with a common set of outcomes, based on defined business use-cases, in aspects like BI Metrics, Machine Learning, etc.

Metrics could include:

  • Latency (real-time and otherwise)
  • Scale of requests
  • Ability to accept change
  • Speed of onboarding new data to insight
  • Ability to accept different analytics and data structures (e.g., graph, etc.)

My thoughts? Both yes and no.

The "Yes" Part

I would still use TPC workload testing for evaluating the various instance types and to check if I can purchase a provisioned rather than on-demand instance type to save money on instance uptimes and burst capacities. TPC-DS remains useful for traditional data warehousing environments and for comparing the performance of various relational database systems under standardized conditions.

The "No" Part

In the past, there were huge Data Warehousing (DW) and Online Transaction Processing (OLTP) systems with monolithic queries. Today, however, data mesh and API-driven architectures are challenging data requests to seconds based on business use cases. For these modern architectures, TPC-DS cannot fully capture the performance nuances.

A Personal Recollection

This topic brought back memories of the days I used to run HammerDB on Oracle 11g database to test the performance of purchased hardware. It was an essential tool back then, and while TPC-DS has its place, the landscape is evolving.

Closing Thoughts

In my humble opinion, other frameworks that test different processing models, such as IoT and transaction processing, are still incredibly relevant. As the data landscape continues to evolve, so too must our benchmarks and performance testing frameworks.

What are your thoughts on this? Do you think TPC-DS still holds value, or is it time to develop new benchmarks for today's diverse data architectures? Let's discuss!