Performance Test on Cloud-native Databases

Hello，I'm binghe~~

In the past few years, we have seen explosive growth in the cloud database market, with fierce competition between service providers aiming for the top. Notable contenders in this space include AWS Aurora, Alibaba Cloud PolarDB, and Huawei GaussDB.

Having been a MySQL practitioner for many years and published A Practical Guide for MySQL Development, Optimization, and O&M, I frequently receive messages from database administrators and architects at companies who are ready to make the leap to cloud, asking for advice on selecting cloud database services. To make things simple, we can make reliable choices around two major indicators – stability and performance. In this article, I'll test and compare the performance of five database services: PolarDB, ApsaraDB for OceanBase, Aurora, GaussDB, and TDSQL-C. The tests will be performed on these databases with the default configurations to provide a baseline reference.

Abstract

In the low-specification category (8 cores, 64 GB), PolarDB outperforms its contenders across the board in both CPU-bound and I/O-bound tests.

In the high-specification category (64 cores, 512 GB), PolarDB also leads with considerable margin. Largely outperforming its contenders in both CPU-bound and I/O-bound tests, PolarDB has demonstrated its edge in today's cloud database offerings.

It is also worth mentioning that in both categories, we can see an obvious deterioration in performance on Aurora databases in read-write and write-only scenarios.

Figure 2. Test results of the instances in the low-specification category (8 cores, 64 GB)

Scope

The following MySQL-compatible database services are tested:

• PolarDB by Alibaba Cloud
  
• Aurora by Amazon Web Services
  
• ApsaraDB for OceanBase by Ant Group
  
• GaussDB by HUAWEI CLOUD
  
• TDSQL-C by Tencent Cloud
  

Controlled Variables

Public cloud database instances are purchased from the cloud service providers and tested out-of-the-box, with no post-configuration. For this test, database instances are divided into two categories, the low-specification category (8 cores, 64 GB), and the high-specification category (64 cores, 512 GB). All instances run MySQL 8.0 and are deployed in the primary/secondary architecture.

Note: ApsaraDB for OceanBase does not offer a 64-core, 512 GB instance type. The 62-core, 400 GB instance type is used instead.

Testing Configurations

CPU-bound and I/O-bound tests are performed on the database services by using the same scripts for each test.

The test results are returned in the following format:

Case Database name, number of tables, table sizes, number of threads, cloud server name, QPS

Where:

• Case: the name of the test script used, which is indicative of the read/write mode. Test scripts used: oltp_read_only, oltp_read_write, and oltp_write_only.
• Database name: the name of the database. Databases used in CPU-bound tests are named tpk_s, while those in I/O-bound tests are named tpk_l.
• Number of tables: the number of tables in each database. Ten tables are specified for each database.
• Table sizes: the number of rows in each table. In CPU-bound tests, each table contains 10 million rows of data. In I/O-bound tests, the table sizes vary based on the category. Tables in the low-specification category contain 40 million rows each, and tables in the high-specification category contain 300 million rows each.
• Number of threads: the number of threads used to perform the test. The number of threads used vary based on the category of the tests. Tests on the low-specification category involve a minimum of 1 thread and a maximum of 128 threads. Tests on the high-specification category involve a minimum of 1 thread and a maximum of 300 threads.
• Cloud server name: the name of the cloud server instance where the database is deployed. Test scripts are not stored in this cloud server.
• QPS: the QPS value, grouped by the number of threads.

The Comparative Analysis section provides side-to-side comparisons of the performance metrics outlined in the preceding section. Raw results from the tests are included in the Tests results section, and are grouped by services.

Comparative Analysis

Low-specification category (8 cores, 64 GB)

The tables in this section provide side-to-side comparisons for databases in the low-specification category.

CPU-bound tests

1）oltp_read_only

2）oltp_read_write

3）oltp_write_only

The CPU-bound test results show that PolarDB outperforms its contenders in every test metric.

I/O-bound tests

1）oltp_read_only

2）oltp_read_write

3）oltp_write_only

In the I/O-bound tests, PolarDB also emerged as the clear winner in every test metric.

High-specification category (64 cores, 512 GB)

The tables in this section provide side-to-side comparisons for databases in the high-specification category.

CPU-bound tests

1）oltp_read_only

2）oltp_read_write

3）oltp_write_only

The CPU-bound test results show that PolarDB outperforms its contenders in every test metric.

I/O-bound tests

1）oltp_read_only

2）oltp_read_write

3）oltp_write_only

In the I/O-bound tests, PolarDB also emerged as the clear winner in every test metric, with obvious performance gains compared to its contenders.

Test Results

This section contains the raw results of the tests performed in this article. To ensure the fairness of the tests, all tests are performed on out-of-the-box offerings of different service providers. Controlled variables and testing configurations used for these tests can be found in their corresponding sections.

Low-specification category (8 cores, 64GB)

PolarDB

1）CPU-bound tests

2）I/O-bound tests

ApsaraDB for OceanBase

1）CPU-bound tests

2）I/O-bound tests

Aurora

1）CPU-bound tests

2）I/O-bound tests

GaussDB

1）CPU-bound tests

2）I/O-bound tests

TDSQL-C

1）CPU-bound tests

2）I/O-bound tests

High-specification category (64 cores, 512 GB)

PolarDB

1）CPU-bound tests

2）I/O-bound tests

ApsaraDB for OceanBase

1）CPU-bound tests

2）I/O-bound tests

Aurora

1）CPU-bound tests

2）I/O-bound tests

GaussDB

1）CPU-bound tests

2）I/O-bound tests

TDSQL-C

1）CPU-bound tests

2）I/O-bound tests

Well, let's call it a day. I'm binghe. See you next time~~

收录于合集 #架构师进阶系列

10个

下一篇复盘：我在真实场景下对几款主流云原生数据库进行极限性能压测的一次总结！！（建议收藏）