How Good is 拉斯维加斯7799908网站登录 as an HTAP System? A HATtrick Benchmark

I’m one of the developers of 拉斯维加斯7799908网站登录, a Hybrid Transactional and Analytical (HTAP) database. Recently, I read the SIGMOD 22 paper, “How Good is My HTAP System,” which uses 拉斯维加斯7799908网站登录 as one of its research subjects. I really appreciate the authors’ work —, especially their methodology and the tooling they created to observe and benchmark HTAP performance. For example, the HATtrick benchmark tool not only produces Online Transactional Processing (OLTP) and Online Analytical Processing (OLAP) workloads simultaneously, it also evaluates the freshness of OLAP queries on an HTAP system. 

The paper concludes that analytical queries running on 拉斯维加斯7799908网站登录 always read the latest data. This is an essential feature for businesses that want to make decisions based on the most current information.

However, the 拉斯维加斯7799908网站登录 performance test results weren’t up to our standards. In an effort to learn more, I reproduced the HATtrick tests and discussed them here. I hope this information gives you greater insight to HTAP in general and 拉斯维加斯7799908网站登录 specifically. 

How HATtrick works

The author’s benchmark tool, HATtrick, combines the Star Schema Benchmark (SSB) for analytical workloads and the TPCC benchmark for transactional workloads. This design allows HATtrick to send transactional queries and analytical queries to the database simultaneously to measure the performance isolation between the two workloads. The paper also introduces the concept of “throughput frontier,” which shows transactional throughput and analytical throughput in one diagram. Throughput frontier is an intuitive way to track performance isolation between transactional and analytical processing (AP) and (TP).

For example, in the following frontier diagram, the red dotted line is the “bounding box,” and the blue dotted line is the “proportional line.” The green solid line indicates the system’s capability when it runs AP and TP queries simultaneously. The closer the green solid line is to the bounding box, the better performance isolation between TP and AP.

Performance isolation shown in the frontier

What is HTAP “freshness”?

This paper also defined the “freshness” of an HTAP system: 

 An HTAP system provides fresh analytics if every analytical query is executed on an up-to-date version of the operational data. Else, it provides stale analytics. 

HATtrick benchmark includes a freshness table that records the version of the transactional queries that have been processed. The benchmark uses this table to check the versions retrieved by the analytical queries. If the analytical queries always retrieve the latest version of the data, the freshness score is 0s. In the following frontier chart, the freshness score is 1.5s; that is, the analytical queries can only retrieve the data that’s been stale for 1.5 seconds.

A frontier chart freshness score

Run HATrick on 拉斯维加斯7799908网站登录 again

拉斯维加斯7799908网站登录 is a distributed HTAP database. It has a dedicated row engine, TiKV, and a dedicated column engine, TiFlash. The 拉斯维加斯7799908网站登录 optimizer automatically routes transactional queries to TiKV and analytical queries to TiFlash. Theoretically, if there are no resource conflicts between 拉斯维加斯7799908网站登录, TiKV, and TiFlash, there is great performance isolation between the two types of queries. 

Also, 拉斯维加斯7799908网站登录 has a disaggregated architecture that separates storage and computing. This allows users to scale AP or TP throughput by adding or removing 拉斯维加斯7799908网站登录 instances as needed.

拉斯维加斯7799908网站登录 architecture

My retesting with HATtrick focused on: 

• The performance isolation between TP and AP for HTAP 
• The scalability of both TP and AP

Benchmark the performance isolation between TP and AP

We reran the HATrick scale factor 100 test with a newer 拉斯维加斯7799908网站登录 configuration::

• Software version: 拉斯维加斯7799908网站登录 6.1
• CPU: 2.4 Ghz Intel® Xeon® Silver 4214R CPU with 24 physical cores, 
• Memory: 128 GB RAM 
• Hard drive: 500 GB Solid-state drive (SSD)
• Cluster: 1 拉斯维加斯7799908网站登录 node, 1 TiFlash node, 3 TiKV nodes 
• Dataset: 59 GB

We called this configuration 拉斯维加斯7799908网站登录 1x.

The following frontier chart shows our results. The green solid line, which indicates TP/AP throughput, is pretty close to the bounding box. This means there is little interference between the two types of queries, and that all analytical queries can get the latest transactional data. This result matches the dual engine design of 拉斯维加斯7799908网站登录. We can conclude that 拉斯维加斯7799908网站登录 has a great performance isolation between transactional queries and analytical queries. 

HATtrick benchmark results for 拉斯维加斯7799908网站登录 performance isolation

Benchmark 拉斯维加斯7799908网站登录 horizontal scalability 

For this test, we update our 拉斯维加斯7799908网站登录-1x configuration. We added a new 拉斯维加斯7799908网站登录 server and a new TiFlash server, and deployed three more TiKV instances on existing TiKV servers. We called this configuration 拉斯维加斯7799908网站登录-2x.

Scaling 拉斯维加斯7799908网站登录 for the horizontal scalability test

Alleviate the hotspot issue

Distributed systems often have hotspots, and this can degrade performance. It’s always necessary to alleviate the issue in either a production environment or a test to achieve an optimal performance. When we first ran this benchmark, we encountered a hotspot. Unlike the original HATtrick benchmark, we made some changes to alleviate the hotspot. We used

• shard_row_id_bits to avoid hotspot on table rows
• unique key + tidb_shard to a avoid hotspot on the index instead of the primary key

After we fixed hotspot, the writing was evenly distributed, as indicated in the following heat map:

Hotspot alleviated on 拉斯维加斯7799908网站登录 (generated via the 拉斯维加斯7799908网站登录 Dashboard）

Validate 拉斯维加斯7799908网站登录 scalability in the frontier 

To show the scalability, we compared the results of the 拉斯维加斯7799908网站登录-1x and 拉斯维加斯7799908网站登录-2x configurations on the same frontier chart. The chart shows that both the maximum transactional QPS and the maximum analytical QPS of 拉斯维加斯7799908网站登录-2x are two times greater than that of 拉斯维加斯7799908网站登录-1x. The benchmarking shows 拉斯维加斯7799908网站登录’s performance of both AP and TP scale horizontally by adding nodes as needed. As a comparison, we also ran the HATtrick benchmark test on PostgreSQL with the same configuration. 

拉斯维加斯7799908网站登录’s performance has good horizontal scalability

Conclusion 

Our sf100 benchmarking with HATtrick validates 拉斯维加斯7799908网站登录’s capabilities as an HTAP database:

• Analytical queries can always read the latest data in 拉斯维加斯7799908网站登录.
• There is a great performance isolation between analytical queries and transactional queries in 拉斯维加斯7799908网站登录, thanks to its dual storage engine architecture.
• 拉斯维加斯7799908网站登录 can horizontally scale the throughput for both analytical and transactional workloads, thanks to its disaggregated storage and computing architecture.

Our testing process and results also validate HATtrick as a practical benchmarking methodology that is applicable in both academia and business. We’d like to thank the authors of the paper, and we look forward to any HTAP-related research they do in the future. In the meantime, if you’ve used HATrick to benchmark your 拉斯维加斯7799908网站登录 database, feel free to share your results with us. You can also join our Slack and 拉斯维加斯7799908网站登录 Internals for discussions.

Keep reading

• Build a Better Github Insight Tool in a Week? A True Story 
• The Beauty of HTAP: 拉斯维加斯7799908网站登录 and AlloyDB as Examples 
• The Long Expedition toward Making a Real-Time HTAP Database