Kafka vs. RocketMQ- Multiple Topic Stress Test Results

简介: The focus of this post is on Kafka and RocketMQ, since they both are high-performance applications. Thus, we will target these two products to see which one of them is better in the above scenarios.

BecomeaDataAnalystinFiveClosingTactics

Introduction


This post delves into how we will simulate the real scenarios below:


  • Message sending and subscription must be concomitant.
  • Multiple subscription ends supported to subscribe only to messages you are interested in.

The focus of this post is on Kafka and RocketMQ, since they both are high-performance applications. Thus, we will target these two products to see which one of them is better in the above scenarios. Before we begin, let’s clarify two concepts first:


What is a Topic?


Topic is an important concept in message-oriented middleware. Every topic represents a message category. With multiple topics, we can categorize and isolate messages.


You can refer to the feeding models in a zoo in the figure below. Every animal can only consume the specific corresponding food.


1

What is a Partition?


Both Kafka and RocketMQ adopt message queues on disks. For the same consumption group, a partition only supports message consumption by one consumption thread. Too few partitions may cause the consumption speed to lag far behind the message generation speed. In actual production environments, a topic will be set in the multi-partition mode to support multiple consumers, as shown in the figure below:


2

Many topics and environments exist in an actual production environment of internet enterprises. Such production environment requires the message-oriented middleware to guarantee the service stability in the coexistence of multiple topics. Let's move on to the testing step to check the handling capacities of Kafka and RocketMQ on multiple topics when message sending and subscription ends coexist.


Testing Objective


We will compare the impact of the number of topics on the performances of Kafka and RocketMQ when sending and receiving ends coexist. In this case, we adopt eight partitions. However, in this stress test, we only pay attention to the performance indicators of the service end. Therefore, the “Exit” criteria for the stress test are as given below.


Keep increasing the stress on the sending end until the system throughput does not increase and the response takes longer. At this time, a performance bottleneck emerges on the service end. Obtain the corresponding optimal throughput of the system, ensuring that there are no messages accumulated throughout the process.


Test scenarios


By default, every topic has eight partitions; every topic matches one subscriber, and the number of topics is increased gradually. The data obtained is as follows:


3

We can see that no matter how many topics there are, both Kafka and RocketMQ can maintain flat TPS on the sending and receiving ends. Simply put, no message accumulation occurs.

Based on the changes in number of topics, we can draw the curves of message handling capacities of Kafka and RocketMQ, as shown in the figure below:

4

From the figure above we can see that:

  • When the number of topics increases from 64 to 256, the throughput of Kafka dropped by 98.37%.
  • When the number of topics increases from 64 to 256, the throughput of RocketMQ dropped by only 16%.

Why the huge difference? The difference is attributable to the fact that every topic and partition of Kafka correspond to one physical file. When the number of topics increases, the policy of deconcentrated storage of messages to disks will lead to disk IO competition to cause performance bottlenecks. In contrast, all the messages in RocketMQ are stored in the same physical file. The number of topics and partitions is just a logic division for RocketMQ. So the increasing number of topics won't generate a huge impact on the RocketMQ performance.

Test conclusion

When the message sending and consumption ends coexist, the increasing number of topics will cause a drastic decline of Kafka's throughput, while RocketMQ delivers a stable performance. Therefore, Kafka is more suitable for business scenarios with only a few topics and consumption ends, while RocketMQ is a better choice for business scenarios with multiple topics and consumption ends.

Appendix:

Test environment

The service end is deployed in the standalone mode. The server configurations are as follows:

5

Application version:

6

Test scripts

7

Summary

In the test above, we saw how RocketMQ almost overwhelms Kafka. The result is hardly surprising because RocketMQ is born to target internet production requirements. Readers now should understand why RocketMQ manages to support the massive messaging services of Alibaba group.

The stress test of multiple-topic scenarios involved in the test only lasted 20 minute. Too short an execution period for a message-oriented middleware product, hence inadequate to identify its stability.

目录
相关文章
|
11天前
|
消息中间件 存储 负载均衡
深入了解Kafka中Topic的神奇之处
深入了解Kafka中Topic的神奇之处
40 0
|
11天前
|
消息中间件 存储 Kafka
RabbitMQ、RocketMQ和Kafka全面对决,谁是最佳选择?
1、应用场景 1.RabbitMQ: 适用于易用性和灵活性要求较高的场景 异步任务处理:RabbitMQ提供可靠的消息传递机制,适用于处理异步任务,例如将耗时的任务放入消息队列中,然后由消费者异步处理,提高系统的响应速度和可伸缩性。 解耦系统组件:通过使用RabbitMQ作为消息中间件,不同的系统组件可以通过消息进行解耦,实现松耦合的架构,提高系统的可维护性和灵活性。 事件驱动架构:RabbitMQ的发布-订阅模式可以用于构建事件驱动架构,将系统中的事件作为消息发布到相应的主题,不同的消费者可以订阅感兴趣的主题进行相应的处理。
226 2
|
11天前
|
消息中间件 分布式计算 监控
Python面试:消息队列(RabbitMQ、Kafka)基础知识与应用
【4月更文挑战第18天】本文探讨了Python面试中RabbitMQ与Kafka的常见问题和易错点,包括两者的基础概念、特性对比、Python客户端使用、消息队列应用场景及消息可靠性保证。重点讲解了消息丢失与重复的避免策略,并提供了实战代码示例,帮助读者提升在分布式系统中使用消息队列的能力。
44 2
|
10天前
|
消息中间件 前端开发 Java
java面试刷题软件kafka和mq的区别面试
java面试刷题软件kafka和mq的区别面试
|
11天前
|
消息中间件 Java Kafka
MQ产品使用合集之对于Kafka作为数据源的情况,官方比较推荐哪种使用方式
消息队列(MQ)是一种用于异步通信和解耦的应用程序间消息传递的服务,广泛应用于分布式系统中。针对不同的MQ产品,如阿里云的RocketMQ、RabbitMQ等,它们在实现上述场景时可能会有不同的特性和优势,比如RocketMQ强调高吞吐量、低延迟和高可用性,适合大规模分布式系统;而RabbitMQ则以其灵活的路由规则和丰富的协议支持受到青睐。下面是一些常见的消息队列MQ产品的使用场景合集,这些场景涵盖了多种行业和业务需求。
24 0
|
11天前
|
消息中间件 监控 负载均衡
rabbitmq与kafka的区别
RabbitMQ提供了强大的可靠性保障,通过持久化机制和消息确认机制来确保消息的可靠传输和消费。而Kafka也提供了类似的可靠性保障,但其持久化机制和消息确认机制的实现方式与RabbitMQ有所不同。
|
11天前
|
消息中间件 负载均衡 监控
【Kafka】Kafka 创建Topic后如何将分区放置到不同的 Broker 中?
【4月更文挑战第13天】【Kafka】Kafka 创建Topic后如何将分区放置到不同的 Broker 中?
|
11天前
|
消息中间件 存储 缓存
【Kakfa】Kafka 的Topic中 Partition 数据是怎么存储到磁盘的?
【4月更文挑战第13天】【Kakfa】Kafka 的Topic中 Partition 数据是怎么存储到磁盘的?
|
11天前
|
消息中间件 负载均衡 监控
Kafka消费者:监听模式VS主动拉取,哪种更适合你?
Kafka消费者:监听模式VS主动拉取,哪种更适合你?
74 0
|
11天前
|
消息中间件 存储 Kafka
【深入浅出 RocketMQ原理及实战】「底层源码挖掘系列」透彻剖析贯穿一下RocketMQ和Kafka索引设计原理和方案
【深入浅出 RocketMQ原理及实战】「底层源码挖掘系列」透彻剖析贯穿一下RocketMQ和Kafka索引设计原理和方案
61 1

热门文章

最新文章