Docs

Message Queue for Kafka

Product Overview

What is Message Queue for Kafka

Message Queue for Kafka is a distributed messaging middleware service that is built based on Apache Kafka. It features high throughput and scalability and provides streaming data publishing/subscription and multi-replica storage. It is widely used in log compression and collection, stream data processing, message decoupling, and traffic peak shaving scenarios. Message Queue for Kafka is ready to use out of the box. You do not have to modify business code. It allows you to focus more on rapid business development while alleviating complex deployment and maintenance.

Product functions

Efficient message transmission: Even with massive message backlogs, Message Queue for Kafka maintains high throughput for sending and receiving messages of Kafka cluster. To improve data operation and maintenance, Kafka instance supports reconsuming consumed messages or clearing up the message backlogs to maintain normal operation.

Cluster deployment: Supports cluster deployment and provides multi-replica redundant storage to ensure high service and data availability.

Monitoring and alarm: Tracks message production and consumption in real time. Monitors consumption latency and message backlog with alarms to help you identify issues promptly.

Application scenarios

Stream computing processing

In finance and scientific computing, large data volume and strong time-efficiency require faster computation and analysis. Traditional data processing structure that collects data first and then stores it in databases for query and analysis later can no longer meet these needs. By working with streaming engines like Flink, the message queue for Kafka can compute and analyse real-time data based on business needs. It can timely deliver analysis results to the next broker.

Traffic peak shaving

In e-commerce systems or large websites, processing capabilities vary among systems. During major events like flash sales or new product launches, sudden spikes from upstream systems may overwhelm downstream systems with lower capacity. Without protection or caching mechanisms, this can cause system overload or crashes, resulting in data loss. The message queue for Kafka can temporarily buffer sudden request spikes from upstream systems when traffic surges occur. This allows downstream systems to process requests based on their actual consumption capacity and enables them to run smoothly and stably within safe limits. This prevents downstream subscriber systems from crashing due to sudden traffic bursts.

Log analysis

Log analysis decouples applications from analytics systems by building a bridge between application systems and log analysis systems. Logs that are generated by multiple application hosts are sent to Kafka cluster in bulk and asynchronously. Log analysis systems can then fetch these logs anytime by using Hadoop or other systematic storage and analysis tools for statistical analysis. The low-latency feature of message queue for Kafka ensures that business operations remain unaffected during log collection. Compared with open-source Kafka, it offers stronger persistence and lower end-to-end latency under the same performance conditions.

Last updated: 2025.08.08 16:58:35