Docs

Message Queue for Kafka

SDK Reference

Overview

Message Queue for Kafka in VolcanoEngine fully complies with the open-source Kafka protocol and supports multi-language SDKs. Clients can access Message Queue for Kafka through both VPC networks and public network access to send and receive messages. Message Queue for Kafka provides sample projects for message production and consumption, as well as reference documentation. Based on these sample projects, this document introduces the main process for using a client to access VolcanoEngine Message Queue for Kafka to send and receive messages.

SDK types

Message Queue for Kafka instances are fully compatible with the open-source Kafka protocol, and you can connect directly using the open-source Kafka client.

SDK types	Usage and description
Apache Kafka client SDK	Message Queue for Kafka is fully compatible with open-source Apache Kafka and supports producing and consuming messages through the open-source Kafka SDK. An open-source Kafka SDK is typically integrated into client applications to manage resources and data in Kafka instances, such as producing and consuming messages. Message Queue for Kafka provides sample projects in multiple languages, such as Go and Java, for your reference. These projects include sample code for message production and consumption under various mechanisms. For more information, see: Kafka Java client SDK Kafka Go client SDK Kafka Python client SDK Kafka C++ client SDK
Volcengine OpenAPI SDK	VolcanoEngine provides multi-language SDKs for the control plane, allowing you to call the OpenAPI of various VolcanoEngine services. By calling API endpoints, you can manage cloud resources while eliminating the need for manual signature and authentication processes, enabling rapid integration with cloud services. Message Queue for Kafka is now integrated with the VolcanoEngine OpenAPI SDK. You can refer to the following documentation to install and use the Volcengine SDK. Go SDK overview Python SDK overview Java SDK overview

Restriction description

Category	Restriction description
Produce message	For latency-sensitive messaging scenarios, it is recommended to optimize sending by configuring linger.ms=0. When message sending fails, a retry mechanism is required. You can configure the number of retry attempts using the retries parameter. It is recommended to set the number of retries to three. Produced callback functions must not block; otherwise, this will cause blocking of client message sending and other issues. It is recommended to reserve sufficient JVM memory on the production side to prevent send blocking and other issues caused by insufficient memory.
consume messages	In special scenarios, Kafka may experience duplicate consumption. The business logic must ensure idempotency in message processing. It is recommended to set the local cache size for messages pulled by consumers appropriately to avoid out-of-memory (OOM) errors. When a consumer thread exits, you should call the consumer's close method to prevent other consumers in the same group from being blocked for the duration of session.timeout.ms. If the consumer's owner thread exits abnormally, the client may fail to initiate consumption requests, which can lead to blocked consumption and other problems. Therefore, it is necessary to ensure the stability of the owner thread. When consuming messages using the long connection poll mode, immediately closing the consumer channel after consumption ends will cause frequent rebalancing, thus blocking the progress of consumption. It is recommended that consumers set the periodic polling interval to 200 milliseconds to maintain the heartbeat state on the server and prevent frequent connections and disconnections caused by heartbeat timeout, which can block the progress of consumption. You should set an appropriate value for the consumer's session based on business requirements. For example, you can set it to 30 seconds using the following parameter: session.timeout.ms=30000. It is recommended that the number of consumers does not exceed the number of partitions in the topic; otherwise, some consumers may not be able to retrieve messages. It is recommended to commit messages only after they have been processed. Otherwise, if business message processing fails, you will not be able to retrieve the failed messages.

Last updated: 2025.09.16 10:31:26