Docs

Message Queue for Kafka

Topic management

Create Topic

A Topic (message subject) is a collection of messages of the same type. It serves as the basic unit for data writing operations in Message Queue for Kafka. This topic describes the steps to create a single Topic.

Background information

In real business scenarios, a Topic often carries the same type of business traffic. Developers decide how to design different Topics based on their system design and data architecture. Each Topic can receive messages sent by multiple producers. It can also allow multiple consumers to consume messages.

Partition is the physical grouping of a Topic. Each Topic can be divided into multiple partitions. Each partition is an ordered queue. When creating a Topic, you must specify the number of partitions. As business traffic grows, you can increase the number of partitions anytime to expand the Topic's capacity.

Message Queue for Kafka controls Topic creation methods supported by Kafka instances through the automatic Topic creation function. By default, this function is disabled.

When disabled, Topics can only be created via the console.

When enabled, you can create Topics through the console or directly specify data to write to a non-existent Topic when generating messages. In this case, the Kafka backend service automatically creates the Topic for you.

tip

Each specification instance provides a certain range of partition numbers and a number of free partitions. If the free partition quota is used up, you need to purchase partitions to continue creating or expanding Topics. If the number of partitions reaches the maximum limit of the specification, you must adjust the partition specification to obtain a larger partition quota before continuing to create or expand Topics.

Prerequisites

A Message Queue for Kafka instance is created, and the instance status is Running. For operation steps, please refer to Create instance.

Create Topic manually

Message Queue for Kafka supports manually creating Topics one by one through the console or OpenAPI.

tip

Once the Topic is created, the Topic name can not be changed. The number of partitions can only be increased, not reduced. Please plan the number of partitions reasonably when creating or modifying.

In scenarios such as cloud migration or instance backup, if multiple Topics need to be created at once, you can batch create Topics by uploading a configuration file. For the operation steps, refer to Batch import Topic.

Create Topic through the console

Select the region where the instance is located in the top menu bar.

On the Instances page, click the specified instance name.

Click the Topic management tab.

Click Create Topic.

Complete the basic configuration for the Topic.

Parameters	Description
Topic name	The name of the Topic. The name must comply with the following naming rules: 3 to 249 characters. Supported characters include English letters, numbers, hyphens (-), dots (.), and underscores (_). Must start with an English letter or number. Can not duplicate an existing Topic name. warning Once created, the Topic name can not be modified, so proceed with caution.
Description	Notes for the topic. It can be used to briefly describe the topic's purpose, corresponding business types, and other information.

Complete the parameter configuration for the Topic.

Parameters	Description
Number of partitions	The number of partitions in this topic. The greater the number of partitions, the higher the consumption concurrency. The default value is 12. The range is 1 to 300 and can not exceed the remaining available partitions of the corresponding Kafka instance. tip When modifying topic configurations, this parameter only supports increasing, not decreasing. The partitions of a topic are distributed internally within the Kafka instance. To ensure even distribution of partitions within the Kafka instance, it is recommended to set the number of topic partitions as an integer multiple of the number of corresponding Kafka instance brokers. You can refer to View node information to check the number of brokers in the Kafka instance.
Number of replicas	The number of partition replicas. Data is backed up on the corresponding number of replicas to ensure high availability of the partition. Data availability is ensured even if one broker fails. The higher the number of replicas, the greater the reliability. The number of replicas can be set to 1, 2, or 3. The default value is 3. tip Increasing the number of replicas consumes instance disk space. Please ensure sufficient capacity. Adding replicas consumes instance bandwidth. We recommend that you operate during off-peak hours. Reducing the number of replicas decreases data availability. Proceed with caution. Replica Topic 1 can not guarantee data reliability and can not perform read/write operations during instance upgrades. We recommend that you choose replica 2 or 3.
Minimum in-sync replicas	Topic minimum in-sync replicas, i.e., the min.insync.replicas parameter. The larger this value, the better the data reliability, but the availability will decrease. When the number of replicas is 1, the minimum in-sync replicas can only be 1. When the number of replicas is 2 or 3, the minimum in-sync replicas can not exceed the number of replicas minus 1.
message retention period	Maximum retention time for messages in this topic when disk space is sufficient, i.e., the message retention period. Consumers need to consume messages before this period ends. Otherwise, messages will be deleted and can not be consumed. The unit is hours, with a range of 1 to 2160. The currently supported maximum retention period is 90 days. By default, it is the instance-level message retention period. You can also specify the message retention period for this topic here, which only applies to this topic.
Maximum message size	The maximum length of a single message in this topic. The value range is 1MB to 12MB. By default, it is the instance-level maximum message size. You can specify the maximum message size for this topic here, which applies only to this topic.

Configure user permissions.

Assign SASL user permissions for the Topic within the specified instance.

By default, the user permission is set to All allowed. That is, all SASL users have permissions to read and write this Topic, even if the user has disabled All Permitted permission. You can also choose custom settings to assign different permission types for different SASL users. Custom permissions include:

Read only: has permission to subscribe to messages but not to produce messages.

Write only: has permission to produce messages but not to subscribe to messages.

Read and write: has both permissions to produce and subscribe to messages.

tip

Custom settings only take effect on SASL users who have disabled All Permitted.

When creating a Topic, if custom permissions are set, Message Queue for Kafka will simultaneously create corresponding ACLs and display them in the ACL list. If you need to modify custom permissions after creating a Topic, we recommend that you proceed through ACL operations. For details, please refer to Create ACL.

Configure message cleanup policy.

DELETE: the default message cleanup policy. When disk space is sufficient, messages within the maximum retention period are retained. When disk space is insufficient, previous messages will be deleted in advance to ensure service availability.

tip

The default disk cleanup threshold for Kafka instances is 90%. The actual threshold is based on your actual use. For details, see Set disk cleanup threshold.

COMPACT: The COMPACT message cleanup policy integrates messages according to their Keys. For messages with the same Key, only the latest value is retained, and previous records are removed.

warning

If the message does not have Key, COMPACT message cleanup policy is not supported.

DELETE, COMPACT: both DELETE and COMPACT message cleanup policies are configured. Messages will be deleted once they meet any one of the cleanup policies.

(optional) Configure topic tags.

Click Add tag, enter the tag key and value to add custom tags to the topic.

Tags identify and classify cloud resources. Adding tags helps recognize and manage topics. You can add up to 20 tags each time. For tag setting rules, refer to Tag setting rules.

Click Create.

You can view created topics on the Topic management tab.

Create topics through OpenAPI

Message Queue for Kafka provides the OpenAPI CreateTopic for manually creating topics.

Automatically create Topics

Before using the auto-create topic function, enable it in the console and specify a non-existent topic when writing messages via the client SDK. The default configuration for auto-created topics is as follows.

Parameters	Default configuration
Topic name	By default, uses the Topic name specified by the client SDK. Must meet the following rules: 3 to 249 characters. Supported characters include English letters, numbers, hyphens (-), dots (.), and underscores (_). Must start with an English letter or number. Can not duplicate an existing Topic name. tip Cannot start with __, or automatic Topic creation will fail. If the Topic name contains invalid characters, it may cause issues such as failure to collect Topic monitoring data.
Description	None
Number of partitions	The Default number of partitions set when enabling the automatic topic creation function.
Number of replicas	Default replica count, set when enabling the auto-create Topic function.
Minimum in-sync replicas	The default is 1.
message retention period	By default, it is the instance's message retention period.
Maximum message size	By default, it is the instance's maximum message size.
User permissions	By default, this topic has no custom permissions. If ACL is enabled, only SASL users with All Permitted permission have read and write access to this topic.
Tags	By default, this topic has no custom tags. To set tags for the Topic, refer to Manage topic tags.
message cleanup policy	For automatically created topic, the message cleanup policy is set to DELETE by default.

tip

We recommend using the auto-create topic function only in testing or data migration scenarios. In actual production environments, especially for core production businesses with high performance and stability requirements, it is recommended to disable this function.

Compared to manually creating a topic, automatically creating a topic takes more time. Besides, it only appears in the topic list two minutes after successful creation.

During the instance configuration process, creating topics by any means is prohibited. Before creating a topic through the client, confirm the instance status is Running.

1 Enable automatic topic creation

Select the region where the instance is located in the top menu bar.

On the Instances page, click the specified instance name.

In the Topic management tab, click Set up automatic topic creation.

Enable the Automatically create Topics and set the following parameters before you click Create.

Parameters	Description
ACL user automatic authorization	Whether to allow all ACL users to create Topics automatically. By default, it is enabled. This means all ACL users can create Topics automatically. Besides, permissions for all users to create and delete Topic under the ACL management tab will be created. tip The on and off of this switch is also controlled by the Topic creation permissions of all users. If creation permissions of all users are removed, this switch will turn off. If creation permissions of all users are enabled, this switch will turn on. Turn off the switch to prevent ACL users from automatically creating Topics. You can then configure Topic management permissions for target users through the ACL permission policy. For more information, see Create ACL. tip If you turn off this switch, the permissions for all users to create and delete Topics will not be removed. To delete creation and deletion permissions of all users, please proceed in the ACL management tab. For more information, refer to Delete ACL.
Default partition count	Default partition count for automatically created Topics. The value range is 1 to 300 and can not exceed the remaining available partitions of the instance.
Default number of replicas	Default partition replica count for automatically created Topics. Supports setting to 1 replica, 2 replicas, or 3 replicas.

In the prompt window, click OK.

tip

For instance created on or before October 26, 2023, the backend service will automatically restart the instance to apply the configuration when setting the default number of partitions or replicas.

2 Automatically create Topic through Kafka client SDK

After enabling the automatic Topic creation function, you can set a non-existent Topic name in the producer-side client SDK and request to write message data to it. Message Queue for Kafka will automatically create this Topic in the corresponding instance upon receiving the request.

You can refer to the following Java example code to set the message writing Topic for the producer instance in the Kafka client SDK. You can also refer to the example project provided by Message Queue for Kafka to develop related business logic.

    ......​
// Set a non-existent topic name. The backend service will automatically create this topic when receiving the request​
  private void setTopic(Properties kafkaProperties) {​
        topic  = kafkaProperties.getProperty("topic");​
    }​
    ......​

Related actions

After creating a Topic, you can modify its configuration and user permissions anytime as needed. You can also view detailed information about the Topic.

Modify Topic configuration: Supports modifying the number of partitions, replicas, minimum in-sync replicas, message retention period, and maximum message size. For details, see Modify Topic configuration.

Manage Topic tags: Supports filtering Topics by tags and batch binding or unbinding tags. For details, see Manage Topic tags.

Edit user permissions: set SASL user permissions for this Topic through ACL. For details, see Create ACL.

View Topic and partition details: check the status, number of partitions, and other information of the Topic. For details, see View Topic details.

Export Topic list: export the Topic list via the console, including Topic names, replica counts, and other settings. For details, see Export Topic list.

Delete Topic: delete a Topic when it is no longer used to save resources. For details, see Delete Topic.

Last updated: 2025.10.15 10:11:14