Docs

Database Transmission Service

Creating a data synchronization task for Milvus

Public network self-hosted Milvus synchronization to Volcano Engine Milvus

This topic describes how to create a task for synchronizing Milvus connected over public network to the Volcano Engine Milvus on the Database Transmission Service (DTS) console.

Prerequisites

A vector database instance of Milvus has been created. For detailed steps, refer to Creating an instance.

The network connectivity and service availability of the source and destination have been confirmed.

When the access method of the database instance is connected over public network and access restrictions are enabled, add the DTS server IP addresses to the allowlist or security group of the database instance before creating the database transmission task. DTS server IP addresses vary by region, as detailed below:

DTS server IP addresses in China (Guangzhou), China (Shanghai), and China (Beijing) are 221.194.189.0/27,157.148.90.32/27,180.184.132.64/27,61.129.54.64/27,220.196.172.32/27,117.135.143.32/27,220.196.168.32/27,116.132.250.32/27,111.45.13.32/27,125.88.248.32/27,180.153.35.32/27,220.181.65.32/27,220.181.65.64/27.

DTS server IP addresses in Asia Pacific (Johor) are 101.47.0.32/27,220.181.65.32/27,220.181.65.64/27,101.47.65.192/27,101.47.65.32/27.

For more information on the CIDR block of DTS servers, refer to Adding the CIDR block of the DTS server.

Precautions

Currently, this feature is available only in China (Guangzhou), China (Shanghai), China (Beijing), and Asia Pacific (Johor) regions.

Milvus database supports Insert and Upsert write modes. For more details, refer to Milvus primary fields and automatic recognition. When synchronizing data, the differences and applicable scenarios of Insert and Upsert write modes are as follows.

Insert: Insert data directly using the Insert command. This applies when Collection has AutoID enabled, and data in the source and target ends (including primary key) are consistent. When DTS restarts or fetches data repeatedly, it may cause data redundancy in the destination.

Upsert: Insert data directly or update existing data using the Upsert command. This is suitable for updating existing data when Collection does not have AutoID enabled, or when Collection has AutoID enabled but business or verification does not rely on the primary key. When DTS restarts or fetches data repeatedly, it may lead to primary key inconsistency, affecting business or verification that depends on the primary key.

During synchronization, do not change the instance configuration. Otherwise, the synchronization task will fail.

Synchronizing instances with SSL enabled is not supported.

The current version supports synchronizing Alibaba Cloud's Zilliz Cloud 2.6 to Volcano Engine Milvus 2.6. to use it, you can submit tickets to contact technical support.

Limits

Type	Description
Source database limits	Database version: currently supports Milvus engine versions 2.5 and 2.6. Bandwidth requirements: Adequate egress bandwidth is required on the source database server. Failure to meet this prerequisite will lead to suboptimal synchronization performance.
Target database limits	Milvus engine versions are 2.5 and 2.6.
Other limits	Role/User permission synchronization is not currently supported. It is recommended that the Milvus versions of the source and target databases remain consistent or synchronize from lower to higher versions to ensure compatibility. If you synchronize from a higher version to a lower version, there may be database compatibility or data loss issues. Prior to executing data synchronization, assess the performance of both source and target databases. It is also advised to perform synchronization during off-peak business hours. Otherwise, during full data synchronization, DTS consumes certain read/write resources on both source and target databases, which may lead to an increase in database load. During the synchronization process, the target database may generate a large volume of logs or data files, significantly increasing disk usage on the target instance. We recommend that you review and configure the log retention policy or data cleanup mechanism of the target database before synchronization to prevent excessive storage usage.

Supported synchronization architectures

One-to-one unidirectional synchronization

One-to-many unidirectional synchronization

Many-to-one unidirectional synchronization

Cascading unidirectional synchronization

For information on the types and usage of synchronization topology supported by DTS, refer to Data synchronization topology.

Database account minimum permission policy

Database	Privilege
Source database	Root permission or granting ClusterAdmin, DatabaseAdmin, and CollectionAdmin permissions to a User.
Target database	Root permission or granting ClusterAdmin, DatabaseAdmin, and CollectionAdmin permissions to a User.

Procedure

Log on to the DTS console.

On the top menu bar, select the project and region.

In the left navigation bar, click Data synchronization.

On the Synchronization task list page, click Create task.

On the Create synchronization task page, locate the Configure source and target database wizard, and configure the following parameters.

Type	Parameter	Description
Basic information	Task name	Optional. Customize the task name. The length must be within 256 characters. tip If not set, the system generates a task name by default according to the naming rule, starting with dts-.
		Link specification	Select the link specifications for the synchronization task as needed. Compact, Medium, and Standard are supported. tip The link specification can not be modified after the task is successfully created. For details on the link specifications, refer to Product specifications.
		Synchronization topology	Select a synchronous topology based on your business needs. Only Unidirectional synchronization is currently supported for the Milvus database. tip For more information about the synchronization topology, refer to Data synchronization topology.
Source database configuration	Select an existing data source	Currently, selecting Milvus data source is not supported. You need to configure the following database instance information.
		Instance type	Select Milvus.
		Access method	Select EndpointPublicMilvus, which is the public network Milvus. When synchronizing a public network Milvus instance, you need to add DTS IP segment access permissions in the source end's allowlist or security group policy in advance. Click View DTS CIDR blocks to view and copy the CIDR block.
		Region	Select the region where the target instance is located.
		Address	Enter the host address of the target self-managed instance. tip When synchronizing data from Alibaba Cloud Zilliz Cloud, the input Address format must be the TLS endpoint in the form of https://host:19530
		Database account	The default account is root. tip When synchronizing data from Alibaba Cloud Zilliz Cloud, ensure the source end Account number can stably read metadata such as Collection list, Schema, and Index, and it is recommended to perform sampling verification before executing officially.
		Database password	Enter the password corresponding to the database account.
		Test connection	Click Test connection to verify the connection to the instance. If the test connection fails, modify the configuration based on the failure prompt, and then click Test connection again to verify the connection. The specific prompts are as follows: Connection authentication failed. Check if the username and password are correct: When this error prompt appears, verify your username and password. Connection authentication failed, unable to create connection normally, contact customer service: When this error prompt appears, submit a ticket for technical support.
Target database configuration	Select an existing data source	Currently, selecting Milvus data source is not supported. You need to configure the following database instance information.
		Instance type	Select Milvus.
		Access method	Select EndpointVolcMilvus, which refers to Volcano Engine Milvus.
		Region	Select the region where the target instance is located.
		Milvus instance	Select the target instance of the vector database Milvus from the drop-down list. tip You can also search for the target instance by instance ID or name.
		Database account	The default account is root.
		Database password	Enter the password corresponding to the database account.
		Test connection	Click Test connection to verify the connection to the instance. If the test connection fails, modify the configuration based on the failure prompt, and then click Test connection again to verify the connection. The specific prompts are as follows: Connection authentication failed. Check if the username and password are correct: When this error prompt appears, verify your username and password. Connection authentication failed, unable to create connection normally, contact customer service: When this error prompt appears, submit a ticket for technical support.
Project and billing	Project	Optional. Select the affiliated project for the task from the drop-down list. By default, it is under default (default project). You can also click Create new project to create a new project. tip After a task is created successfully, if you need to change its affiliated project, you can move the resource to another project. A project is a virtual concept that includes a set of resources, users, and roles. A project allows for a unified view and management of a set of resources, and controls the permissions of users and roles within the project for those resources.
		Tag	(Optional) Click Add tag to add a tag to the instance. Tags can help you categorize cloud resources, control access to them, and allocate bills and costs. A tag is a key-value pair that identifies a cloud resource. You can use tags to categorize resources with the same characteristics from different dimensions, such as purpose or owner, making it easier to filter, aggregate, and manage your cloud resources. For more details about tags, refer to Tag overview. The rules for tag key-value pairs are as follows: Key Supports any language, characters, numbers, spaces, or symbols (_.:/=+-@). Case-sensitive, and the length is 1 to 128 characters. In UTF-8 encoded Unicode format. Keys cannot start with any form of volc:, such as Volc: or VOLC:. (Optional) Value Supports any language, character, digit, space, or symbol (_.:/=+-@). Case-sensitive, and the length is 0 to 256 characters. In UTF-8 encoded Unicode format. tip Support fuzzy search for selecting existing tag keys and tag values. A maximum of 20 tags can be associated with a task in a single operation. A maximum of 50 tags can be associated with a single task. For a single resource, tag keys must be unique, and each tag key can have only one value.
		Billing method	Select a billing method: Pay-as-you-go or Subscription. tip For detailed billing information, refer to Billing instructions.
		Subscription duration	Select the duration of the subscription as needed. tip When selecting Subscription in Billing method, you can set the purchase time and also check Automatic renewal. The system automatically renews before task expiration.

Click Next: configure synchronization object.

On the Configure synchronization object wizard page, configure the related parameter information as shown in the table below.

tip

The synchronization task defaults to Full initialization.

If you do not need to start the synchronization task immediately, click Save in the lower-right corner of the page after configuring the synchronization object.

After saving, you will be redirected to the Synchronization task list page. The task status will be To be started, and you can configure the synchronization task at a suitable time. For configuration methods, refer to Configuring a synchronization task.

Name			Description
Synchronization type			Select the synchronization type as needed. Currently, only full initialization is supported.
Write mode			Select the data write mode as needed. The differences and applicable scenarios of the two write modes are as follows. Insert: When using Insert to write data, Milvus does not check the uniqueness of the primary key on the destination and writes data directly. The Insert write mode applies to scenarios where the collection has AutoID enabled and data (including primary key) on the source and target ends are consistent. When DTS restarts or fetches data repeatedly, it may cause data redundancy in the destination. Upsert: When using Upsert to write data, Milvus checks whether the primary key exists at the destination. If it does not exist, data is written directly. If it exists, the original data is deleted and the new data is written. When the Collection has AutoID enabled, Milvus ignores the primary key information of the original data and generates a new primary key. Upsert write mode applies to updating existing data when the Collection does not have AutoID enabled, or when Collection has AutoID enabled but business or verification does not rely on the primary key. When DTS restarts or fetches data repeatedly, it may lead to primary key inconsistency, affecting business or verification that depends on the primary key. tip Whether Milvus enables AutoID and the choice of write mode have a significant impact on business. For more information, refer to Milvus primary field and automatic recognition.
Synchronization object			In the Synchronization objects area, select the objects to be synchronized, and in the Selected object area, confirm the objects to be synchronized. tip Regular expression search for target objects is supported in the Synchronization object or Selected object areas.
Advanced configuration	Error retry time	If a synchronization task fails due to a disconnection between the source and target databases, the system immediately retries connecting to them multiple times within the specified time. Value range: 600–86400, in seconds (s). The default value is 600 seconds.

Click Pre-check.

Optional. If a pre-check has been performed for the created task, you can confirm whether to restart the pre-check based on the prompts. If you need to restart, click OK in the Whether to restart the pre-check dialog box.

If the instances of the source and target databases are not modified, there is no need to restart the pre-check.

If you modify the source and target database instance information or the synchronization task parameters, it is recommended to restart the pre-check to avoid task startup failures or interruptions.

On the Pre-check configuration wizard page, perform the following actions based on the check result:

Click Details in the Actions column of the pre-check results to view detailed alert and failure messages. After modifying the databases and tables accordingly, click Retry.

Click Ignore in the Actions column to ignore the alert and continue starting the synchronization task. You can also click Ignore all to ignore all alert messages with one click.

tip

If you choose to Ignore the alert and start the synchronization task, it may cause data inconsistencies and business risks.

For details about the pre-check, refer to Pre-check items.

At the bottom-right of the Pre-check page, click Confirm order.

Select the configuration method according to the billing method of the task creation:

Pay-as-you-go

On the Confirm order configuration wizard page, confirm the configuration information again. Read the Service-Specific Terms for BytePlus Database Products and Services and check I have read and agree to Service-Specific Terms for BytePlus Database Products and Services, and click Create and start a task.

Subscription

On the Confirm order configuration wizard page, confirm the configuration information again. Read the Service-Specific Terms for BytePlus Database Products and Services and check I have read and agree to Service-Specific Terms for BytePlus Database Products and Services, and click Create and start a task.

On the Confirm billing information configuration wizard page, verify the instance’s billing information.

Optional. Click Voucher redemption. In the Voucher redemption dialog box, enter the voucher code and click Redeem.

Optional. In the voucher area, select an available voucher as needed. Only one voucher can be selected at a time to offset part of the cost. You can also choose not to use a voucher.

Click Next step.

tip

Once the Payment is completed on the configuration wizard page, a Successful payment message means your task has been created.

After successfully creating the data synchronization task, you can return to the DTS console to view the data synchronization task information on the data synchronization task list.

Related APIs

API	Description
CreateTransmissionTask	Call the CreateTransmissionTask API to create a DTS task.
ModifyTransmissionTask	Call the ModifyTransmissionTask API to modify the transmission task.

Last updated: 2026.07.31 11:38:49