Docs

Database Transmission Service

MySQL connected over public network as the source

Synchronizing to Elasticsearch connected over express network

This topic describes how to create a task for synchronizing the MySQL connected over public network to ElasticSearch connected over express network on the Database Transmission Service (DTS) console.

Prerequisites

You have set up MySQL in a public network environment and created a database account.

Set the target information as needed, as detailed below:

When using VPN to perform data synchronization, the local network where the self-managed database is located has been connected to Volcano Engine via VPN gateway. For detailed steps, refer to Building a cloud VPC and on-premises multi-data-center network interconnection.

If data synchronization requires a dedicated line connection, you need to establish a dedicated line connection between a single cloud VPC and a single on-premises data center network. For more information, refer to Configuring direct connections.

The network connectivity and service availability of the source and destination have been confirmed.

Optional. Check each synchronization object in the source and target databases according to the instructions in the pre-check items. For more details, refer to Pre-check items (MySQL) and Pre-check items (Elasticsearch).

Precautions

When the source database is a self-managed RDS for MySQL, pay attention to the following information:

During synchronization, if the source database switches to the primary, the synchronization task will fail.

During synchronization, if the source database is restarted, the synchronization task will be temporarily interrupted and automatically attempt to recover. If the task does not recover for a long time, submit a ticket for technical support.

tip

If the synchronization object is the entire database, you can create a heartbeat table that is updated or written with data every second.

During data synchronization, if duplicate primary keys or unique keys are encountered in the target database, DTS will not retain the conflicting data at the destination. Instead, the data from the source will overwrite the data at the destination. When a conflict overwrite occurs, if there is a table with the same name in the target database, it may cause data inconsistency issues or task failures as detailed below. It is recommended to use table name mapping or delete or rename the table with the same name in the target database to avoid data inconsistency and reduce business risks.

When the table structure is consistent, data from full synchronization overwrites records in the target table with the same primary key or unique key values as in the source database.

When the table structure is inconsistent, initialization of data may fail, and only partial column data can be synchronized or synchronization may fail.

During the database and table structure initialization, DTS synchronizes foreign keys from the source database to the target database.

During full initialization and incremental synchronization, DTS temporarily disables constraint checks and foreign key cascade operations at the Session level. When the task runs, if the source database has cascade update or delete operations, data inconsistency may occur.

On the source end, if a database is not included in the synchronization objects, DDL operations can not be performed on databases that are included in the synchronization objects, or the synchronization task will fail.

Do not perform frequent DDL operations on the same table, as repeated DDL operations may cause DTS metadata update failures and risk transmission interruptions. Method 1: In the console, DDL operations are spaced at least 1 to 3 seconds apart.

Limits

Application limits

Type	Description
Source database limits	Database version: MySQL 5.5, 5.6, 5.7, and 8.0 instances are supported. tip For MySQL 5.5 instances, only schema and data synchronization are supported. User initialization is not available. When the source database instance version is 5.6 or higher, gtid_mode and enforce_gtid_consistency must both be set to ON. tip When the MySQL version is earlier than 8.0 and the parameter gtid_mode is disabled, foreign key cascading deletion may fail during data synchronization, resulting in data differences. Bandwidth requirements: Adequate egress bandwidth is required on the source database server. Failure to meet this prerequisite will lead to suboptimal synchronization performance. Supports the InnoDB database storage engine. MariaDB database engine is not supported currently. Tables to be synchronized must have a primary key or a unique non-null constraint, and their fields must be unique. Otherwise, data inconsistency may occur. Binlogs must meet the following requirements: The Binlog feature must be enabled, and the parameter binlog_format set to row, binlog_row_image set to full. Otherwise, pre-check will report an error, and data synchronization tasks can not start successfully. Binlogs must be retained for at least 24 hours, and a retention period of more than seven days is recommended. Otherwise, the DTS task may fail because it can not obtain the logs, potentially leading to data loss. It is recommended that the number of tables in a single task should not exceed 20,000, and the number of databases should not exceed 1,000. When a large number of databases and tables need to be synchronized, it is recommended to split them into multiple tasks. The source database does not support configuration of multi-node proxy addresses. During incremental data synchronization, DTS must establish a stable long connection with specific database instances. The load balancing or routing mechanism of multi-node proxy may cause connection drift or log parsing failure. It is recommended to configure a single node connection address directly or use a database gateway (DG) for access.
Target database limits	Current support for the following instance versions: Elasticsearch instances: versions 6.7.1, 6.8.23, and 7.10.2. Opensearch instances: versions 2.9.0 and 3.3.0.
Other limits	Prior to executing data synchronization, assess the performance of both source and target databases. It is also advised to perform synchronization during off-peak business hours. Otherwise, during full data initiation, DTS consumes certain read/write resources on both source and target databases, which may lead to an increase in database load. If the source database is RDS for MySQL 8.0, when configuring synchronization tasks, you must manually grant the relevant data synchronization permissions directly to the following accounts: The accounts for the source database are granted relevant permissions through the Role. The accounts that need to perform User synchronization are granted relevant permissions through the Role. Synchronization tasks only support synchronizing tables.

Operation limits

Currently, only synchronous DML statements are supported to be synced to Elasticsearch.

Supported synchronization architectures

One-to-one unidirectional synchronization

One-to-many unidirectional synchronization

Many-to-one unidirectional synchronization

For information on the types and usage of synchronization topology supported by DTS, refer to Data synchronization topology.

Supported SQL operations

Operation type	SQL statement
DML	INSERT, UPDATE, DELETE

Minimum database permission policy

Database	Schema initialization	Full initialization	Incremental synchronization
Source database	SELECT permission	SELECT permission	REPLICATION SLAVE, REPLICATION CLIENT, and SELECT permissions.
Elasticsearch normal user	Index: read, write, edit, and delete permissions Document: add, delete, modify, and query permissions

Procedure

Log on to the DTS console.

On the top menu bar, select the project and region.

In the left navigation bar, click Data synchronization.

On the Data synchronization list page, click Create task.

On the Create synchronization task page, locate the Configure source and target database wizard, and configure the following parameters.

Type	Name	Description
Basic information	Task name	Optional. Customize the task name. The length must be within 256 characters. tip If not set, the system generates a task name by default according to the naming rule, starting with dts-.
		Link specification	Select the link specifications for the synchronization task as needed. Compact, Medium, and Standard are supported. tip The link specification can not be modified after the task is successfully created. For details on the link specifications, refer to Product specifications.
		Synchronization topology	Select a synchronization topology based on your business needs. Only Unidirectional synchronization is currently supported. tip For more information about the synchronization topology, refer to Data synchronization topology.
Source database configuration	Select an existing data source	You can choose whether to use an existing instance as needed: Use existing instances: Click Select an existing data source, then in the Select data source control panel, choose the target region from the Region drop-down list, and select the target data source. Supports selecting data sources by data source name and ID. For methods to create data sources, refer to Data source management. Do not use existing instances: If not using an existing instance, you need to configure the following database instance information.
		Instance type	Select MySQL.
		Access method	SelectMySQL connected over public network. tip When synchronizing a MySQL connected over public network instance, add access permissions for the DTS IP range in the source's allowlist or security group policy in advance. Click View DTS CIDR blocks to view and copy the CIDR block.
		Region	Select the region where the target instance is located.
		Host address	Enter the host address of the target database's self-managed instance.
		Port	Enter the port number of the target self-managed instance.
		Database account	Enter the database account of the target instance.
		Database password	Enter the password corresponding to the database account of the target instance.
		Connection method	Select an Unencrypted connection or SSL-secured connection as needed to establish a more secure and reliable synchronization link. tip When selecting SSL secure connection, you need to enable the SSL encryption feature of the MySQL connected over public network instance in advance. If the synchronization task chooses Unencrypted connection and runs normally, enabling SSL encryption on the MySQL connected over public network instance may cause the unencrypted task to fail.
		SSL encryption certificate	Click Upload SSL encryption certificate to upload the CA certificate. When the Connection method is set to SSL-secured connection, this configuration is supported.
		Test connection	Click Test connection to verify the connection to the instance. If the test connection fails, modify the configuration based on the failure prompt, and then click Test connection again to verify the connection. The details are as follows: Connection authentication failed. Check the network: When this error prompt appears, confirm network connectivity. Connection authentication failed. Check the username and password: When this error prompt appears, verify your username and password. Invalid IP address: When this error prompt appears, verify the accuracy of the IP address.
Target database configuration	Select an existing data source	You can choose whether to use an existing instance as needed: Use existing instances: Click Select an existing data source, then in the Select data source control panel, choose the target region from the Region drop-down list, and select the target data source. Supports selecting data sources by data source name and ID. For methods to create data sources, refer to Data source management. Do not use existing instances: If not using an existing instance, you need to configure the following database instance information.
		Instance type	Select Elasticsearch.
		Access method	Select Elasticsearch connected over express network.
		Region	Select the region where the target instance is located.
		Address	Enter the host address of Elasticsearch connected over express network. tip Currently, only IPv4 addresses are supported for access. IPv6 addresses are not yet supported.
		VPC	Select the target VPC from the drop-down list as follows: Direct connect: Select any VPC under the CEN when achieving data connectivity through a dedicated line CEN. VPN connection: Select the VPC bound to the VPN gateway when achieving data connectivity through VPN.
		Subnet	Select a subnet within the VPC from the drop-down list. tip When synchronizing a dedicated network Elasticsearch instance, you must add access permissions for the Subnet IP range in the source end's allowlist or security group policy in advance. Click View subnet CIDR blocks to view and copy CIDR blocks.
		Database account	Enter the database account of the target instance.
		Database password	Enter the password corresponding to the database account of the target instance.
		Test connection	Click Test connection to verify the connection to the instance. If the test connection fails, modify the configuration based on the failure prompt, and then click Test connection again to verify the connection. The details are as follows: Connection authentication failed. Check the network: When this error prompt appears, confirm network connectivity. Connection authentication failed. Check the username and password: When this error prompt appears, verify your username and password. Connection authentication failed, unable to create connection normally, contact customer service: When this error prompt appears, submit a ticket to contact technical staff. Invalid IP address: When this error prompt appears, verify the accuracy of the IP address. The Express network requires the subnet to have * available IPs. The current number of available IPs in the subnet is insufficient, and the task can not be created. Check the subnet resources in the VPC: When this error prompt appears, adjust the number of available IPs in the VPC subnet to ensure enough IPs in the target VPC subnet.
Project and billing	Project	Optional. Select the affiliated project for the task from the drop-down list. By default, it is under default (default project). You can also click Create new project to create a new project. For more information, refer to Creating a new project. tip To grant sub-user privileges to access resources in a specific project, refer to Modifying project privileges. After a task is created successfully, if you need to change its affiliated project, you can move the resource to another project. For more information, refer to Moving resources to a project. A project is a virtual concept that includes a set of resources, users, and roles. A project allows for a unified view and management of a set of resources, and controls the permissions of users and roles within the project for those resources.
		Tag	(Optional) Click Add tag to add a tag to the instance. Tags can help you categorize cloud resources, control access to them, and allocate bills and costs. A tag is a key-value pair that identifies a cloud resource. You can use tags to categorize resources with the same characteristics from different dimensions, such as purpose or owner, making it easier to filter, aggregate, and manage your cloud resources. For more details about tags, refer to Tag overview. The rules for tag key-value pairs are as follows: Key Supports any language, characters, numbers, spaces, or symbols (_.:/=+-@). Case-sensitive, and the length is 1 to 128 characters. In UTF-8 encoded Unicode format. Keys cannot start with any form of volc:, such as Volc: or VOLC:. (Optional) Value Supports any language, character, digit, space, or symbol (_.:/=+-@). Case-sensitive, and the length is 0 to 256 characters. In UTF-8 encoded Unicode format. tip Support fuzzy search for selecting existing tag keys and tag values. A maximum of 20 tags can be associated with a task in a single operation. A maximum of 50 tags can be associated with a single task. For a single resource, tag keys must be unique, and each tag key can have only one value.
		Billing method	Select a billing method: Pay-as-you-go or Subscription. tip For detailed billing information, refer to Billing instructions.
		Subscription duration	Select the duration of the subscription as needed. tip When selecting Subscription in Billing method, you can set the purchase time and also check Automatic renewal. The system automatically renews before task expiration.

Click Next: configure synchronization object.

On the Configure synchronization object wizard page, configure the related parameter information as shown in the table below.

tip

If you do not need to start the synchronization task immediately, click Save in the lower-right corner of the page after configuring the synchronization object.

After saving, you will be redirected to the Synchronization task list page. The task status will be To be started, and you can configure the synchronization task at a suitable time. For configuration methods, refer to Configuring a synchronization task.

Name			Description
Synchronization type			Select the synchronization type as needed. Currently, Schema initialization, Full initialization, and Incremental synchronization are supported. tip Incremental synchronization is selected by default. When you select Schema initialization or Full initialization, after the pre-check is completed, DTS will initialize the data of the objects to be synchronized from the source instance in the target instance, which serves as the basis for subsequent incremental data synchronization.
Index concatenation rule			Select index concatenation rules as needed, supporting Table name and Database name + Table name options: Table name: When you select Table name, the index name created in the target Elasticsearch instance is the same as the table name. Database name + table name: When you select Database name + table name, the index name created in the target Elasticsearch instance is the database name + table name. tip After a synchronization task is successfully created, modifying the index concatenation rule will reset the names of the synchronization objects. Proceed with caution.
Primary key conflict handling			Select the method to handle when the target database already has rows with the same primary key value. Currently, only Overwrite on conflict is supported, which means using source database data to overwrite target database data, allowing the synchronization task to start normally.
Synchronization starting point			Set to pull Binlog logs from a specified position or timestamp. The values are as follows: System default: The default is the task start time. Specify GTID: Enter the specified GTID in the input box, formatted as source_id:transaction_id. Set GTID_EXECUTED: Enter the specified GTID_EXECUTED in the input box, formatted as source_id:transaction_id. If multiple exist, separate them with commas (,). For how to obtain the instance's GTID_EXECUTED, refer to How to obtain the GTID_EXECUTED of the instance. Specify timestamp: Click the input box to select a date, click Select time to set the time point, then click OK. tip When Synchronization type selects Full initialization, this step is not supported. When Synchronization type selects both Full initiation and Incremental synchronization, the Starting timestamp defaults to the current time point and can not be modified. GTID and GTID_EXECUTED must be legal values from the source end. Otherwise, pre-check reports an error. When selecting Specify timestamp, the time you set must be later than the earliest Binlog time on the source end. If there is no corresponding time Binlog, the closest Binlog starting point will be selected.
Sync type selection			Select the operation types to synchronize. Currently, only DML statements are supported, which include DELETE, Insert, and Update. All DML operations are selected by default and can be adjusted as needed.
Whether to enable foreign key check?			Select whether to enable foreign key check as needed. When enabled, the tool supports synchronizing cascading updates and deletes of foreign keys from the source, ensuring data integrity and consistency throughout the synchronization.
Synchronization object			Select the tables to synchronize. Meanwhile, you can perform the following actions in the Selected object area: Rename databases or tables: Click the pencil icon next to the target database or table to rename them, supporting query of target objects via Normal search and Global search methods. If the entire database is selected in the Synchronization object area, only the database name can be modified in the Selected object area, and the database cannot be expanded. Click Batch mapping to modify database or table names in bulk. For more information, refer to Database and table batch mapping. Filter and delete unnecessary targets: The search box in the Selected object area supports fuzzy search to batch delete unwanted databases, tables, and other targets. Filter data in tables: Check Filter data enables filtering of data within the table. tip Regular expression search for target objects is supported in the Synchronization object or Selected object areas.
Filter condition			Optional. Filtering data during a full data scan improves synchronization performance. Click the pen icon in the Filter condition column for the target database, table, or view, set the filter condition, then click Verification statement to validate the condition. tip This feature is currently in beta testing. To use it, submit a ticket for technical support. This field can be configured only if Filter data is selected for the Synchronization object's database, table, or view. Filter condition currently supported =, !=, <, >, <=, >=, in, is null, and, or.
Database and table field configuration			Configure related information of the database and table fields. You can route documents to shards of the Elasticsearch instance as custom routing: Type name: You can customize the Type name. Click Batch set Type name to batch set Type names. Naming rules are as follows: It can not contain English commas (, ) or symbols (#). It can not start with an English period (.). It can not start with an underscore (_) except for (_doc). The length must be within 255 characters, and the name must not contain blank characters at the beginning or end. tip Currently, only ES instances of version 7.x and earlier support custom Type names. ES instances of version 8.x and later, as well as OpenSearch versions, do not support custom Type names. If the target index already exists, the Type name must be consistent with the custom name you set. Otherwise, schema migration fails. Whether _routing: Specify whether to enable custom routing mode. It also supports setting the _routing of multiple database names to yes or no. Yes: You can custom route documents to shards in Elasticsearch. No: Documents can only route to shards in Elasticsearch via _id. _routing column: Customize indexing a document into an Elasticsearch shard. The default is _id. _id value: Customize the document _id value. When selecting multiple fields to form _id, connect with underscores (_).
Advanced configuration	Enable ETL configuration	Choose whether to enable the ETL function. For more information about ETL, refer to Introduction to ETL. Yes: indicates enabling ETL functions. After enabling ETL rules, click Quick configuration field mapping/filtering rules, configure synchronization objects in the Quick configuration field mapping/filtering rules dialog box, and click OK to quickly generate ETL statements. After enabling the ETL function, you can configure data processing ETL rules by referring to the DSL syntax. Enter the data processing ETL rules, i.e., the DSL script, in the Enter ETL statements or use ETL assistant to generate statements...text box, and click Validate syntax to verify whether the data processing statements are correct. You can also use the ETL assistant to generate ETL statements in natural language. In the upper-right corner of the ETL statement text box, click ETL assistant to input ETL requirements, and press Enter. After generating the ETL, copy the generated ETL statements into the ETL text box. The ETL assistant offers a variety of ETL statement generation capabilities. For more information, refer to ETL assistant capabilities. Click Debug to test the effects of the ETL rules you created. For details on debugging, refer to Verifying ETL rules via ETL Playground tool. No: indicates not enabling ETL functions.
		Error retry time	If a synchronization task fails due to a disconnection between the source and target databases, the system immediately retries connecting to them multiple times within the specified time. Value range: 600–86400, in seconds (s). The default value is 600 seconds.

Last updated: 2026.07.31 11:38:48