Docs

Database Transmission Service

Volcano Engine RDS for PostgreSQL as the source

Synchronizing to PostgreSQL connected over express network

This topic describes how to create a task for synchronizing the Volcano Engine PostgreSQL to PostgreSQL connected over express network on the Database Transmission Service (DTS) console.

Prerequisites

An instance, a database, and an account for PostgreSQL have been created. For detailed steps, refer to Creating an instance and Creating a database and an account.

Set the target information as needed, as detailed below:

When using VPN to perform data synchronization, the local network where the self-managed database is located has been connected to Volcano Engine via VPN gateway. For detailed steps, refer to Building a cloud VPC and on-premises multi-data-center network interconnection.

If data synchronization requires a dedicated line connection, you need to establish a dedicated line connection between a single cloud VPC and a single on-premises data center network. For more information, refer to Configuring direct connections.

The network connectivity and service availability of the source and destination have been confirmed.

Precautions

To ensure data consistency in real time, it is recommended to select structure initialization, full initialization, and incremental synchronization.

If the created task is a bidirectional synchronization task, it is not recommended to use sequence as the auto-increment ID, otherwise data inconsistencies may result.

The synchronization object granularity can be either database or table. If tables are selected as synchronization objects, other objects such as views, triggers, functions, or stored procedures will not be synchronized to the target database. If you need to synchronize other objects, create a new synchronization task.

Only data from the primary database supports synchronization. Data from the secondary database is not supported.

Since full initialization performs INSERT operations concurrently, the collection of the target database generates fragments. Therefore, the collection storage space of the target database is larger than that of the source database after full initialization is complete.

During the pre-check, if you ignore the alert and start the synchronization task, it may lead to data inconsistencies and pose business risks. For details about the pre-check, refer to Pre-check items (PostgreSQL).

During synchronization, if there are CREATE TABLE or RENAME TABLE operations on the source end, select the entire database containing the table or the entire database where the table is renamed to when configuring the synchronization object. Otherwise, the newly created or renamed table will not synchronize to the destination.

For RDS for PostgreSQL synchronization tasks, the source database account must create triggers, functions, and other objects in the source database. Therefore, during the synchronization process, do not modify the access permissions of objects such as triggers, functions, or internal tables created by this account in the source database.

During data synchronization, if duplicate primary keys or unique keys are encountered in the target database, DTS will not retain the conflicting data at the destination. Instead, the data from the source will overwrite the data at the destination. When a conflict overwrite occurs, if there is a table with the same name in the target database, it may cause data inconsistency issues or task failures as detailed below.

When the table structure is consistent, data from full synchronization overwrites records in the target table with the same primary key or unique key values as in the source database.

When the table structure is inconsistent, initialization of data may fail, and only partial column data can be synchronized or synchronization may fail.

When creating a data synchronization task via express network, the system creates several network cards by default in your selected VPC, and the network cards are mounted by default to the subnet you selected. The network card name format is dts_shuttle_********.

You need to ensure the availability of the logical replication slot of the instance after switching the primary node. If it is unavailable, the task will be interrupted and cannot be recovered after switching the primary node, and a new task needs to be created.

If you use the major version upgrade function of the Volcano Engine RDS for PostgreSQL database, the logical replication slot in existing RDS for PostgreSQL synchronization tasks may be lost after the upgrade, resulting in the failure of RDS for PostgreSQL synchronization tasks. If this occurs, you need to recreate the PostgreSQL synchronization task.

Limits

Application limits

Type	Description
Source	Bandwidth requirements: Adequate egress bandwidth is required on the source database server. Failure to meet this prerequisite will lead to suboptimal synchronization performance. Tables to be synchronized must have a primary key or a unique non-null constraint, and their fields must be unique. Otherwise, data inconsistency may occur. It is recommended that the number of tables in a single task should not exceed 20,000, and the number of databases should not exceed 50. When a large number of databases and tables need to be synchronized, it is recommended to split them into multiple tasks. The database or table name of the object to be synchronized can not contain question marks (?). Otherwise, the task will fail. For incremental synchronization, the requirements for WAL logs are as follows: You must enable WAL logs, that is, set the value of the wal_level parameter to logical. WAL logs must be retained for at least 24 hours, and a retention period of more than seven days is recommended. Otherwise, the DTS task may fail because it can not obtain the logs, potentially leading to data loss.
others	A single synchronization task can only synchronize one database. To synchronize multiple databases, configure a synchronization task for each one. During the incremental data synchronization process, if the granularity of the synchronization object is schema, ensure that the table-level replication identity REPLICA IDENTITY is set to FULL. If a new table is created in the schema to be synchronized, or the table to be synchronized is reconstructed using the RENAME command, you need to execute the ALTER TABLE schema.table REPLICA IDENTITY FULL; command before writing data to the table. tip When executing the ALTER TABLE schema.table REPLICA IDENTITY FULL; command, replace schema and table with the actual schema and table names. During incremental data synchronization, DTS creates a replication slot with the prefix dts_sync_ in the source database to replicate data. DTS automatically cleans up historical replication slots every 5 to 10 minutes to prevent them from continuously accumulating and taking up disk space. tip When a synchronization task is released or fails, DTS actively cleans up the replication slot. If a primary/secondary switchover occurs in RDS for PostgreSQL, you need to log in to the secondary database to manually clean up the replication slot. Prior to executing data synchronization, assess the performance of both source and target databases. It is also advised to perform synchronization during off-peak business hours. Otherwise, during full data synchronization, DTS consumes certain read/write resources on both source and target databases, which may lead to an increase in database load. Since full data synchronization performs concurrent insert operations, the table in the target database generates fragments. Therefore, the storage space of the table in the target database will be larger than that of the table in the source database after the full synchronization is complete. Verify that the character sets of the source and target databases are consistent before synchronization. Otherwise, pre-check may fail.

Operation limits

Do not perform DDL operations on library or table structure changes during the database & table structure initialization and full initialization phases, or the data synchronization task will fail.

If only full initialization is performed, do not write new data to the source database, or the source and target data will be inconsistent. To ensure data consistency in real time, it is recommended to select structure initialization, full initialization, and incremental synchronization.

Minimum database permission policy

The account used for data synchronization must have the following permissions:

The source database account needs to have superuser permissions to create logical replication slots.

The target database account needs to have permissions to create a schema. If the target database schema already exists, the synchronization account must be the target schema owner.

Supported SQL statements

Operation type	SQL operation statements
DML	INSERT, UPDATE, DELETE
DDL	CREATE TABLE, ALTER TABLE, DROP TABLE, TRUNCATE TABLE CREAT INDEX, DROP INDEX CREATE SEQUENCE, ALTER SEQUENCE, DROP SEQUENCE DROP RULE CREATE TYPE tip TYPE supports SHELL, ENUM, RANGE, or COMPOSE.

Procedure

Log on to the DTS console.

On the top menu bar, select the project and region.

In the left navigation bar, click Data synchronization.

On the Synchronization task list page, click Create task.

On the Create synchronization task page, locate the Configure source and target database wizard, and configure the following parameters.

Type	Parameter	Description
None	Task name	Optional. Customize the task name. Naming rules are as follows: Must start with a Chinese character or letter. Must be 64 characters or less. It can contain only Chinese characters, letters, digits, underscores (_), and hyphens (-).
		Link specification	Select the link specifications for the synchronization task as needed. Compact, Medium, and Standard are supported. tip The link specification can not be modified after the task is successfully created. For details on the link specifications, refer to Product specifications.
		Synchronization topology	Select a synchronous topology based on your business needs. Unidirectional synchronization and Bidirectional synchronization are currently supported. tip For more information about the synchronization topology, refer to Data synchronization topology. When you select bidirectional synchronization for the synchronous topology, it is not recommended to use a sequence as the auto-increment ID.
Source database configuration	Select an existing data source	You can choose whether to use an existing instance as needed: Use existing instances: Click Select an existing data source, then in the Select data source control panel, choose the target region from the Region drop-down list, and select the target data source. Supports selecting data sources by data source name and ID. For methods to create data sources, refer to Data source management. Do not use existing instances: If not using an existing instance, you need to configure the following database instance information.
		Instance type	Select PostgreSQL.
		Access method	Select Volcano Engine RDS for PostgreSQL.
		Region	Select the region where the target instance is located.
		Cross-Volcano Engine account?	Select whether to transmit data cross-account as needed, with the following options: This account: Selecting this means data transmission occurs only within this account. Cross-account: Selecting this means data transmission occurs between two accounts. tip When transmitting data across accounts, log on to the Database Transmission Service (DTS) console through the account of the destination. Before cross-account data transmission, create and authorize an IAM role in the source root user. For detailed steps, refer to How to authorize users for cross-account data transmission.
		Cross-account ID	Enter the Volcano Engine account ID to which the source database belongs.
		Cross-account roles	Enter the IAM role authorized for DTS to access cross-account resources.
		PostgreSQL instance	Select the target instance of RDS for PostgreSQL from the drop-down list. tip You can also search for the target instance by instance ID or name.
		Database name	Enter the database name.
		Database account	Enter the database account of the target instance.
		Database password	Enter the password corresponding to the database account of the target instance.
		Test connection	Click Test connection to verify the connection to the instance. If the test connection fails, modify the configuration based on the failure prompt, and then click Test connection again to verify the connection. The specific prompts are as follows: Connection authentication failed. Check if the username and password are correct: When this error prompt appears, verify your username and password. Connection authentication failed, unable to create connection normally, contact customer service: When this error prompt appears, submit a ticket for technical support.
Target database configuration	Select an existing data source	You can choose whether to use an existing instance as needed: Use existing instances: Click Select an existing data source, then in the Select data source control panel, choose the target region from the Region drop-down list, and select the target data source. Supports selecting data sources by data source name and ID. For methods to create data sources, refer to Data source management. Do not use existing instances: If not using an existing instance, you need to configure the following database instance information.
		Instance type	Select PostgreSQL.
		Access method	Select PostgreSQL connected over express network.
		Region	Select the region of the virtual private cloud.
		Host address	Enter the host address of the target self-managed instance. tip Currently, only IPv4 addresses are supported for access. IPv6 addresses are not yet supported.
		Port	Enter the port number of the target self-managed instance.
		Database name	Enter the database name of the target self-managed instance.
		Virtual private cloud	Select the target VPC from the drop-down list as follows: Direct connect: Select any VPC under the CEN when achieving data connectivity through a dedicated line CEN. VPN connection: Select the VPC bound to the VPN gateway when achieving data connectivity through VPN.
		Subnet	Select a subnet within the VPC from the drop-down list. tip When synchronizing a PostgreSQL connected over express network, you need to add subnet CIDR block access permissions to the allowlist or security group policy on the source end in advance. Click View subnet CIDR blocks to view and copy CIDR blocks.
		Database account	Enter the database account of the target instance.
		Database password	Enter the password corresponding to the database account of the target instance.
		Test connection	Click Test connection to verify the connection to the instance. If the test connection fails, modify the configuration based on the failure prompt, and then click Test connection again to verify the connection. The specific prompts are as follows: Connection authentication failed. Check the network: When this error prompt appears, confirm network connectivity. Connection authentication failed. Check the username and password: When this error prompt appears, verify your username and password. Connection authentication failed, unable to create connection normally, contact customer service: When this error prompt appears, submit a ticket to contact technical staff. Invalid IP address: When this error prompt appears, verify the accuracy of the IP address. The Express network requires the subnet to have * available IPs. The current number of available IPs in the subnet is insufficient, and the task can not be created. Check the subnet resources in the VPC: When this error prompt appears, adjust the number of available IPs in the VPC subnet to ensure enough IPs in the target VPC subnet.
None	Project	Optional. Select the affiliated project for the task from the drop-down list. By default, it is under default (default project). You can also click Create new project to create a new project. For more information, refer to Creating a new project. tip To grant sub-user privileges to access resources in a specific project, refer to Modifying project privileges. After a task is created successfully, if you need to change its affiliated project, you can move the resource to another project. For more information, refer to Moving resources to a project. A project is a virtual concept that includes a set of resources, users, and roles. A project allows for a unified view and management of a set of resources, and controls the permissions of users and roles within the project for those resources.
		Tag	Optional. Click Add tag to add a tag to the instance. Tags can help you categorize cloud resources, control access to them, and allocate bills and costs. A tag is a key-value pair that identifies a cloud resource. You can use tags to categorize resources with the same characteristics from different dimensions, such as purpose or owner, making it easier to filter, aggregate, and manage your cloud resources. For more details about tags, refer to Tag overview. The rules for tag key-value pairs are as follows: Key Supports any language, characters, numbers, spaces, or symbols (_.:/=+-@). Case-sensitive, and the length is 1 to 128 characters. In UTF-8 encoded Unicode format. Keys cannot start with any form of volc:, such as Volc: or VOLC:. (Optional) Value Supports any language, character, digit, space, or symbol (_.:/=+-@). Case-sensitive, and the length is 0 to 256 characters. In UTF-8 encoded Unicode format. tip Supports fuzzy search for selecting existing tag keys and tag values. A maximum of 20 tags can be associated with a task in a single operation. A maximum of 50 tags can be associated with a single task. For a single resource, tag keys must be unique, and each tag key can have only one value.
		Billing method	Select a billing method: Pay-as-you-go or Subscription. tip For detailed billing information, refer to Billing instructions.
		Subscription duration	Select the duration of the subscription as needed. tip When selecting Subscription in Billing method, you can set the purchase time and also check Automatic renewal. The system automatically renews before task expiration.

Click Next: configure synchronization object.

On the Configure synchronization object wizard page, configure the related parameter information as shown in the table below.

tip

If you do not need to start the synchronization task immediately, click Save in the lower-right corner of the page after configuring the synchronization object.

After saving, you will be redirected to the Synchronization task list page. The task status will be To be started, and you can configure the synchronization task at a suitable time. For configuration methods, refer to Configuring a synchronization task.

Parameter	Description
Synchronization type	Select the synchronization type as needed. Currently, Schema initialization, Full initialization, and Incremental synchronization are supported. including: If only full initialization is required, you can select both Schema initialization and Full initialization. If you need to perform no-downtime synchronization, you can select Structure initialization, Full initialization, and Incremental synchronization at the same time. tip Do not write new data to the source instance during data synchronization when Incremental synchronization is not selected to ensure data consistency. To ensure data consistency in real time, it is recommended to select structure initialization, full data initialization, and incremental data synchronization.
Synchronization object	In the Synchronize object area, select the objects you want to synchronize, and in the Selected object area, confirm the objects you want to synchronize. tip If a table is selected as the synchronization object, other dependent objects, such as views, triggers, functions, or stored procedures, must be manually selected to be synchronized to the target database. Otherwise, a warning prompt will appear during the pre-check. To ensure data synchronization performance and task stability, TRIGGER, RULE, FOREIGN KEY, SEQUENCE, and other objects of RDS for PostgreSQL will synchronize only after the incremental synchronization completes. Filter and delete unnecessary targets: The search box in the Selected object area supports fuzzy search to batch delete unwanted databases, tables, and other targets.
Skip post-schema synchronization	When selecting schema initialization in synchronization type, DTS automatically synchronizes post-schema after the incremental synchronization ends. You may also skip post-schema synchronization, meaning DTS will not synchronize post-schema objects. PostgreSQL database post-schema synchronization objects include ordinary trigger, event trigger, RULE, exclusive constraint, foreign key constraint, check constraint, Domain constraint, and Sequence value setting. Yes: Represent skipping post-schema synchronization, i.e., DTS does not synchronize post-schema objects. No: Represent not skipping post-schema synchronization, i.e., DTS synchronizes post-schema objects. tip Whether to skip post-schema synchronization is supported only when the synchronization topology is Unidirectional synchronization and the migration type includes Schema initialization.
Error retry time	If a synchronization task fails due to a disconnection between the source and target databases, the system immediately retries connecting to them multiple times within the specified time. Value range: 600–86400, in seconds (s). The default value is 600 seconds.
Full synchronization rate	Set the full synchronization rate as needed. To prevent high synchronization rates from affecting the source database's performance, adjust the following parameters based on the read and write pressure of the source and target databases accordingly: Rows synchronized per second: Enable and set the rows synchronized per second as needed. The default value is 100000, and the value range is 1-100000. Data volume to synchronize per second: Enable this option and set the data volume as needed. The default value is 25000, the value range is 1-25000, and the unit is KB. Synchronization task concurrency: Set the concurrency of synchronization tasks as needed. The default value is 8, with a range of 1 to 64. tip You can configure this parameter when selecting Synchronization type as Full initiation. An excessively high synchronization rate may affect the performance of the source database. Adjust appropriately based on the read and write pressure of the source and target databases.

Last updated: 2026.03.17 15:11:52