Docs

Data Migration Service

DMS 1.0 Migration Guide

Create migration task - use public network

This article explains how to create a migration task using the public network in the DMS console.

Background information

The DMS supports the following source cloud providers and data source types:

AWS Cloud

Tencent Cloud

Huawei Cloud

Qiniu Cloud

Kingsoft Cloud

Google Cloud

Other S3 protocol object storage

Object Storage - Volcano Engine

warning

The DMS only supports migrating data between Volcano Engine buckets in the same region.

When migrating files, HTTP standard attributes are migrated by default, including Content-Type, Content-Disposition, Content-Encoding, Content-Language, Cache-Control, and Expires.

Prerequisites

Ensure that you have registered a Volcano Engine account and completed real name verification. For details, see Account Registration and Real Name Verification.

You have obtained relevant information about the data source, such as AccessKey ID and Secret AccessKey, Endpoint, and bucket name.

You have created a bucket in Object Storage. For details, see Create Bucket.

Notes

If the source bucket has versioning enabled, DMS only migrates the latest versions of objects to the target bucket and does not migrate previous versions.

Steps

On the Migration tasks page, click Create migration task.

On the Select network page, configure the following parameters.

Parameter	Description
Task name	Specify a name for this migration task. The naming rules are as follows: Between 3 and 32 characters. Contain only lowercase letters, numbers, or hyphens (-). The name cannot start or end with a hyphen (-).
Network type	Select Public network.

Click Next step: Set up source and destination to configure the following source and destination parameters.

Source type is Object Storage

Parameters			Description
Source type			Select the data source type. For this scenario, choose Object Storage.
Data source			Select a specific data source: Alicloud: Migrate data from Alicloud to Volcano Engine Object Storage TOS. Tencent Cloud: Migrate data from Tencent Cloud to Volcano Engine Object Storage TOS. Huawei Cloud: Migrate data from Huawei Cloud to Volcano Engine Object Storage TOS. Qiniu Cloud: Migrate data from Qiniu Cloud to Volcano Engine Object Storage TOS. Kingsoft Cloud: Migrate data from Kingsoft Cloud to Volcano Engine Object Storage TOS. Google Cloud: Migrate data from Google Cloud to Volcano Engine Object Storage TOS. Other S3 protocol storage: Supports object storage compatible with the S3 protocol and AWS Signature V4 authentication. Object Storage–Volcano Engine: Migrate data from Volcano Engine Object Storage to TOS in the same region. warning When the data source is Object Storage–Volcano Engine, you can only select buckets in the same region as the source. To migrate data across regions, use cross-region replication. For details, see Cross-region replication.
Access Key ID			Used to access the source object storage: AccessKey ID.
Access Key Secret			AccessKey Secret for accessing the source object storage.
Source bucket selection method			Set how to select the source bucket. The modes are described below: Obtain and select available source buckets: With your authorization, DMS directly reads the source storage bucket list. You do not need to fill in any information. Click Refresh to load the source bucket list, and then select a storage bucket from the Source Bucket Name list. Enter the source bucket name: Manually enter the source bucket name, source Region, and Endpoint information.
Source bucket name			Specify the name of the source object storage bucket for migration. tip Your account must have HeadBucket, ListObjects, HeadObject, and GetObject permissions for the source bucket, or the migration may not produce the expected result.
Source Region			Specify the Region where the source object storage is located.
Endpoint			Specify the access domain name for the source object storage. Enter http:// or https:// before the address. warning Only access domain names assigned by default to the source object storage are supported. Custom domain names, AP domains, and other types are not supported. Do not include the bucket name, or connection to the source will fail.
Specify migration scope	Includes tip Include and exclude functions are mutually exclusive.	After enabling Specify migration scope, you can define the object migration scope by specifying objects or object prefixes. tip When using Upload file or Download address to specify objects or object prefixes, the line break in the list file must be \n. Prefix There are three ways to specify the object prefix, as described below: Specify prefix: Enter the object prefix and click Add field to set multiple prefixes, up to 10 object prefixes supported. For example, to migrate all objects in the user/data directory of the source bucket, set the object prefix to user/data/. To migrate all files in the source bucket with the prefix event, set the object prefix to event. Upload file: If you have many objects to migrate, you can record the object prefixes in a file and upload this list file to the console. List file constraints are as follows: The uploaded list file must be in txt format. Each line in the list file is one object prefix. For multiple prefixes, enter each on a new line. The uploaded list file size cannot exceed 100MB. The number of prefixes in the list file cannot exceed 100,000. Object prefixes in the list file must comply with UTF-8 encoding standards. Download URL: Record object prefixes in the list file, then enter the public access URL of the list file containing the signed and valid object prefixes. If your object prefix list file is too large to upload using a browser, you can save the list file to object storage and enter the public access URL of the list file in object storage. List file constraints are as follows: The list file must be in txt format. Each line in the list file represents one object prefix. For multiple object prefixes, enter each on a new line. The list file size must not exceed 100MB. The number of prefixes in the list file must not exceed 100,000. The object prefixes in the list file must comply with UTF-8 encoding rules. warning When specifying an object prefix, spaces are not supported in object prefixes. Object We support three ways to specify objects as described below: Manual input: Enter the full path of the object name, excluding the bucket name. Each line must contain only one object name. For multiple objects, enter each name on a new line. You can specify up to 1,000 objects. For example, for user/data/flower.png under bucket A, you need to enter user/data/flower.png. Upload file: If you need to migrate many objects, you can record their names in a file and then upload this list file to the console. List file requirements are as follows: The uploaded list file must be in txt format. Each line in the list file should contain one object name. For multiple objects, enter each name on a new line. The uploaded list file must not exceed 1GB. Object names in the list file must comply with UTF-8 encoding rules. Download link: Record object names in a list file, then provide the public link to the signed object list file that is still within its validity period. If your object list file is large and not suitable for browser upload, you can save it to Object Storage and provide the public link of the file in Object Storage. The following are the list file constraints: The list file must be in txt format. Each line in the list file should contain one object name. For multiple objects, enter each on a new line. The size of the list file must not exceed 1GB. Object names in the list file must follow UTF-8 encoding rules. tip Object names cannot contain spaces when specifying object names. To rename a file, use object name + space + new object name. For example, you can use a.jpg b.jpg to rename the a.jpg file to b.jpg.
		Exclude	After you enable specify migration range, you can exclude specific objects and migrate only the remaining ones. Currently, only object prefixes can be used to specify exclusions. There are three supported ways to set object prefixes, as follows: Specify prefix: Enter the object prefix, then click Add field. You can set up to 10 different prefixes. For example, to exclude all objects in the user/data directory of the source bucket, set the object prefix to user/data/. To exclude all objects with the event prefix, set the object prefix to event. Upload file: If you have many objects to exclude, record their prefixes in a file and upload the list to the console. The file must meet these requirements: The uploaded list file must be in txt format. Each row in the file is an object prefix. Enter multiple prefixes on separate lines. The uploaded list file size must not exceed 5MB. The number of prefixes in the list file must not exceed 5,000. The object prefixes in the list file must comply with UTF-8 encoding rules. Download link: Record the excluded object prefixes in a list file, then provide the public access link to the signed object prefix list file that is within its validity period. If your object prefix list file is too large for browser upload, you can save it to object storage and provide its public access link. Constraints for list files are as follows: The list file must be in txt format. Each line in the list file is an object prefix. For multiple object prefixes, enter each on a new line. The list file size must not exceed 5MB. The number of prefixes in the list file must not exceed 5,000. The object prefix in the list file must comply with UTF-8 encoding rules warning Spaces are not supported in object prefixes when specifying the prefix. When specifying the object prefix by uploading a file or using a download URL, line breaks in the list file must be \n.

Source type is URL list

warning

When source type is URL list, the file size of each URL in the URL list file must not exceed 300 GB.

When source type is URL list, each line in the URL list must not exceed 65,536 characters (64 KB).

Currently, only URL list files encoded in UTF-8 are supported.

Parameter	Description
Source type	Type of data source; for this scenario, select URL list.
Upload URL list file	Select a method to upload the URL list file: Manual input: Enter the file URL directly. The constraints are as follows: Begin with http:// or https:// and must include a valid URL. Only one URL per line. Enter multiple URLs on separate lines, up to 1,000 URLs supported. Upload file: If you need to migrate many files, record the URLs in a file and upload that list to the console. The constraints are as follows: The list file must be in txt format. Each line in the list file should contain one URL to be migrated. Enter multiple URLs on separate lines. The uploaded list file must not exceed 1GB. Download address: If your URL list file is large and not suitable for browser upload, you can save the URL list file to object storage and then select Download address to enter the URL access address of the list file in object storage. The download address supports the following two ways to obtain file addresses: Select file list: Set source data authorization mode, choose source bucket name and source file name. DMS will read the corresponding storage bucket list and file. warning You can migrate data by file list only when the source data source is an object storage bucket under the current Volcano Engine account and the regions of the source and destination buckets are the same. Enter access address: Enter the public access address of the URL list file that is signed and within the validity period. tip When selecting a URL list file using Upload file or Download address, the line break in the URL list file must be \n. When migrating using Manual input or Upload file, the URL path (the part after the domain name /) is used as the object's name by default. If you want to rename the file, you can use URL + space + object file name. For example, you can access http://xxx.xxx.xxx/xxx/a.jpg b.jpg, and rename the a.jpg file to b.jpg.

Destination parameter

Parameter	Description
Destination type	Destination type; currently, only Object Storage (Volcengine TOS) is supported.
Destination write mode	Select the destination write mode. The modes are described below: Classic Mode: Migrate data using AccessKey ID and AccessKey Secret Cross-service authorization mode: Use this mode to migrate object storage buckets under the current account without entering details such as AccessKey ID or AccessKey Secret. After you grant permission, the data migration service can directly read the bucket list and files. warning If you use a subaccount to create a migration task and select cross-service authorization mode, you need to grant the subaccount permission to create IAM policies. For details, see FAQ.
Access Key ID	AccessKey ID for accessing the destination object storage.
Access Key Secret	AccessKey Secret for accessing the destination object storage.
Destination bucket name	Specify the destination bucket name in object storage. warning If the source data is Object Storage - Volcano Engine, the destination bucket must be in the same region as the source bucket. You must ensure that the current account has HeadBucket, HeadObject, PutObject, ListObjects, CreateMultipartUpload, AbortMultipartUpload, CompleteMultipartUpload, and UploadPart permissions on the destination bucket, otherwise the migration result may not meet expectations.

Click Next: Set task parameters to configure the following parameters.

warning

Click Next: Set task parameters, and the system automatically verifies the source connection information. If the information is incorrect, a permission precheck prompt dialog will appear, notifying you of insufficient source or destination permissions. Continuing might result in unexpected migration outcomes.

When the source type is URL list, you can only configure Skip archived objects, Skip failed objects, Range-get migration, Overwrite policy, and File storage method, traffic control, and sample verification parameters on the Set task parameters page.

Parameter	Description
Migrate incremental files	Disabled by default. When enabled, you can customize incremental migration parameters. The parameter descriptions are as follows: Incremental migration interval: After the initial (full) migration, this specifies how often to start an incremental migration task, in hours. Only integers from 1 to 24 are supported. Number of incremental migrations: After the initial (full) migration is complete, this specifies the number of incremental migrations to perform. Only integers from 1 to 100 are supported.
Skip archive objects	Set whether to Skip Archived Objects. This is enabled by default, as follows: Enable Skip Archived Objects: Files of delayed access storage type will not be migrated during data migration. Disable Skip Archived Objects: During data migration, thawed files of delayed access storage type will be migrated, while unthawed files of delayed access storage type will not be migrated. tip The delayed access storage types for each data source are as follows: Alicloud: Archive Storage (Archive), Cold Archive Storage (ColdArchive) Tencent Cloud: Archive Storage (ARCHIVE), Deep Archive Storage (DEEP_ARCHIVE) Huawei Cloud: Archive Storage (COLD), Deep Archive Storage (DEEP_ARCHIVE) Qiniu Cloud: Archive Storage, Deep Archive Storage Kingsoft Cloud: Archive Storage type Other S3 protocol storage: GLACIER, DEEP ARCHIVE. Object Storage - Volcano Engine: Archive Storage (ARCHIVE), Cold Archive Storage (COLD_ARCHIVE) Google Cloud Storage does not offer a delayed access storage type. If your data source is Google Cloud, there is no need to set the Skip Archive Objects parameter.
Skip failed objects	If you do not have permission for a source object during migration, that object will fail to migrate. You can set whether to skip failed objects. Enabled by default. Details: When skip failed objects is enabled, migration will continue regardless of how many objects fail. Turn off Skip Failed Objects: If 20 consecutive objects fail to migrate during the process, the task will be paused and marked as failed.
Range-get migration	Set whether to enable range-get migration. Disabled by default. After enabling range-get migration, DMS supports migrating files with resumable transfers. If network interruptions occur, files are retransferred starting from the breakpoint, reducing extra traffic and lowering costs. warning The range-get migration parameter can be set only when the source type is URL list. Before enabling range-get migration, ensure the source supports this feature; otherwise, range-get migration will not take effect.
Renaming files	Disabled by default. When enabled, you can set file renaming rules. File renaming supports regular expression replacement. Enter the regular expression for files to rename and the replacement string. For example: Add the prefix prefix-path/: Set the regular expression to ^ and the replacement text to prefix-path/. Change the prefix prefix-old/ to prefix-new/: Set the regular expression to ^prefix-old/ and the replacement text to prefix-new/. Remove the prefix prefix-old/: Set the regular expression to ^prefix-old/ and leave the replacement text empty. Change the suffix .txt to .csv: Set the regular expression to .txt$ and the replacement text to .csv. Rename the file abc.jpg to aaa.jpg: Set the regular expression to abc and the replacement text to aaa.
List with delimiter	Due to source bucket configurations (such as bucket acceleration), TOS can only access files in the root directory of the source bucket. Delimiter-based listing lets you access files in multi-level directories, as explained below: Enable delimiter-based listing: TOS can access files in multi-level directories of the source bucket, ensuring all files are migrated to TOS. Disable delimiter-based listing: Migrating files will be faster, but if bucket acceleration is enabled on the source bucket, not all files may be migrated to TOS. warning If you enable delimiter-based listing, set specified migration scope to include, and specify the range using a prefix (including specified prefix, file upload, and download address), the prefix must end with a slash (/). If the source bucket does not return files in lexicographical order when listing files, the results may be incomplete or cause an infinite loop, preventing the migration task from completing successfully We recommend mounting the source bucket locally and migrating files using DMS 2.0 semi-managed mode. For details, see DMS 2.0 Quick Start.
Overwrite policy	You can specify the overwrite policy for files with the same name in the source and destination. Policy descriptions are as follows: Overwrite all: For files with the same name, the source directly overwrites the destination without any checks. Do not overwrite: For files with the same name, the source is skipped without any checks. Overwrite based on last modified time: When a file with the same name exists on both the source and destination, the lastmodify field (last modified time) determines whether to overwrite. Overwriting happens only if the source file’s last modified time is later than the destination file’s.
Ignore destination callback	When the covering strategy for the migration task is set to No overwrite, DMS checks whether the files to be migrated already exist at the destination. If the destination bucket has HeadObject request back-to-source enabled and the source site has the file to be migrated, DMS will consider the file as already existing in the destination bucket. As a result, the migration will be skipped, leading to missing files in the final migration count. After enabling Ignore destination back-to-source, DMS ignores the back-to-source rules of the destination bucket and migrates the files directly. If you use the "don't overwrite same-named files" strategy and your destination bucket has back-to-source enabled, we recommend turning on the "Ignore destination back-to-source" feature.
File storage method	Set the storage class for files migrated to TOS. You can select from Standard Storage, Keep Original Storage Attributes, Infrequent Access Storage, Archive Flashback Storage, Intelligent Tiering Storage, Archive Storage, Cold Archive Storage, and Deep Cold Archive Storage. For details about TOS storage classes, see Storage type. The following are parameter descriptions for Retain original storage attributes and Use intelligent tiering for all files: Retain original storage attributes: When migrating to TOS, the file storage type is set according to the corresponding relationship between S3 and TOS. For more information, see Correspondence between S3 and TOS when migrating data. Use intelligent tiering for all files: When migrating to TOS, files are stored in the Intelligent Tiering frequent access layer. tip Deep Cold Archive is currently in closed beta in TOS. To set the migrated files to Deep Cold Archive storage type, contact your account manager. If the migration source uses the delayed access storage type, change its status to instant access before migration. Otherwise, the migration will fail. If the availability zone attributes of the destination and source files differ, the destination takes precedence by default. When the source type is a URL list, retaining the original storage attributes is not supported.
File migration start time	Filter files to be migrated based on the last modified time. The default option is Migrate all. Parameter instructions are as follows: Migrate all: Migrate all files in the source bucket. Specify time range: Filter files for migration. Three methods are supported, as described below: Specify start time: Only set the start date; leave the end date blank. Only files modified after this date are migrated. Specify end time: Only set the end date; leave the start date blank. Only files modified before this date are migrated. Specify time range: Set both start and end dates. Only files modified within this time frame are migrated.
Traffic control	Disabled by default; when disabled, the default speed is 1000 MB/s. Enable the speed switch and click Add field to set migration task throttling periods and maximum speed. Speed limit details are as follows: The default maximum speed is 100 MB/s. The maximum speed can be set from 0 MB/s to 1000 MB/s.
Sampling verification	Disabled by default. When sample verification is enabled, DMS retrieves 100 bytes of each object from the source and 100 corresponding bytes from the destination for comparison to ensure data consistency between the source and destination. For a detailed introduction to sampling verification, see Data Consistency Check. warning Using the sampling verification feature generates additional outbound public traffic at the source, which incurs corresponding charges from the source object storage.

Click Next: Confirm task information to review the migration task parameters on this page.

After confirming the parameters, read and select Storage product and service terms, then click Submit.

warning

After submitting, a cost reminder dialog will pop up. The DMS itself does not charge any fees. However, migrating data from the public network to Volcano Engine generates public network traffic fees, which are charged by the storage service provider of the source site.

In the Fee Reminder dialog box, click Acknowledged, Next.

(Optional) If you select Cross-Service Authorization, you also need to click Confirm in the popup Cross-Service Authorization dialog box.

tip

After the task is created, it will start automatically. You can view its creation time and status in the Migration Task List. Additionally, click the custom list field icon in the upper-right corner of the Migration Task List page, select Storage Migration Progress and File Quantity Migration Progress to view the migration progress.

Last updated: 2026.04.08 18:10:43