This article explains how to create a migration task using the public network in the DMS console. ## Background information The DMS supports the following source cloud providers and data source types: * AWS Cloud * Tencent Cloud * Huawei Cloud * Qiniu Cloud * Kingsoft Cloud * Google Cloud * Other S3 protocol object storage * Object Storage \- Volcano Engine

warning

The DMS only supports migrating data between Volcano Engine buckets in the same region.

When migrating files, HTTP standard attributes are migrated by default, including Content\-Type, Content\-Disposition, Content\-Encoding, Content\-Language, Cache\-Control, and Expires. ## Prerequisites * Ensure that you have registered a Volcano Engine account and completed real name verification. For details, see [Account Registration](https://www.volcengine.com/docs/6261/64925) and [Real Name Verification](https://www.volcengine.com/docs/6261/64935). * You have obtained relevant information about the data source, such as AccessKey ID and Secret AccessKey, Endpoint, and bucket name. * You have created a bucket in Object Storage. For details, see [Create Bucket](https://www.volcengine.com/docs/6349/75024). ## Notes If the source bucket has versioning enabled, DMS only migrates the latest versions of objects to the target bucket and does not migrate previous versions. ## Steps 1. Log in to the [DMS Console](https://console.volcengine.com/dms). 2. On the **Migration tasks** page, click **Create migration task**. 3. On the **Select network** page, configure the following parameters. |**Parameter** |**Description** | |---|---| |Task name |Specify a name for this migration task. The naming rules are as follows:

* Between 3 and 32 characters.

* Contain only lowercase letters, numbers, or hyphens (\-).

* The name cannot start or end with a hyphen (\-). | |Network type |Select **Public network**. | 4. Click **Next step: Set up source and destination** to configure the following source and destination parameters. * **Source type** is **Object Storage**![图片](https://portal.volccdn.com/obj/volcfe/cloud-universal-doc/upload_782f5b7cf2c54b0de7230bb5e70ca928.png) |**Parameters** ||**Description** | |---|---|---| |Source type ||Select the data source type. For this scenario, choose **Object Storage**. | |Data source ||Select a specific data source:

* **Alicloud**: Migrate data from Alicloud to Volcano Engine Object Storage TOS.

* **Tencent Cloud**: Migrate data from Tencent Cloud to Volcano Engine Object Storage TOS.

* **Huawei Cloud**: Migrate data from Huawei Cloud to Volcano Engine Object Storage TOS.

* **Qiniu Cloud**: Migrate data from Qiniu Cloud to Volcano Engine Object Storage TOS.

* **Kingsoft Cloud**: Migrate data from Kingsoft Cloud to Volcano Engine Object Storage TOS.

* **Google Cloud**: Migrate data from Google Cloud to Volcano Engine Object Storage TOS.

* **Other S3 protocol storage**: Supports object storage compatible with the S3 protocol and AWS Signature V4 authentication.

* **Object Storage–Volcano Engine**: Migrate data from Volcano Engine Object Storage to TOS in the same region.

warning

When the data source is Object Storage–Volcano Engine, you can only select buckets in the same region as the source. To migrate data across regions, use cross\-region replication. For details, see Cross\-region replication.

| |Access Key ID ||Used to access the source object storage: AccessKey ID. | |Access Key Secret ||AccessKey Secret for accessing the source object storage. | |Source bucket selection method ||Set how to select the source bucket. The modes are described below:

* **Obtain and select available source buckets**: With your authorization, DMS directly reads the source storage bucket list. You do not need to fill in any information. Click **Refresh to load the source bucket list**, and then select a storage bucket from the **Source Bucket Name** list.

* **Enter the source bucket name**: Manually enter the source bucket name, source Region, and Endpoint information. | |Source bucket name ||Specify the name of the source object storage bucket for migration.

tip

Your account must have HeadBucket, ListObjects, HeadObject, and GetObject permissions for the source bucket, or the migration may not produce the expected result.

| |Source Region ||Specify the Region where the source object storage is located. | |Endpoint ||Specify the access domain name for the source object storage. Enter `http://` or `https://` before the address.

warning

Only access domain names assigned by default to the source object storage are supported. Custom domain names, AP domains, and other types are not supported.

Do not include the bucket name, or connection to the source will fail.

| |Specify migration scope |Includes

tip

Include and exclude functions are mutually exclusive.

|After enabling **Specify migration scope**, you can define the object migration scope by specifying objects or object prefixes.

tip

When using Upload file or Download address to specify objects or object prefixes, the line break in the list file must be \n.

* **Prefix**

There are three ways to specify the object prefix, as described below:

* **Specify prefix**: Enter the object prefix and click **Add field** to set multiple prefixes, up to 10 object prefixes supported. For example, to migrate all objects in the `user/data` directory of the source bucket, set the **object prefix** to `user/data/`. To migrate all files in the source bucket with the prefix `event`, set the **object prefix** to `event`.

* **Upload file**: If you have many objects to migrate, you can record the object prefixes in a file and upload this list file to the console. List file constraints are as follows:

* The uploaded list file must be in `txt` format.

* Each line in the list file is one object prefix. For multiple prefixes, enter each on a new line.

* The uploaded list file size cannot exceed 100MB.

* The number of prefixes in the list file cannot exceed 100,000.

* Object prefixes in the list file must comply with UTF\-8 encoding standards.

* **Download URL**: Record object prefixes in the list file, then enter the public access URL of the list file containing the signed and valid object prefixes. If your object prefix list file is too large to upload using a browser, you can save the list file to object storage and enter the public access URL of the list file in object storage. List file constraints are as follows:

* The list file must be in `txt` format.

* Each line in the list file represents one object prefix. For multiple object prefixes, enter each on a new line.

* The list file size must not exceed 100MB.

* The number of prefixes in the list file must not exceed 100,000.

* The object prefixes in the list file must comply with UTF\-8 encoding rules.

warning

When specifying an object prefix, spaces are not supported in object prefixes.

* **Object**

We support three ways to specify objects as described below:

* **Manual input**: Enter the full path of the object name, excluding the bucket name. Each line must contain only one object name. For multiple objects, enter each name on a new line. You can specify up to 1,000 objects. For example, for `user/data/flower.png` under bucket A, you need to enter `user/data/flower.png`.

* **Upload file**: If you need to migrate many objects, you can record their names in a file and then upload this list file to the console. List file requirements are as follows:

* The uploaded list file must be in `txt` format.

* Each line in the list file should contain one object name. For multiple objects, enter each name on a new line.

* The uploaded list file must not exceed 1GB.

* Object names in the list file must comply with UTF\-8 encoding rules.

* **Download link**: Record object names in a list file, then provide the public link to the signed object list file that is still within its validity period. If your object list file is large and not suitable for browser upload, you can save it to Object Storage and provide the public link of the file in Object Storage. The following are the list file constraints:

* The list file must be in `txt` format.

* Each line in the list file should contain one object name. For multiple objects, enter each on a new line.

* The size of the list file must not exceed 1GB.

* Object names in the list file must follow UTF\-8 encoding rules.

tip

Object names cannot contain spaces when specifying object names.

To rename a file, use object name + space + new object name. For example, you can use a.jpg b.jpg to rename the a.jpg file to b.jpg.

| ||Exclude |After you enable **specify migration range**, you can exclude specific objects and migrate only the remaining ones. Currently, only object prefixes can be used to specify exclusions. There are three supported ways to set object prefixes, as follows:

* **Specify prefix**: Enter the object prefix, then click **Add field**. You can set up to 10 different prefixes. For example, to exclude all objects in the `user/data` directory of the source bucket, set the **object prefix** to `user/data/`. To exclude all objects with the `event` prefix, set the **object prefix** to `event`.

* **Upload file**: If you have many objects to exclude, record their prefixes in a file and upload the list to the console. The file must meet these requirements:

* The uploaded list file must be in `txt` format.

* Each row in the file is an object prefix. Enter multiple prefixes on separate lines.

* The uploaded list file size must not exceed 5MB.

* The number of prefixes in the list file must not exceed 5,000.

* The object prefixes in the list file must comply with UTF\-8 encoding rules.

* **Download link**: Record the excluded object prefixes in a list file, then provide the public access link to the signed object prefix list file that is within its validity period. If your object prefix list file is too large for browser upload, you can save it to object storage and provide its public access link. Constraints for list files are as follows:

* The list file must be in `txt` format.

* Each line in the list file is an object prefix. For multiple object prefixes, enter each on a new line.

* The list file size must not exceed 5MB.

* The number of prefixes in the list file must not exceed 5,000.

* The object prefix in the list file must comply with UTF\-8 encoding rules

warning

Spaces are not supported in object prefixes when specifying the prefix.

When specifying the object prefix by uploading a file or using a download URL, line breaks in the list file must be \n.

| * **Source type** is **URL list**

warning

When source type is URL list, the file size of each URL in the URL list file must not exceed 300 GB.

When source type is URL list, each line in the URL list must not exceed 65,536 characters (64 KB).

Currently, only URL list files encoded in UTF\-8 are supported.

|**Parameter** |**Description** | |---|---| |Source type |Type of data source; for this scenario, select **URL list**. | |Upload URL list file |Select a method to upload the URL list file:

* **Manual input**: Enter the file URL directly. The constraints are as follows:

* Begin with `http://` or `https://` and must include a valid URL.

* Only one URL per line. Enter multiple URLs on separate lines, up to 1,000 URLs supported.

* **Upload file**: If you need to migrate many files, record the URLs in a file and upload that list to the console. The constraints are as follows:

* The list file must be in `txt` format.

* Each line in the list file should contain one URL to be migrated. Enter multiple URLs on separate lines.

* The uploaded list file must not exceed 1GB.

* **Download address**: If your URL list file is large and not suitable for browser upload, you can save the URL list file to object storage and then select **Download address** to enter the URL access address of the list file in object storage. The download address supports the following two ways to obtain file addresses:

* **Select file list**: Set **source data authorization mode**, choose **source bucket name** and **source file name**. DMS will read the corresponding storage bucket list and file.

warning

You can migrate data by file list only when the source data source is an object storage bucket under the current Volcano Engine account and the regions of the source and destination buckets are the same.

* **Enter access address**: Enter the public access address of the URL list file that is signed and within the validity period.

tip

When selecting a URL list file using Upload file or Download address, the line break in the URL list file must be \n.

When migrating using Manual input or Upload file, the URL path (the part after the domain name /) is used as the object's name by default. If you want to rename the file, you can use URL + space + object file name. For example, you can access http://xxx.xxx.xxx/xxx/a.jpg b.jpg, and rename the a.jpg file to b.jpg.

| * Destination parameter ![图片](https://portal.volccdn.com/obj/volcfe/cloud-universal-doc/upload_40a4965d3d18816e2fa172ec4edd3bcc.png) |**Parameter** |**Description** | |---|---| |Destination type |Destination type; currently, only **Object Storage** (Volcengine TOS) is supported. | |Destination write mode |Select the destination write mode. The modes are described below:

* **Classic Mode**: Migrate data using AccessKey ID and AccessKey Secret

* **Cross\-service authorization mode**: Use this mode to migrate object storage buckets under the current account without entering details such as AccessKey ID or AccessKey Secret. After you grant permission, the data migration service can directly read the bucket list and files.

warning

If you use a subaccount to create a migration task and select cross\-service authorization mode, you need to grant the subaccount permission to create IAM policies. For details, see FAQ.

warning

If the source data is Object Storage \- Volcano Engine, the destination bucket must be in the same region as the source bucket.

You must ensure that the current account has HeadBucket, HeadObject, PutObject, ListObjects, CreateMultipartUpload, AbortMultipartUpload, CompleteMultipartUpload, and UploadPart permissions on the destination bucket, otherwise the migration result may not meet expectations.

| 5. Click **Next: Set task parameters** to configure the following parameters.

warning

Click Next: Set task parameters, and the system automatically verifies the source connection information. If the information is incorrect, a permission precheck prompt dialog will appear, notifying you of insufficient source or destination permissions. Continuing might result in unexpected migration outcomes.

When the source type is URL list, you can only configure Skip archived objects, Skip failed objects, Range\-get migration, Overwrite policy, and File storage method, traffic control, and sample verification parameters on the Set task parameters page.

![图片](https://portal.volccdn.com/obj/volcfe/cloud-universal-doc/upload_4c90a6c6e9870f261fac6fff31e13059.png) |**Parameter** |**Description** | |---|---| |Migrate incremental files |Disabled by default. When enabled, you can customize incremental migration parameters. The parameter descriptions are as follows:

* **Incremental migration interval**: After the initial (full) migration, this specifies how often to start an incremental migration task, in hours. Only integers from 1 to 24 are supported.

* **Number of incremental migrations**: After the initial (full) migration is complete, this specifies the number of incremental migrations to perform. Only integers from 1 to 100 are supported. | |Skip archive objects |Set whether to **Skip Archived Objects**. This is enabled by default, as follows:

* Enable **Skip Archived Objects**: Files of delayed access storage type will not be migrated during data migration.

* Disable **Skip Archived Objects**: During data migration, thawed files of delayed access storage type will be migrated, while unthawed files of delayed access storage type will not be migrated.

tip

The delayed access storage types for each data source are as follows:

Alicloud: Archive Storage (Archive), Cold Archive Storage (ColdArchive)

Tencent Cloud: Archive Storage (ARCHIVE), Deep Archive Storage (DEEP_ARCHIVE)

Huawei Cloud: Archive Storage (COLD), Deep Archive Storage (DEEP_ARCHIVE)

Qiniu Cloud: Archive Storage, Deep Archive Storage

Kingsoft Cloud: Archive Storage type

Other S3 protocol storage: GLACIER, DEEP ARCHIVE.

Object Storage \- Volcano Engine: Archive Storage (ARCHIVE), Cold Archive Storage (COLD_ARCHIVE)

Google Cloud Storage does not offer a delayed access storage type. If your data source is Google Cloud, there is no need to set the Skip Archive Objects parameter.

| |Skip failed objects |If you do not have permission for a source object during migration, that object will fail to migrate. You can set whether to **skip failed objects**. Enabled by default. Details:

* When **skip failed objects** is enabled, migration will continue regardless of how many objects fail.

* Turn off **Skip Failed Objects**: If 20 consecutive objects fail to migrate during the process, the task will be paused and marked as failed. | |Range\-get migration |Set whether to enable **range\-get migration**. Disabled by default. After enabling **range\-get migration**, DMS supports migrating files with resumable transfers. If network interruptions occur, files are retransferred starting from the breakpoint, reducing extra traffic and lowering costs.

warning

The range\-get migration parameter can be set only when the source type is URL list.

Before enabling range\-get migration, ensure the source supports this feature; otherwise, range\-get migration will not take effect.

| |Renaming files |Disabled by default. When enabled, you can set file renaming rules. File renaming supports regular expression replacement. Enter the regular expression for files to rename and the replacement string. For example:

* Add the prefix `prefix-path/`: Set the regular expression to `^` and the replacement text to `prefix-path/`.

* Change the prefix `prefix-old/` to `prefix-new/`: Set the regular expression to `^prefix-old/` and the replacement text to `prefix-new/`.

* Remove the prefix `prefix-old/`: Set the regular expression to `^prefix-old/` and leave the replacement text empty.

* Change the suffix `.txt` to `.csv`: Set the regular expression to `.txt$` and the replacement text to `.csv`.

* Rename the file `abc.jpg` to `aaa.jpg`: Set the regular expression to `abc` and the replacement text to `aaa`. | |List with delimiter |Due to source bucket configurations (such as bucket acceleration), TOS can only access files in the root directory of the source bucket. Delimiter\-based listing lets you access files in multi\-level directories, as explained below:

* Enable **delimiter\-based listing**: TOS can access files in multi\-level directories of the source bucket, ensuring all files are migrated to TOS.

* Disable **delimiter\-based listing**: Migrating files will be faster, but if bucket acceleration is enabled on the source bucket, not all files may be migrated to TOS.

warning

If you enable delimiter\-based listing, set specified migration scope to include, and specify the range using a prefix (including specified prefix, file upload, and download address), the prefix must end with a slash (/).

If the source bucket does not return files in lexicographical order when listing files, the results may be incomplete or cause an infinite loop, preventing the migration task from completing successfully We recommend mounting the source bucket locally and migrating files using DMS 2.0 semi\-managed mode. For details, see DMS 2.0 Quick Start.

| |Overwrite policy |You can specify the overwrite policy for files with the same name in the source and destination. Policy descriptions are as follows:

* **Overwrite all**: For files with the same name, the source directly overwrites the destination without any checks.

* **Do not overwrite**: For files with the same name, the source is skipped without any checks.

* **Overwrite based on last modified time**: When a file with the same name exists on both the source and destination, the `lastmodify` field (last modified time) determines whether to overwrite. Overwriting happens only if the source file’s last modified time is later than the destination file’s. | |Ignore destination callback |When the **covering strategy** for the migration task is set to **No overwrite**, DMS checks whether the files to be migrated already exist at the destination. If the destination bucket has HeadObject request back\-to\-source enabled and the source site has the file to be migrated, DMS will consider the file as already existing in the destination bucket. As a result, the migration will be skipped, leading to missing files in the final migration count.

After enabling **Ignore destination back\-to\-source**, DMS ignores the back\-to\-source rules of the destination bucket and migrates the files directly. If you use the "don't overwrite same\-named files" strategy and your destination bucket has back\-to\-source enabled, we recommend turning on the "Ignore destination back\-to\-source" feature. | |File storage method |Set the storage class for files migrated to TOS. You can select from **Standard Storage**, **Keep Original Storage Attributes**, **Infrequent Access Storage**, **Archive Flashback Storage**, **Intelligent Tiering Storage**, **Archive Storage**, **Cold Archive Storage**, and **Deep Cold Archive Storage**. For details about TOS storage classes, see [Storage type](https://www.volcengine.com/docs/6349/104493). The following are parameter descriptions for **Retain original storage attributes** and **Use intelligent tiering for all files**:

* **Retain original storage attributes**: When migrating to TOS, the file storage type is set according to the corresponding relationship between S3 and TOS. For more information, see [Correspondence between S3 and TOS when migrating data](https://www.volcengine.com/docs/6500/696757).

* **Use intelligent tiering for all files**: When migrating to TOS, files are stored in the Intelligent Tiering frequent access layer.

tip

Deep Cold Archive is currently in closed beta in TOS. To set the migrated files to Deep Cold Archive storage type, contact your account manager.

If the migration source uses the delayed access storage type, change its status to instant access before migration. Otherwise, the migration will fail.

If the availability zone attributes of the destination and source files differ, the destination takes precedence by default.

When the source type is a URL list, retaining the original storage attributes is not supported.

| |File migration start time |Filter files to be migrated based on the last modified time. The default option is **Migrate all**. Parameter instructions are as follows:

* **Migrate all**: Migrate all files in the source bucket.

* **Specify time range**: Filter files for migration. Three methods are supported, as described below:

* Specify start time: Only set the start date; leave the end date blank. Only files modified after this date are migrated.

* Specify end time: Only set the end date; leave the start date blank. Only files modified before this date are migrated.

* Specify time range: Set both start and end dates. Only files modified within this time frame are migrated. | |Traffic control |Disabled by default; when disabled, the default speed is 1000 MB/s.

Enable the speed switch and click **Add field** to set migration task throttling periods and maximum speed. Speed limit details are as follows:

* The default maximum speed is 100 MB/s.

* The maximum speed can be set from 0 MB/s to 1000 MB/s. | |Sampling verification |Disabled by default. When **sample verification** is enabled, DMS retrieves 100 bytes of each object from the source and 100 corresponding bytes from the destination for comparison to ensure data consistency between the source and destination. For a detailed introduction to sampling verification, see [Data Consistency Check](https://www.volcengine.com/docs/6500/1154510).

warning

Using the sampling verification feature generates additional outbound public traffic at the source, which incurs corresponding charges from the source object storage.

| 6. Click **Next: Confirm task information** to review the migration task parameters on this page. 7. After confirming the parameters, read and select **Storage product and service terms**, then click **Submit**.

warning

After submitting, a cost reminder dialog will pop up. The DMS itself does not charge any fees. However, migrating data from the public network to Volcano Engine generates public network traffic fees, which are charged by the storage service provider of the source site.

8. In the **Fee Reminder** dialog box, click **Acknowledged, Next**. 9. (Optional) If you select **Cross\-Service Authorization**, you also need to click **Confirm** in the popup **Cross\-Service Authorization** dialog box.

tip

After the task is created, it will start automatically. You can view its creation time and status in the Migration Task List. Additionally, click the custom list field icon in the upper\-right corner of the Migration Task List page, select Storage Migration Progress and File Quantity Migration Progress to view the migration progress.

Data Migration Service