Docs

Torch Object Storage

Data migration

Online migration with zero service disruption

This topic offers a solution for migrating your data online from third-party services to TOS without disrupting your services.

Business scenario

If your business relies on third-party services and needs to transition to TOS, maintaining service stability and preventing any negative operational impact during the data migration process is essential. This topic will guide you through using Data Migration Service (DMS) and TOS mirroring rules to ensure a seamless migration process.

How it works

Migration flowchart

Migration process

Migrate all existing data to TOS using DMS, Rclone, or other tools. Your original business system will continue to use the original storage service during this process.

Configure a mirroring rule for your TOS bucket, specifying the origin URL where your source data resides.

Update your business system to switch to TOS for data storage. After the update, newly generated data will be written directly to TOS. Any previously unmigrated data will be retrieved from the origin server via mirroring rules to ensure business continuity.

To complete the data and business migration, use DMS, Rclone, or other tools to migrate any incremental data generated during the migration period.

Migration steps

Prerequisites

Select the migration tool based on your previous object storage service provider and interface type.

Assess the storage capacity, object count, bandwidth, and QPS capabilities of your previous service provider.

Step 1: Migrate all existing data using DMS

This section demonstrates the migration process using DMS. For instructions on migrating with Rclone, see Accessing TOS with Rclone.

Currently, DMS supports data migration via a public network or a dedicated line. You can migrate data from object storage solutions from multiple cloud service providers, or migrate data using a list of URLs.

On the Migration tasks page, click Create migration task.

On the Set the source and destination page, configure the source and destination parameters.

tip

For more information on the parameters for creating a migration task, see Create migration task.

For a list of data sources supported by DMS, see Data sources.

When migrating for the first time, select migrate all for Migration file start time.

If you are concerned that migrating data will use up your business's bandwidth and disrupt normal operations, you can set Flow control parameters to specify rate-limiting periods and maximum rates for migration tasks. For more information, see Create migration task.

Once you have finished configuring, click Submit.

Step 2: Configure a mirroring rule

Once the mirroring rule is set, if the requested object does not exist in the bucket, TOS fetches the object from the specified origin server and returns it to the requester. Simultaneously, TOS writes the object to the bucket.

In the left navigation pane, click Buckets. In the Buckets list, click the bucket you want to manage.

In the left navigation pane, select Basic configurations > Back-to-Origin rules. On the Back-to-Origin rules page, click Create rule.

On the Create Back-to-Origin Rule page, configure the following parameters.

Parameter	Description
Back-to-Origin method	Select Mirroring.
Back-to-Origin condition	Currently, only HTTP Status Code 404 is supported.
Back-to-Origin address	Configure the address to which TOS will redirect once the mirroring conditions are met. If the source bucket is from a third-party cloud service provider, enter the bucket's domain name. Protocol: Select https or http. Domain name: It must be in domain format. IP addresses are not supported. Description Click Add back to source address to add more addresses. You can add up to 5 back-to-origin addresses. If you enter multiple addresses, TOS will randomly select one to send the request. If the request to that address fails, TOS will select a different address.
Whether to carry request parameter	Enable this option.
Whether to redirect based on the origin server	Select whether to redirect the request based on the 3XX status code of the origin server. A request can only be redirected once. This option is disabled by default.
HTTP Header Delivery Rules	Select the rule for passing response headers when TOS accesses the specified origin server. The options are as follows: Transparently Transmit All HTTP Headers (Default): When selected, all parameters are passed through, except the following headers: Host, Content-Length, Date, and Authorization. Transparently Transmit Specified HTTP Headers: Define the specific headers that are allowed to be passed through. You can add a maximum of 10 headers. Transparently Transmit No HTTP Headers: When selected, all parameters are blocked from being passed through. Not Transparently Transmit Specified HTTP Headers: Define the specific headers prohibited from being passed through. You can add a maximum of 10 headers.

Once you have finished configuring, click Submit.

Step 3: Integrate your business system with TOS

After migrating data to TOS, follow these steps to switch your business to TOS:

Create an Identity and Access Management (IAM) user, grant it the necessary permissions, and obtain its access key ID and secret access key. Refer to the following documentation for instructions:

Create user

Configure an IAM policy

View the Access Key ID and Secret Access Key

Select the appropriate TOS SDK for integration based on your business needs. For a list of SDK languages supported by TOS, see SDK overview.

Use the TOS SDK to develop and test business interfaces for TOS.

After testing is complete, upgrade your business system software during off-peak hours to switch to TOS.

Step 4: Migrate incremental data using DMS

The process for migrating incremental data is the same as in Step 1. For more information, see Create migration task.

Recommended configurations when creating the migration task:

Incremental migration interval: Set the interval to 10 hours if the data volume to migrate is less than 100 TiB. Set it to 24 hours if the data volume is greater than 100 TiB.

Incremental migrations count: 5.

Overwrite policy: Do not overwrite.

Migration file start time: Set this to a time earlier than the start time of the first full data migration task.

Last updated: 2026.09.16 10:50:51