Lake AI Service
Lake AI Service
- Documentation
Lake AI ServiceMetadata managementStorage optimization and governancePartition lifecycle management
Copy page
Download PDFCopy pageDownload PDFLAS Catalog table files are stored by default in TOS standard storage. After configuring table lifecycle management, the system automatically archives long-unused data to TOS archive storage based on the update time of the table or partition, reducing storage costs. This is suitable for scenarios with large volumes of cold data and a need to control data lake storage costs. This document describes how to configure and delete table lifecycles, constraints, and common troubleshooting.
- Currently, only Hive tables support the storage optimization—partition lifecycle management feature.
- The user must have write permission for the target table; otherwise, they cannot use the partition lifecycle management feature for the table.
- Cleaning partition data will incur queue resource fees.
- The billing item is: cpu_hour_{region}, which refers to the queue resource fee billed by the hour (pay-as-you-go, billed hourly).
- The resource consumption for executing partition cleanup tasks for each customer does not exceed 10 CU per hour.
On the left-side list of the LAS Console > Metadata > Catalog page, select the target table to enter the table details page. Select the Storage optimization>partition lifecycle management tab, click the Enable button, and go to the partition lifecycle management configuration page.
Configure partition lifecycle management according to actual business requirements. Setting a very short duration is not recommended, because retrieving data from TOS archive storage to TOS standard storage requires an unfreezing operation in TOS and will incur corresponding retrieval fees. For retrieval fee reference, see: Data retrieval fees .
warning
Currently, there are certain constraints on TOS's lifecycle management, and LAS follows these same constraints:
- Effective time of lifecycle rules : After a lifecycle rule is created, it does not immediately take effect on all objects in the bucket. There will be a certain delay. That is, within 24 hours after the lifecycle rule is created, TOS will load the rule. After the rule is loaded, TOS will start execution within 24 hours.
- Rule execution priority : The execution priority of lifecycle rules varies depending on the overlap status of different prefixes.
- Quantity : A single bucket can have up to 1,000 lifecycle rules, and the total size of the JSON descriptions for all lifecycle rules in a bucket cannot exceed 20 KB. The prefix length of a rule (that is, the storage path of the table) cannot exceed 1,024 bytes.
- Limitations after archiving data: Before using the CopyObject, GetObject, SetObjectMeta, or UploadPartCopy interfaces to set archived storage objects, or before moving, renaming, or downloading archived storage objects, the object must be restored first; otherwise, the operation will fail.
- When you perform an insert overwrite operation on table data, the old data will be deleted and new data will be uploaded. The upload time of the new data files is the partition modification time for the table. The lifecycle countdown for the table will start from the last update time.
- When you perform an insert into operation on table data, the update time of historical data will not change, and the update time of newly inserted data will be the latest time. This may result in a situation where old partition data in the table meets the archiving criteria and enters archive storage, while new partition data remains in standard storage.
Parameters | Note |
|---|---|
partition archive rule configuration | Lifecycle management supports configuration according to three types of rules:
warning Partition archiving is an asynchronous task. Partition lifecycle rules will take effect within 24 hours after being set. |
partition field | When configuring lifecycle by partition name, you can select the partition field. The partition field is specified when creating or editing a table. |
date format | The format of the lifecycle date can be selected when configuring lifecycle by partition name.
|
reserved partition range | You can retain partition data for a specified number of days. |
After the feature is enabled, partition archiving takes effect within 24 hours. You can view optimization records and results on the partition lifecycle rule page.
On the left-side list of the LAS Console > Metadata > Catalog page, select the target table to enter the table details page. Select the Storage optimization>partition lifecycle management tab:
- Click the button to disable partition lifecycle management.
- Click Modify configuration on the right to modify the partition archiving rule configuration.
tip
If the data table is deleted, the table's lifecycle management feature will be automatically deleted together with the table.
- How to configure lifecycle when the number of tables exceeds 1,000
- It is recommended to set the lifecycle at the database level for the database storage path. On the TOS lifecycle management page, set the lifecycle rule with the database storage path as the prefix. This provides a default lifecycle rule at the database level. If the table's lifecycle rule is different from the database's, enable the table's lifecycle rule.
- A single TOS bucket is limited to 1,000 rules. You can consider using multiple TOS buckets.
- Error: Found two rules with the same prefix
Prefix overlap must be enabled. For detailed instructions, see: Setting prefix overlap . - Error: The operation is not valid for the object's storage class
Archived data cannot be moved or read, meaning operations such as renaming or selecting tables are restricted. The data must be restored to standard storage before it can be used. For detailed instructions, see: Restore files .
