Lake AI Service
Lake AI Service
- Documentation
Lake AI ServiceMetadata managementStorage optimization and governanceOrphan file management
Copy page
Download PDFCopy pageDownload PDFLAS provides a variety of storage optimization features, helping users improve query performance and optimize storage resources through capabilities such as orphan file governance. The following provides a detailed description and operation guide for the orphan file governance feature within storage optimization.
Currently, only Iceberg tables are supported for the storage optimization - orphan file governance feature.
Enable and configure
Log in to LAS Console , select the region in the upper left corner as appropriate, select Lake management > Metadata in the left navigation pane, select the corresponding Database > Data Table, and go to the data table list page.
Select the target table, navigate to the Table Details > Storage optimizationpage, enable the Orphan file governancefeature, and configure the orphan file governance parameters.
Parameters
Configuration description
Resource allocation
Set the number of resource CUs allocated for the current orphan file governance task.
Currently, public queue resources are supported for executing storage optimization tasks. After setting resource allocation, the corresponding quota of public queue resources will be occupied when executing storage optimization tasks.Orphan file retention policy
Set the retention time range for orphan files. Orphan files that exceed the configured time range will be cleaned up during optimization task execution.
- If the time range configured here conflicts with the
older_thanconfiguration in Custom configuration below, the configuration in Custom configuration will take precedence.
Scheduling
Select whether to enable periodic scheduling.
- Disable periodic scheduling: After task configuration is completed, you need to manually trigger task execution.
- Enable periodic scheduling: You also need to configure the scheduling strategy, including granularity and interval. After task configuration is completed, tasks will be automatically executed according to the configured scheduling cycle.
Stop Pre Task
Configure whether to forcibly terminate the preceding optimization task. After enabling the switch, the preceding optimization task is forcibly terminated each time a scheduled task is executed.
Custom configuration
Configure custom parameters involved in the orphan file governance task as needed during its execution. For details, see Reference: Custom parameter description below.
- If the time range configured here conflicts with the
Reference: Custom parameter description
Iceberg
The following Iceberg custom parameters are supported for configuration.
Custom parameters | Parameter description |
|---|---|
older_than | Only delete orphan files that are older than the specified timestamp (to prevent accidental deletion of temporary files being written). |
location | Specify the table path to scan (typically used for partitioned tables or custom locations). |
dry_run | Preview mode: only lists files to be deleted without actually deleting them. |
max_concurrent_deletes | Controls the maximum number of threads for concurrent file deletion. |
file_list_view | (Reserved field; in some implementations, used to provide the view name for the list of files to be deleted. Usually can be ignored.) |
equal_schemes | Whether to require URI scheme consistency when comparing file paths (for example, hdfs:// and s3a:// are considered different). |
equal_authorities | Whether to require URI authority consistency when comparing file paths (for example, namenode1:9000 and namenode2:9000 are considered different). |
prefix_mismatch_mode | Handling strategy when a file path prefix does not match the table location. Options include FAIL (error), DELETE (still delete), and SKIP (skip). |
prefix_listing | Whether to enable prefix-based object storage listing optimization (applicable to S3, OSS, and more) to improve scan efficiency. |
After enabling and configuring the orphan file governance feature, you can manually trigger optimization tasks on the orphan file governance page and view optimization records and results.
