Feature overview #

Target users #

This feature is primarily intended for users and operations teams who need to efficiently manage massive, fine-grained Lance datasets stored in vePFS.

Feature description #

As business grows, the amount of data accumulated in vePFS increases rapidly. Traditional manual data inspection, statistics, and cleanup methods are not only inefficient, but also carry a high risk of accidental deletion when dealing with massive, fine-grained directories. The LAS lifecycle management module is designed to address this challenge by providing an automated and refined data lifecycle management solution. It helps users establish intelligent lifecycle management mechanisms to replace cumbersome and high-risk manual operations, ensuring effective data management throughout the entire lifecycle and significantly reducing data security risks caused by manual errors.

Core capabilities #

• Automated rule execution: Users can flexibly configure replacement rules according to business requirements. The system will automatically perform cleanup operations on data in vePFS storage based on these rules.
• Intelligent replacement triggering: The system can intelligently determine and trigger replacement tasks based on storage utilization (watermark) and data access policies (such as last access time and access frequency), ensuring efficient use of storage space.
• Granular task tracking: Provides detailed task execution history, status tracking, and operation logs to facilitate user auditing and troubleshooting.

Prerequisite operations and configuration #

Before using the lifecycle management feature, please be sure to complete the following prerequisite configurations; otherwise, the feature will not function properly.

Mount service confirmation #

Please ensure that your vePFS file system meets the following conditions:

• Associated mount service: The target vePFS file system must already be associated with a mount service.
• Client management node configuration:
  • The client management node must use the default password .
  • The client management node must have installed or support the installation of Cloud Assistant (ECS Assistant).

Lance SDK update #

The lifecycle management feature relies on the data access tracking capability of the Lance SDK. Please ensure that the Lance SDK you are using has been updated to version 0.32 or later. The trace feature in the new SDK version is the foundation for reporting data access information.

Environment variable configuration #

To enable the Lance SDK to communicate properly with the LAS backend service, you need to configure the following environment variables in the environment where you run training or data processing tasks.

Key features and operations #

Create and manage eviction rules #

Eviction rules are the core of the lifecycle management feature, defining when (When), what (What), and how (How) the system performs data cleanup.

Create eviction rules

1. Go to the LAS console and select "Lifecycle management" in the left navigation bar.
2. Click the "Create rules" button to enter the rule configuration page.
   

Parameter details:

• The rule name (RuleName): Specify an easily identifiable name for your rule. This is a required field.
• Rule description (Description): Provide a detailed description of the rule to facilitate future management. This is optional.
• Storage media (StorageMedium): Specify the storage type for the eviction target. Currently, only vePFS is supported.
• vePFS path (VepfsPaths):
  • File system ID (FileSystemId): Select the vePFS file system instance to which this rule will be applied. This is a required field.
  • Directory path (Path): Specify the directory to be cleaned (for example, /data/images/). This is a required field.
• Storage water level settings (Watermark):
  • High watermark (HighWatermark): When the file system storage usage reaches this threshold, the eviction task is triggered automatically. For example, set to 80%. This is a required field.
  • Low watermark (LowWatermark): During the execution of the eviction task, when storage usage drops to this threshold, the task will stop automatically. If not set, all eligible data will be cleaned up. This is optional.
• Active policies (Policy): Define the conditions that the data set to be evicted must meet.
  • Logical relationship (Operator):
    • AND: The data set will be evicted only if all conditions are met.
    • OR: The data set will be evicted if any one condition is met.
  • Conditions:
    • Last access time (LastAccessInDay): Specify data sets that were accessed N days ago.
    • Access frequency (AccessFrequency): Specifies datasets with a total access count less than N.
• Dirty data configurations (SkipDirtyData):
  • Yes: If an error occurs during the recycling process (such as file permission issues), skip the error and continue execution.
  • No: Stop the task immediately upon encountering any error.

View and filter rules

On the lifecycle management rule list page, you can view all created rules.

• Search and filter: Supports fuzzy search and filtering by rule name, ID, and status.
• Sort: Supports sorting rules by update time.

Update recycling rules

On the rule list page, locate the target rule and click the "Edit" button to enter the update interface. You can modify all fields except "Storage medium."

Delete recycling rules

On the rule list page, locate the target rule and click the "Delete" button. The system will display a confirmation prompt. Please proceed with caution; deleted rules cannot be recovered.

Manual scan

The "Manual scans" feature is used to immediately trigger a check of the file system storage usage level. On the rule list page, locate the target rule and click "Manual scans" to trigger.

View recycling tasks and logs #

Task list

On the rule list page, click any rule to view all historical recycling tasks under that rule.

The list displays each task's ID, status (Running/Failed/Succeeded), start time, and end time.

Task details and logs

Click any task ID in the task list to enter the task details page.

The details page includes the following information:

• Task basic information: Includes task ID, status, start and end time, and other information.
• Recycled datasets (RecycledDatasets): Lists the paths, sizes, and other information of all datasets successfully cleaned in this task.
• Execution log (Log): Provides the complete execution log for the task, which can be used for troubleshooting and auditing.

Permission description #

The lifecycle management module is not fully open at present. To use it, please contact allowlist support.