Docs

File Storage vePFS

Data Flow

Best practices for configuring vePFS Data Flow directories

Best practices for configuring vePFS Data Flow directories

This topic provides typical examples of directory configuration for creating Data Flow tasks.

Warm-up tasks

Directory-based warm-up

Prerequisites:

The vePFS file system is mounted at /mnt/vepfs on client nodes.

Data Flow paths are configured as follows:

File System path: /dataset.

Object storage path: /bucket/prefix.

File paths before and after warm-up:

TOS source path before warm-up:

/bucket/prefix/a/b/c/file-001

vePFS destination path after warm-up:

/mnt/vepfs/dataset/a/b/c/file-001

File list-based warm-up

Prerequisites:

The vePFS file system is mounted at /mnt/vepfs on client nodes.

Data Flow paths are configured as follows:

File System path: /dataset.

Object storage path: /bucket/prefix.

File paths before and after warm-up:

TOS source path before warm-up:

/bucket/prefix/a/b/c/file-001
/bucket/prefix/a/b/c/file-002
/bucket/prefix/a/b/c/file-003

File list format for warm-up:

prefix/a/b/c/file-001
prefix/a/b/c/file-002
prefix/a/b/c/file-003

vePFS destination path after warm-up:

/mnt/vepfs/dataset/a/b/c/file-001
/mnt/vepfs/dataset/a/b/c/file-002
/mnt/vepfs/dataset/a/b/c/file-003

Settlement tasks

Directory-based settlement

Prerequisites:

The vePFS file system is mounted at /mnt/vepfs on client nodes.

Data Flow paths are configured as follows:

File System path: /dataset.

Object storage path: /bucket/prefix.

File paths before and after settlement:

vePFS source path before settlement:

/mnt/vepfs/dataset/a/b/c/file-001

TOS destination path after settlement:

/bucket/prefix/a/b/c/file-001

File list-based settlement

Prerequisites:

The vePFS file system is mounted at /mnt/vepfs on client nodes.

Data Flow paths are configured as follows:

File System path: /dataset.

Object storage path: /bucket/prefix.

File paths before and after settlement:

File list format for settlement:

/dataset/a/b/c/file-001
/dataset/a/b/c/file-002
/dataset/a/b/c/file-003

TOS destination path after settlement:

/bucket/prefix/a/b/c/file-001
/bucket/prefix/a/b/c/file-002
/bucket/prefix/a/b/c/file-003

Delete tasks

Deleting files only from file systems

Directory-based deletion

Prerequisites:

The vePFS file system is mounted at /mnt/vepfs on client nodes.

Data Flow paths are configured as follows:

File System path: /dataset.

Object storage path: /bucket/prefix.

Target deletion path in vePFS:

/mnt/vepfs/dataset/

Only vePFS files will be deleted. TOS files will remain intact.

File list-based deletion

Prerequisites:

The vePFS file system is mounted at /mnt/vepfs on client nodes.

Data Flow paths are configured as follows:

File System path: /dataset.

Object storage path: /bucket/prefix.

File list format for deletion:

/dataset/a/b/c/file-001
/dataset/a/b/c/file-002
/dataset/a/b/c/file-003

Only vePFS files will be deleted. TOS files will remain intact.

Deleting files from both file systems and TOS buckets

Directory-based deletion

Prerequisites:

The vePFS file system is mounted at /mnt/vepfs on client nodes.

Data Flow paths are configured as follows:

File System path: /dataset.

Object storage path: /bucket/prefix.

Target deletion path in vePFS:

/mnt/vepfs/dataset/a/b/c/file-001

Target deletion path in TOS:

/bucket/prefix/a/b/c/file-001

File list-based deletion

Prerequisites:

The vePFS file system is mounted at /mnt/vepfs on client nodes.

Data Flow paths are configured as follows:

File System path: /dataset.

Object storage path: /bucket/prefix.

File list format for deletion:

/dataset/a/b/c/file-001
/dataset/a/b/c/file-002
/dataset/a/b/c/file-003

Target deletion path in TOS:

/bucket/prefix/a/b/c/file-001
/bucket/prefix/a/b/c/file-002
/bucket/prefix/a/b/c/file-003

Last updated: 2026.03.03 10:21:01