Docs

Volcengine Kubernetes Engine

vePFS Client access methods

Using dynamic vePFS PVs

Volcengine Kubernetes Engine (VKE) allows you to use vePFS through the Container Storage Interface (CSI). This topic describes how to create vePFS PersistentVolumes (PVs) and PersistentVolumeClaims (PVCs) and how workloads use dynamic vePFS PVs.

tip

[Beta - apply for trial]: Dynamic vePFS PV provisioning and the Intelligent Computing edition are in the beta stage. Submit an application if you need to use these features.

Prerequisites

The csi-vepfs add-on has been installed. For details, see Installing add-ons.

A vePFS filesystem and mount service have been created, and the mount service has been bound to the filesystem. For details, see Creating a filesystem, Creating a mount service, and Binding filesystems.

(Optional) The kubectl tool has been installed locally. For detailed steps, see Installing and setting up kubectl.

Usage limits

When you use vePFS, take note of the following limitations or impacts related to deployment compatibility, add-on uninstallation, service mounting, and operational performance.

Restrictions on deployment compatibility

One vePFS mount service cannot be used for both the vePFS CSI solution and the solution where vePFS is pre-mounted to the host at the same time. For a vePFS file system instance, if you have used a VKE node to pre-mount vePFS and supplied it to pods through hostpath, you need to create an additional vePFS mount service and configure the new vePFS mount service for use by vePFS CSI.

The solution for using vePFS through CSI and the solution for using vePFS through the vePFS-Manager add-on cannot be used at the same time. We recommend using the CSI solution for new clusters. To apply the latest CSI solution to existing resources, submit a ticket to the technical support.

vePFS CSI controller will be preferentially scheduled to nodes that have the label vepfs.csi.volcengine.com/controller=preferred. To frequently scale the VKE cluster node pool, we recommend that you preferentially schedule the CSI controller to resident nodes, which are not affected by scaling. This ensures that vePFS CSI can quickly process member consistency between the VKE cluster and the vePFS cluster. Otherwise, when nodes running the vePFS CSI controller are scaled in and deleted, the vePFS CSI controller may be unable to maintain member consistency between the VKE cluster and the vePFS cluster, resulting in slow vePFS CSI operations.

Nodes that have already been added to the vePFS cluster through the Virtual Private Cloud (VPC) network will not be able to use the RDMA network. To switch to the RDMA network, remove the VKE node from the vePFS file system cluster.

Restrictions on add-on uninstallation

If the csi-vepfs add-on is uninstalled, running pods can continue to access vePFS storage until they finish running. However, if csi-vepfs has already been uninstalled, pod deletion will be blocked, and the pods can be deleted again after csi-vepfs is reinstalled. Therefore, we do not recommend that you uninstall the csi-vepfs add-on when pods are running.

csi-vepfs enables the file system client to run in a dedicated container, and the vePFS client will occupy the computing resources of the Kubernetes node where the container is located. By default, the client uses 4 CPUs and 4 GiB of memory. If node resources are insufficient, both the csi-vepfs client container and the pod will fail to start.

If you delete the vePFS mount service in advance, the vePFS CSI add-on cannot be uninstalled.

After you delete the vePFS CSI add-on, you cannot confirm whether some resources can be deleted, which may result in residual resources.

vePFS mount service secret: When you create a vePFS PV, VKE will also create a Secret that stores the addresses and login keys of the three management nodes of the vePFS mount service. You can run the following command to query the preceding Secret in the VKE cluster.

kubectl get -n kube-system secrets | grep "^vepfs-secret-mount-[0-9a-z]*"

Security groups on the VKE node: To ensure that the VKE node can communicate with the vePFS file system cluster, vePFS CSI will also add the security group of the management service ECS instance to the security group list of the VKE node. However, when CSI is being uninstalled, whether the security group is still being used by the user cannot be confirmed. Therefore, you cannot unbind the security group from the VKE node. You can view or delete the security group bound to the ECS node to which the vePFS mount service belongs in the ECS console.

Restrictions on service mounting

A single vePFS instance can be bound to multiple mount services. A single mount service can also be bound to multiple vePFS instances.

A single ECS instance can be added to only one vePFS mount service, which is equivalent to a client cluster. One mount service can be bound to multiple file system storage instances.

A node can be bound to only one mount service. Therefore, we recommend that you bind one VKE cluster to one vePFS mount service.

StorageClasses dynamically provision PVs by using filesets. Therefore, the number of PVs varies based on different types of filesets. For details, see Considerations on Fileset.

vePFS does not support the use of images of Ubuntu and veLinux CentOS Compatible Edition. For detailed client restrictions, see Client restrictions.

Restrictions on operational performance

The creation of dynamic PVs in a vePFS filesystem runs in serial mode. If multiple dynamic PVs are created simultaneously, the total creation time will be longer.

The initial startup of a container that uses vePFS on a node is approximately 15 seconds slower than subsequent startups. This is because vePFS CSI needs to compile the kernel driver for vePFS on the node. This process consumes more time. However, this situation only occurs when a container that uses a vePFS PV is created for the first time.

Creating through the console

Step 1: Create an SC

In the left navigation pane, click Cluster. In the right part of the page, click the target cluster name to open the management page of this cluster.

In the left navigation bar of the cluster management page, click Storage > PersistentVolume. Click Create PersistentVolume to configure a PV.

Parameter	Description
Name	Specify a name for the StorageClass. The name must be unique within the cluster.
StorageClass type	Select a storage type. In this scenario, select vePFS.
Availability zone	Availability Zones (AZs) are physically isolated infrastructure units within a single region, featuring independent power and networking systems. AZs communicate with each other over internal networks.
Billing type	Select a resource billing type as needed. Currently, pay-as-you-go is supported.
Instance type	Select the file system specifications for creating a PV. Performance metrics vary based on different specifications. For more information about product specifications, see Product specifications. tip [Beta - apply for trial]: The Intelligent Computing edition is currently in beta. To use it, submit an application.
Project	Assign the StorageClass to a project so that you can perform operations such as maintenance and authorization at the project level. By default, the default project is selected. For details, see Project management.
File system	Sequentially select the created vePFS file system, mount service, and security group, and then configure the advanced settings of the mount service based on your business requirements.
		Mount service: Select the mount service bound to the file system based on your business requirements. warning You can select a mount service only in the running state and in the same VPC as the cluster. The vePFS mount service will occupy 2 GiB of resources on the mount node. Ensure the proper planning of cluster resources.
		Advanced configurations: Configure advanced settings related to the mount service. You can view and specify the following parameters. Reserved memory: The reserved memory of the vePFS mount service occupies 2 GiB of resources on the mount node and cannot be modified. Access key: The default generated secret. This key stores the initial password for vePFS client node access. If the initial password has been changed, replace it with the corresponding node password. warning If the password is incorrect, vePFS will fail to be mounted. Make sure that the password is correct.
		Security group: Select the security group of the mount service client based on your business requirements. The security group will be automatically added to the main NIC of the user node through cross-service authorization. This adds the user node to the mount service cluster and connects the user cluster to the vePFS server.
Fileset	Configure the required fileset quotas based on your business requirements. This prevents a single fileset from occupying excessive resources, including file quota, bandwidth QoS, and IOPS QoS. tip The file quota cannot exceed the upper limit of iNode.
Reclaim policy	The reclaim policy. Available options include: Delete: When you delete the PVC, the data, the backend storage fileset of the PV, and the PV itself will be immediately deleted. Retain: When you delete the PVC, the data, the backend storage fileset of the PV, and the PV itself remain intact. You need to manually delete them. danger The deleted data cannot be restored. Proceed with prudence when selecting the Delete policy. We recommend that you select Retain to prevent accidental data deletions.
PersistentVolume binding mode	The binding mode. The following information is included: Immediate: Performs PV binding and allocation when the PVC is created. WaitForFirstConsumer: Delays PV binding and allocation until a pod that uses the PVC is created.

Click Confirm.

Step 2: Create a PVC

In the left navigation bar of the cluster management page, click Storage > PersistentVolumeClaim. Click Create PersistentVolumeClaim to configure a PVC.

Parameter	Description
Namespace	Select the namespace to which the PVC belongs. We recommend that you select a custom namespace or the default namespace created by the system. For details about namespaces, see Namespace overview.
Name	Specify a name for the PVC. The name must be unique within the cluster.
StorageClass type	Select a storage type. In this scenario, select vePFS.
Access mode	Select a PV access mode. You can select ReadWriteMany, which enables read and write access by multiple nodes.
Billing type	Select a resource billing type as needed. Currently, only Pay-as-you-go is supported.
How to create	The method of creating a PVC. Available methods include Create based on StorageClass and Use an existing PersistentVolume. For this example, select Create based on StorageClass.
StorageClass	Select an appropriate StorageClass based on your business requirements. When you create an application, it will be automatically bound to a PV that uses the same StorageClass and has a capacity no less than that specified in the PVC.
Capacity	The declared storage usage.

Click Confirm.

Step 3: Create an application and mount the PV

After the dynamic PV of the vePFS type is created, add the PV to a workload that will use the PV. Supported workloads include Deployment, StatefulSet, Job, CronJob, and DaemonSet.

You can add a PV to a workload when creating it or updating it. For details, see Workloads.

warning

When a workload is being created, a dedicated container for mounting the vePFS client is started first on the node. The vePFS client will occupy the computing resources of the Kubernetes node where the container is located. By default, the client uses 4 CPUs and 4 GiB of memory. If node resources are insufficient, both the csi-vepfs client container and the pod will fail to start.

Creating through kubectl

If you connect to the cluster, you can use a dynamic vePFS PV by using kubectl.

Step 1: Configure connection authentication

Run the following commands in sequence to store the SSH connection authentication information of the vePFS management node in a Secret.

tip

The addresses, passwords, and hostnames in the following example are for demonstration purposes only. Replace them with actual configurations based on your business requirements.

tip

If you have created a PV of type vePFS via the console, VKE will synchronously create a Secret that stores the addresses and login credentials of the management nodes for the mount service. You can specify a custom Secret name by using the adminHostsSecretName parameter, or check whether the Secret exists in the cluster by running the kubectl get -n kube-system secrets | grep "^vepfs-secret-mount-[0-9a-z]*" command.

Step 2: Create a StorageClass

Create a YAML file for the SC. The following vke-pfs-sc.yaml file serves as an example:

apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
name: vke-pfs-sc # StorageClass name.
provisioner: vepfs.csi.volcengine.com # # The driver that the storage mount depends on.
parameters:
fsid: vepfs-cnsh672*** # Filesystem ID.
mountServiceID: mount-6de0*** # Mount service ID.
fileLimit: "100000" # Fileset file quota.
maxIOPS: "20" # Fileset IOPS QoS limit.
maxBandwidth: "30" # Fileset bandwidth QoS limit measured in MB/s.
mountpoint: /vepfs-cnsh672*** # Mount point specified for binding the mount service to the filesystem.
securityGroupID: sg-3679xag6w*** # Security group for mount point clients.
adminHostsSecretName: pfs-secret # Secret name.
adminHostsSecretNamespace: kube-system # Namespace where the Secret resides.
reclaimPolicy: Delete # PV reclaim policy. Value: Retain or Delete.
volumeBindingMode: Immediate # Volume binding mode. Values include Immediate (indicating immediate binding and allocation) and WaitForFirstConsumer (indicating delayed binding and allocation).

The key parameters are described in the following table.

Parameter	Description
provisioner	The driver for mounting a PV. In this example, you need to set the value to vepfs.csi.volcengine.com, which indicates the csi-vepfs add-on.
fsid	The ID of the filesystem. tip You can view the ID of the created filesystem in the console. For details, see Viewing filesystem details.
mountServiceID	The ID of the mount service.
fileLimit	Configure the fileset file quota. For details, see Configuring Fileset quota and QoS settings.
maxIOPS	Configure a fileset IOPS QoS limit. For details, see Configuring Fileset quota and QoS settings.
maxBandwidth	Configure a fileset bandwidth QoS limit (measured in MB/s). For details, see Configuring Fileset quota and QoS settings.
mountpoint	The mount point specified for binding the mount service to the filesystem. For details, see Binding a filesystem.
securityGroupID	The security group for mount point clients.
adminHostsSecretName	The name of the Secret that stores authentication information. The authentication information in the Secret must be consistent with that of the mount service.
adminHostsSecretNamespace	The namespace of the Secret for authentication. The authentication information in the Secret must be consistent with that of the mount service.
reclaimPolicy	The PV reclaim policy. Valid values: Delete: When you delete the PVC, the PV and the data in the PV will be simultaneously deleted. Retain: When you delete the PVC, the PV and the data in the PV will not be simultaneously deleted. You need to manually delete them. warning We recommend that you select Retain to prevent accidental data deletions. The deleted data cannot be restored. Proceed with prudence when selecting the Delete policy.
volumeBindingMode	The binding mode of the PV. Values: Immediate: Performs PV binding and allocation when the PVC is created. WaitForFirstConsumer: Delays PV binding and allocation until a pod that uses the PVC is created.

Run the following command to create the StorageClass.

kubectl create -f vke-pfs-sc.yaml

Step 3: Create a PVC

Create a YAML file for the PVC. The following vke-pfs-pvc.yaml file serves as an example:

apiVersion: v1
kind: PersistentVolumeClaim
metadata:
name: csi-pfs-pvc # PVC name.
namespace: default # Namespace to which the PVC belongs.
spec:
accessModes:
- ReadWriteMany # PVC access mode. Currently, only ReadWriteMany is supported. ReadWriteMany allows read and write access from multiple nodes.
resources:
requests:
storage: 20Gi # Declared storage usage.
storageClassName: vke-pfs-sc # StorageClass name.

The key parameters are described in the following table.

Parameter	Description
accessModes	The PVC access mode. You can only select ReadWriteMany, which enables read and write access by multiple nodes.
storage	The declared storage capacity. Unit: Gi. It must be consistent with the capacity of the created PV.
storageClassName	The name of the PV, which must be consistent with the name configured for the PV.

Run the following command to create the PVC.

kubectl create -f vke-pfs-pvc.yaml

Step 4: Create an application and mount the PV

Create a YAML file for the Deployment. The following deployment-nginx-dynamic.yaml file serves as an example:

apiVersion: apps/v1
kind: Deployment
metadata:
name: deployment-nginx-dynamic
spec:
selector:
matchLabels:
app: nginx
template:
metadata:
labels:
app: nginx
spec:
containers:
- name: nginx
image: doc-cn-beijing.cr.volces.com/vke/nginx-demo:v1.0 # Specify the image file to be deployed.
ports:
- containerPort: 80
name: nginx
volumeMounts:
- name: pfs
mountPath: /tmp # Mount location of the PV in the container.
volumes:
- name: pfs
persistentVolumeClaim:
claimName: csi-pfs-pvc # The name of the created PVC.

Run the following command to create the Deployment.

kubectl apply -f deployment-nginx-dynamic.yaml

warning

Validating results

After creating a workload and mounting a PV to it as described in the preceding steps, you can verify the volume mounting status and persistent storage via the following methods.

Run the following command to check whether the PV dynamically provisioned based on the PVC has been created and bound to the PVC.

kubectl get pvc

The expected result is as follows:

NAME STATUS VOLUME CAPACITY ACCESS MODES STORAGECLASS
csi-pfs-pvc Pending pvc-6e8c646b-e3e9-4d02-836e-3b66******** 20Gi RWX vke-pfs-sc

Run the following command to check whether the PV is mounted to the specified path in the pod.

kubectl exec vke-efs-static-deployment-xxx -- df | grep tmp

The expected response is as follows.

fs_vepfs-cnsh67*** 6442450944 27478016 6414972928 1% /tmp

Check the mount path in the pod to view the data in the PV.

kubectl exec vke-nas-static-deployment-xxx -- ls /tmp

The expected response is as follows.

lost+found

Write a file to the PV.

kubectl exec vke-nas-static-deployment-xxx -- touch /tmp/test

Delete the pod.

kubectl delete pod vke-nas-static-deployment-xxx

Check the mount path of the PV in the newly created pod.

kubectl exec vke-nas-static-deployment-xxx -- ls /tmp

The expected result is as follows. You can see that the file written to the PV still exists and is not deleted along with the pod. This indicates that persistent storage of data is achieved.

lost+found
test