Docs

Volcengine Kubernetes Engine

vePFS Client access methods

Using static vePFS PVs

VKE supports the use of vePFS through Container Storage Interface (CSI). This topic describes how to create a PersistentVolume (PV) and PersistentVolumeClaim (PVC) based on vePFS and how workloads use statically provisioned vePFS PVs.

tip

[Beta - apply for trial]: The static vePFS PV and the Intelligent Computing edition are in the beta stage. Submit an application if you need to use these features.

Prerequisites

The csi-vepfs add-on has been installed. For details, see Installing add-ons.

A vePFS filesystem and mount service have been created, and the mount service has been bound to the filesystem. For details, see Creating a filesystem, Creating a mount service, and Binding filesystems.

(Optional) The kubectl tool has been installed locally. For details, see Installing and setting up kubectl.

Usage limits

When you use vePFS, take note of the following limitations or impacts related to deployment compatibility, add-on uninstallation, service mounting, and operational performance.

Restrictions on deployment compatibility

One vePFS mount service cannot be used for both the vePFS CSI solution and the solution where vePFS is pre-mounted to the host at the same time. For a vePFS file system instance, if you have used a VKE node to pre-mount vePFS and supplied it to pods through hostpath, you need to create an additional vePFS mount service and configure the new vePFS mount service for use by vePFS CSI.

The solution for using vePFS through CSI and the solution for using vePFS through the vePFS-Manager add-on cannot be used at the same time. We recommend using the CSI solution for new clusters. To apply the latest CSI solution to existing resources, submit a ticket to the technical support.

vePFS CSI controller will be preferentially scheduled to nodes that have the label vepfs.csi.volcengine.com/controller=preferred. If you plan to frequently scale the VKE cluster node pool, we recommend preferentially scheduling the CSI controller to resident nodes (not affected by scaling). This ensures vePFS CSI can process member consistency between the VKE cluster and the vePFS cluster more quickly. Otherwise, when nodes running the vePFS CSI controller are scaled in and deleted, the vePFS CSI controller may be unable to maintain member consistency between the VKE cluster and the vePFS cluster, resulting in slow vePFS CSI operations.

Nodes that have already been added to the vePFS cluster through the Virtual Private Cloud (VPC) network will not be able to use the RDMA network. To switch to the RDMA network, remove the VKE node from the vePFS file system cluster.

Restrictions on add-on uninstallation

If the csi-vepfs add-on is uninstalled, running pods can continue to access vePFS storage until they finish running. However, if csi-vepfs has already been uninstalled, pod deletion is blocked, and the pods can then be successfully deleted after csi-vepfs is reinstalled. Therefore, we do not recommend that you uninstall the csi-vepfs add-on when pods are running.

csi-vepfs enables the file system client to run in a dedicated container, and the vePFS client will occupy the computing resources of the Kubernetes node where the container is located. By default, the client uses 4 CPUs and 4 GiB of memory. If node resources are insufficient, both the csi-vepfs client container and the pod will fail to start.

If you delete the vePFS mount service in advance, the vePFS CSI add-on cannot be uninstalled.

After you delete the vePFS CSI add-on, you cannot confirm whether some resources can be deleted, which may result in residual resources.

vePFS mount service secret: When you create a vePFS PV, VKE will also create a Secret that stores the addresses and login keys of the three management nodes of the vePFS mount service. You can run the following command to query the preceding Secret in the VKE cluster.

kubectl get -n kube-system secrets | grep "^vepfs-secret-mount-[0-9a-z]*"

Security groups on the VKE node: To ensure that the VKE node can communicate with the vePFS file system cluster, vePFS CSI will also add the security group of the management service ECS instance to the security group list of the VKE node. However, when CSI is being uninstalled, whether the security group is still being used by the user cannot be confirmed. Therefore, you cannot unbind the security group from the VKE node. You can view or delete the security group bound to the ECS node to which the vePFS mount service belongs in the ECS console.

Restrictions on service mounting

A single vePFS instance can be bound to multiple mount services. A single mount service can also be bound to multiple vePFS instances.

A single ECS instance can be added to only one vePFS mount service, which is equivalent to a client cluster. One mount service can be bound to multiple file system storage instances.

A node can be bound to only one mount service. We recommend binding one vePFS mount service to a VKE cluster.

vePFS does not support the use of images of Ubuntu and veLinux CentOS Compatible Edition. For detailed client restrictions, see Client restrictions.

Restrictions on operational performance

The initial startup of a container that uses vePFS on a node is approximately 15 seconds slower than subsequent startups. This is because vePFS CSI needs to compile the kernel driver for vePFS on the node. This process consumes more time. However, this situation only occurs when a container that uses a vePFS PV is created for the first time.

Creating through the console

Step 1: Create a PV

In the left navigation pane, click Cluster. In the right part of the page, click the target cluster name to open the management page of this cluster.

In the left navigation bar of the cluster management page, click Storage > PersistentVolume. Click Create PersistentVolume to configure a PV.

Parameter	Description
How to create	Select a method for creating the PV. Currently, you can select Static creation.
Name	Specify a name for the PV. The name must be unique within the cluster.
StorageClass type	Select a storage type. In this scenario, select vePFS.
Access type	Select an access type for the PV. In this scenario, select vePFS Client.
Access mode	Select a PV access mode. You can select ReadWriteMany, which enables read and write access by multiple nodes.
Instance type	Select the file system specifications for creating a PV. Performance metrics vary based on different specifications. For more information about product specifications, see Product specifications. tip [Beta - apply for trial]: The Intelligent Computing edition is currently in beta. To use it, submit an application.
Project	Assign the StorageClass to a project so that you can perform operations such as maintenance and authorization at the project level. By default, the default project is selected. For details, see Project management.
File system	Sequentially select the created vePFS file system, mount service, and security group, and then configure the advanced settings of the mount service based on your business requirements.
		Mount service: Select the mount service bound to the file system based on your business requirements. warning Only mount services in the running state are supported, which need to be in the same VPC as the cluster. The vePFS mount service will occupy 2 GiB of resources on the mount node. Ensure the proper planning of cluster resources.
		Advanced configurations: Configure advanced settings related to the mount service. You can view and specify the following parameters. Reserved memory: The reserved memory of the vePFS mount service occupies 2 GiB of resources on the mount node and cannot be modified. Access key: The default generated secret. This key stores the initial password for vePFS client node access. If the initial password has been changed, replace it with the corresponding node password. warning If the password is incorrect, vePFS will fail to be mounted. Make sure that the password is correct.
Security group	Select the security group of the mount service client based on your business requirements. The security group will be automatically added to the main NIC of the user node through cross-service authorization. This adds the user node to the mount service cluster and connects the user cluster to the vePFS server.

Click Confirm.

Step 2: Create a PVC

In the left navigation bar of the cluster management page, click Storage > PersistentVolumeClaim. Click Create PersistentVolumeClaim to configure a PVC.

Parameter	Description
Namespace	Select the namespace to which the PVC belongs. We recommend that you select a custom namespace or the default namespace created by the system. For details about namespaces, see Namespace overview.
Name	Specify a name for the PVC. The name must be unique within the cluster.
StorageClass type	Select a storage type. In this scenario, select vePFS.
Access mode	Select a PV access mode. You can select ReadWriteMany, which enables read and write access by multiple nodes.
Billing type	Select a resource billing type as needed. Currently, only Pay-as-you-go is supported.
How to create	The method of creating a PVC. Available methods include Create based on StorageClass and Use an existing PersistentVolume. For this example, select Use an existing PersistentVolume.
PersistentVolume	Specify the vePFS PV.

Click Confirm.

Step 3: Create an application and mount the PV

After the static PV of the vePFS type is created, add the PV to a workload that will use the PV. Supported workloads include Deployment, StatefulSet, Job, CronJob, and DaemonSet.

You can add a PV to a workload when creating it or updating it. For details, see Workloads.

Creating through kubectl

If you connect to the cluster, you can use a static vePFS PV by using kubectl.

Step 1: Configure connection authentication

Run the following commands in sequence to store the SSH connection authentication information of the vePFS management node in a Secret.

tip

The addresses, passwords, and hostnames in the following example are for demonstration purposes only. Replace them with actual configurations based on your business requirements.

Step 2: Create a PV

Create a YAML document for the PV. The following vke-pfs-static-pv.yaml file serves as an example: You can mount the entire file system or a file system subdirectory based on your business requirements.

Mounting the entire file system
Mounting a subdirectory of the filesystem

The following YAML example demonstrates how to mount the entire vePFS filesystem:

apiVersion: v1
kind: PersistentVolume
metadata:
name: pfs-static-pv # PV name. The name must be unique within the cluster.
namespace: default
annotations:
pv.kubernetes.io/provisioned-by: vepfs.csi.volcengine.com
spec:
accessModes:
- ReadWriteMany # PV access mode. The only supported mode is ReadWriteMany, which allows read and write access by multiple nodes.
capacity:
storage: 20Gi # Declared PV capacity. This field is required by Kubernetes only for validation. It does not impose an actual capacity limit.
csi:
driver: vepfs.csi.volcengine.com # Driver required for storage mount.
readOnly: false
nodeStageSecretRef: # The name of the Secret that stores authentication information. The authentication information in the Secret must be consistent with that of the mount service.
name: pfs-secret # Secret name.
namespace: kube-system # The namespace of the Secret.
controllerPublishSecretRef:
name: pfs-secret
namespace: kube-system
volumeAttributes:
fsid: vepfs-cnsh67*** # Filesystem ID.
hpcClusterID: vepfs-cluster-id-*** # The ID of the HPC cluster. Binding to an HPC cluster is required only when the Intelligent Computing edition is used.
mountServiceID: mount-6de0*** # Mount service ID.
securityGroupID: sg-3679xag6w*** # Security group for mount point clients.
mountpoint: /vepfs-cnsh67*** # Mount point specified for binding the mount service to the filesystem.
volumeHandle: vepfs-cnsh67*** # The unique identifier of the PV. Keep it consistent with fsid.
persistentVolumeReclaimPolicy: Retain # PV reclaim policy. Value: Retain or Delete.
volumeMode: Filesystem # PV type. Value: Filesystem (using the volume as a filesystem. It will be mounted by the pod to a directory) or Block (using the volume as a block device).
The YAML example for mounting a vePFS file system subdirectory is as follows:
warning
You can mount vePFS file system subdirectories only when you statically create a vePFS PV by using YAML.
When you create a PV, you need to use a created vePFS file system subdirectory. Otherwise, volume cannot be mounted during pod creation.

apiVersion: v1
kind: PersistentVolume
metadata:
name: pfs-static-pv # PV name. The name must be unique within the cluster.
namespace: default
annotations:
pv.kubernetes.io/provisioned-by: vepfs.csi.volcengine.com
spec:
volumeMode: Filesystem
accessModes: # PV access mode. The only supported mode is ReadWriteMany, which allows read and write access by multiple nodes.
- ReadWriteMany
capacity:
storage: 20Gi # Declared PV capacity. This field is required by Kubernetes only for validation. It does not impose an actual capacity limit.
persistentVolumeReclaimPolicy: Retain # PV reclaim policy. Value: Retain or Delete.
csi:
driver: vepfs.csi.volcengine.com # The driver required for volume mounting. In this example, set the value to vepfs.csi.volcengine.com, which indicates the csi-vepfs add-on.
readOnly: false
nodeStageSecretRef: # The Secret corresponding to the vePFS mount service.
name: pfs-secret
namespace: kube-system
controllerPublishSecretRef:
name: pfs-secret
namespace: kube-system
volumeAttributes:
fsid: vepfs-cnbjexxxxac6e7b3 # Filesystem ID.
hpcClusterID: vepfs-cluster-id-xxx # The ID of the HPC cluster. You need to bind an HPC cluster only when fsid uses the Intelligent Computing edition.
mountServiceID: mount-ff0b**** # Mount service.
securityGroupID: msadgh-14332 # Security group for vePFS mount point clients.
mountpoint: /vepfs-cnbjexxxxac6e7b3 # The mount point, which corresponds to the ID of the filesystem.
subpath: /l1.dir1/l2.dir1 # The subdirectory of the filesystem for mounting. You need to use a created vePFS filesystem subdirectory.
volumeHandle: vepfs-cnbjexxxxac6e7b3 # Filesystem ID.

The key parameters are described in the following table.

Parameter	Description
accessModes	The PV access mode. Only ReadWriteMany is supported, which allows the PV to be read and written by multiple pods.
storage	Declared PV capacity. Unit: Gi. This field is required by Kubernetes. The value has no actual meaning and is only for validation.
driver	The driver for mounting a PV. In this example, you need to set the value to vepfs.csi.volcengine.com, which indicates the csi-vepfs add-on.
nodeStageSecretRef	The name and namespace of the Secret for authentication. The authentication information in the Secret must be consistent with that of the mount service.
controllerPublishSecretRef	The name and namespace of the Secret for authentication. The authentication information in the Secret must be consistent with that of the mount service.
fsid	The ID of the filesystem. tip You can view the ID of the created filesystem in the console. For details, see Viewing filesystem details.
hpcClusterID	ID of the HPC cluster. Binding to an HPC cluster is required only when the Intelligent Computing edition is used.
mountServiceID	The ID of the mount service.
securityGroupID	The security group for mount point clients.
mountpoint	The mount point specified for binding the mount service to the filesystem. For details, see Binding a filesystem.
volumeHandle	The unique identifier of the PV. Keep it consistent with the name of the PV.
persistentVolumeReclaimPolicy	The PV reclaim policy. The following information is included: Retain: The Retain policy. When the PVC bound to the PV is deleted, the PV is retained. Delete: The Delete policy. When the PVC bound to the PV is deleted, the PV is deleted synchronously.
volumeMode	The PV type. The following information is included: Filesystem: (Default) Use the PV as a file system. The PV will be formatted and mounted to a file system (such as ext4 and xfs), and the file system will be mounted to a specified path in the container. You can use the ls command (such as ls /data) in the container to view the mounted directories or files, and directly read and write files. Block: Use the PV as a block device. It is directly exposed to the container without being formatted or mounted to a file system. You can perform operations on the block device by using paths such as /dev/block/xxx, and applications must support read and write operations on the raw block device.

Run the following command to create the PV.

kubectl create -f vke-vepfs-static-pv.yaml

Step 3: Create a PVC

Create a YAML file for the PVC. The following vke-pfs-static-pv.yaml file serves as an example:

apiVersion: v1
kind: PersistentVolumeClaim
metadata:
name: pfs-static-pvc # PVC name.
namespace: default # Namespace to which the PVC belongs.
spec:
accessModes:
- ReadWriteMany # PVC access mode. Currently, only ReadWriteMany is supported. ReadWriteMany allows read and write access from multiple nodes.
resources:
requests:
storage: 20Gi # Declared PV capacity. This field is required by Kubernetes only for validation. It does not impose an actual capacity limit.
volumeMode: Filesystem # The PV type. Valid values: Filesystem (using the PV as a filesystem) and Block (using the PV as a block device). The value must be consistent with the type configured in the PV.
volumeName: pfs-static-pv # Name of the created PV.

The key parameters are described in the following table.

Parameter	Description
accessModes	The PVC access mode. You can only select ReadWriteMany, which enables read and write access by multiple nodes.
storage	The declared storage capacity. Unit: Gi. It must be consistent with the capacity of the created PV.
volumeMode	The PV type. It must be consistent with the type configured for the PV. The following information is included: Filesystem: (Default) Use the PV as a file system. The PV will be formatted and mounted to a file system (such as ext4 and xfs), and the file system will be mounted to a specified path in the container. You can use the ls command (such as ls /data) in the container to view the mounted directories or files, and directly read and write files. Block: Use the volume as a block device. It is directly exposed to the container without being formatted or mounted to a file system. You can perform operations on the block device by using paths such as /dev/block/xxx, and applications must support read and write operations on the raw block device.
volumeName	The name of the PV, which must be consistent with the name configured for the PV.

Run the following command to create the PVC.

kubectl create -f vke-pfs-static-pvc.yaml

Run the following command to view the result of the static association between the PVC and PV.

kubectl get pvc

If the response is similar to the following one, the binding is successful.

NAME STATUS VOLUME CAPACITY ACCESS MODES STORAGECLASS AGE
vke-pfs-static-pvc Bound vke-pfs-static-pv 10Pi RWX 2m36s

Step 4: Create an application and mount the PV

Create a YAML file for the Deployment. The following deployment-nginx-static.yaml file serves as an example:

apiVersion: apps/v1
kind: Deployment
metadata:
name: deployment-nginx-static
spec:
selector:
matchLabels:
app: nginx
template:
metadata:
labels:
app: nginx
spec:
containers:
- name: nginx
image: doc-cn-beijing.cr.volces.com/vke/nginx-demo:v1.0 # Specify the image file to be deployed.
ports:
- containerPort: 80
name: nginx
volumeMounts:
- name: pfs
mountPath: /tmp # Mount location of the PV in the container.
volumes:
- name: pfs
persistentVolumeClaim:
claimName: pfs-static-pvc # The name of the created PVC.

Run the following command to create the Deployment.

kubectl apply -f deployment-nginx-static.yaml

warning

When a workload is being created, a dedicated container for mounting the vePFS client is started first on the node. The vePFS client will occupy the computing resources of the Kubernetes node where the container is located. By default, the client uses 4 CPUs and 4 GiB of memory. If node resources are insufficient, both the csi-vepfs client container and the pod will fail to start.

Validating results

After creating a workload and mounting a PV to it as described in the preceding steps, you can verify the volume mounting status and persistent storage via the following methods.

Run the following command to check whether the PV is mounted to the specified path in the pod.

kubectl exec vke-efs-static-deployment-xxx -- df | grep tmp

The expected response is as follows.

fs_vepfs-cnsh67*** 6442450944 27478016 6414972928 1% /tmp

Check the mount path in the pod to view the data in the PV.

kubectl exec vke-nas-static-deployment-xxx -- ls /tmp

The expected response is as follows.

lost+found

Write a file to the PV.

kubectl exec vke-nas-static-deployment-xxx -- touch /tmp/test

Delete the pod.

kubectl delete pod vke-nas-static-deployment-xxx

Check the mount path of the PV in the newly created pod.

kubectl exec vke-nas-static-deployment-xxx -- ls /tmp

The expected result is as follows. You can see that the file written to the PV still exists and is not deleted along with the pod. This indicates that persistent storage of data is achieved.

lost+found
test