This topic describes how to use the PV mounting capability in the VCI Agent Sandbox scenario to implement dynamic mounting of PVs for Sandbox in warm pool instances. This meets the diverse storage requirements of different businesses and tenants.

Feature description

The dynamic mounting mode does not require predefined PVs in the template. SandboxClaim specifies the container mount directory (mountPath) when claiming the sandbox instance and supports adding new PVs and volumeMount. VCI injects the CSI mounting add-on in the form of Hidercar (shadow container) to ensure security. This applies to the scenarios where the mount directory can only be determined when the sandbox instance is running, or where an existing annotation contains mountPath and low-cost migration is required. Currently, only TOS-type PVs are supported.

The dynamic mounting capability is enabled and configured by adding annotations to SandboxTemplate and SandboxClaim. The following table describes the core annotation.

Object	Annotation key	Annotation sample value	Description
SandboxTemplate (podTemplate)	vke.volcengine.com/csi-sidecar-inject	false	Whether to enable the dynamic mounting capability. Available values: true: Enable dynamic mounting and inject Hidercar. false: (Default) Disabled.
SandboxClaim	vke.volcengine.com/dynamic-volume-mount-config	[{"volumeName":"data-vol","pvName":"pv-tos-shared","mountPath":"/mnt/data","subPath":"/tenant-a"}]	Declare the volumeMount information to be added for each pod. Value format: JSON array. The supported fields in the array are described as follows: volumeName: The target volume name. If left empty, the system automatically uses pvName as volumeName. pvName: The name of the PV corresponding to the volume. mountPath: The mount directory inside the container. It can be any absolute path inside the container. Including this field means using the dynamic mounting mode. subPath: The subpath under the PV root directory. It is used for multi-tenant directory isolation. It must be a valid subpath relative to the PV root directory and must not contain ...

Limits

When using the dynamic mounting capability, please note the following restrictions.

Limitation item	Description
Strong dependency on template annotations	SandboxTemplates must have the vke.volcengine.com/csi-sidecar-inject: "true" annotation. Otherwise, all requests with mountPath will be rejected.
Restrictions on a single primary container	The dynamic mounting mode processes a single business primary container by default and preferentially selects the first container whose name does not contain the sidecar keyword. Multi-container scenario (not recommended). If dynamic mounting needs to be used, the name of the non-primary container must contain the sidecar keyword. Otherwise, the first container will be selected by default.
In-place restart is not supported	The dynamic mounting mode cannot be used simultaneously with container in-place update and writable layer persistence. The mount point may be lost after the container restarts, and the sandbox needs to be recreated.
Claiming restrictions	A sandbox in the warm pool will only be claimed once. If dynamic mounting fails, times out, or is abnormal, the sandbox instance will be cleaned up directly, and a cold start will be applied. The instance will not be returned to the warm pool for reuse, preventing data from entering an intermediate state.
Timeout limit	The timeout for the dynamic mounting operation is 30 seconds. After the timeout, the claim fails, and the sandbox instance is cleaned up.
Storage type limit	The dynamic mounting mode only supports TOS storage.
subPath security validation	The path traversal symbols .. are not allowed in subPath. A subPath must be a valid subdirectory relative to the PV root directory.
Security responsibilities and risks	For security responsibility and risk descriptions related to the dynamic mounting capability, see PV mounting overview.

Prerequisites

A cluster required by the VCI service has been created. For details, see Using VCI in VPC-CNI clusters and Using VCI in Flannel clusters.

The VCI Agent Sandbox component, agent-sandbox, has been installed in the VCI business cluster to support CRDs such as SandboxTemplate, SandboxWarmPool, and SandboxClaim. For detailed operations, see Installation guide.

A TOS storage bucket (Bucket) has been created. For details, see Creating buckets.

The kubectl tool has already been installed locally, and the target cluster has been connected by using kubectl. For detailed description, see Connecting to clusters using kubectl.

Procedure

Step 1: Create a SandboxTemplate

Create a SandboxTemplate and declare vke.volcengine.com/csi-sidecar-inject: "true" in the annotations of podTemplate to enable dynamic mounting mode. The example SandboxTemplate YAML file dynamic-mount-template.yaml is shown below.

apiVersion: extensions.agents.x-k8s.io/v1alpha1
kind: SandboxTemplate
metadata:
name: dynamic-mount-template # SandboxTemplate name.
namespace: default # The namespace of the SandboxTemplate.
spec:
podTemplate:
metadata:
annotations:
vke.volcengine.com/csi-sidecar-inject: "true" # Whether to enable dynamic mounting (Hidercar injection). Values: true (enabled) and false (disabled).
vke.volcengine.com/burst-to-vci: enforce # Forced scheduling to VCI.
spec:
containers:
- name: app
image: doc-cn-beijing.cr.volces.com/vke/nginx-demo:v1.0 # The address and version of the container image. Replace the value with your actual information. If you need to use a public image, make sure that the cluster can access the public network.
command: ["/bin/sh", "-c", "sleep 36000"]
resources:
limits:
cpu: "2"
memory: 4Gi

Execute the following command to create the SandboxTemplate.

kubectl create -f dynamic-mount-template.yaml

Step 2: Create a SandboxWarmPool

The warm pool will warm up a batch of Sandboxes in the Running state based on the template, so that claim requests directly hit the warmed instances. The Hidercar and Init containers required for dynamic mounting are also injected at this stage.

Create SandboxWarmPool. The example YAML file dynamic-mount-warmpool.yaml is shown below.

apiVersion: extensions.agents.x-k8s.io/v1alpha1
kind: SandboxWarmPool
metadata:
name: dynamic-mount-warmpool # Warm pool name.
namespace: default # Namespace of the warm pool.
spec:
replicas: 10 # The number of sandboxes in the prewarming pool. A larger value means more prewarmed instances and a higher claim hit rate, but also higher resource consumption. We recommend evaluating based on peak workload concurrency.
sandboxTemplateRef:
name: dynamic-mount-template # The name of the associated SandboxTemplate.
updateStrategy:
type: Recreate

Run the following command to create the SandboxWarmPool.

kubectl apply -f dynamic-mount-warmpool.yaml

Step 3: Create a Secret

Access TOS credentials through Secret management.

Create a YAML file for the Secret. The example file vke-tos-secret.yaml is as follows.

apiVersion: v1
kind: Secret
metadata:
name: tos-key # Secret name.
namespace: default # Namespace of the Secret.
type: Opaque # Secret type. In this example, the generic type is selected to store arbitrary key-value pairs.
data:
akId: ****** # The Base64-encoded account AK. Replace the value with your actual information.
akSecret: ****** # The Base64-encoded SK that matches with the AK. Replace the value with your actual information.

Run the following command to create the Secret.

kubectl create -f vke-tos-secret.yaml

Step 4: Create TOS PV

Prepare the target PV for dynamic mounting. Currently, the dynamic mounting mode only supports TOS storage.

Create a YAML file for the PV. The example YAML file pv-tos-shared.yaml is shown below.

apiVersion: v1
kind: PersistentVolume
metadata:
name: pv-tos-shared # The name of the PV.
spec:
csi:
driver: tos.csi.volcengine.com # CSI driver type. In this scenario, it must be set to tos.csi.volcengine.com, indicating the use of the csi-tos add-on.
nodePublishSecretRef:
name: tos-key # Name of the Secret related to the existing TOS access configuration.
namespace: default # Namespace of the Secret.
nodeStageSecretRef: # Used to reference a Kubernetes Secret to pass its authentication information to the csi-tos node plugin.
name: tos-key # Name of the Secret related to the existing TOS access configuration.
namespace: default # Namespace of the Secret.
volumeAttributes:
bucket: vke-tos # Name of the existing TOS bucket.
path: /
url: https://tos-s3-cn-beijing.ivolces.com # TOS bucket domain.
volumeHandle: pv-tos-shared # The unique identifier of the PV. Keep it consistent with the PV name. volumeHandle should uniquely correspond to <url>:<bucket><path>. Do not map the same volumeHandle to different <url>:<bucket><path> items. Otherwise, this may result in unexpected mounting.

Run the following command to create the PV.

kubectl create -f pv-tos-shared.yaml

Step 5: Create sandbox

This topic provides the following two ways to create a sandbox.

Claim through SandboxClaim

Create through E2B SDK

By creating a SandboxClaim and specifying mountPath, initiate a claim for the Sandbox and explicitly declare the dynamic mounting configuration. If the annotation contains the mountPath field, the dynamic mounting mode is applied. The controller executes the CSI mounting through the Hidercar channel and mounts the subdirectory of the specified PV to any absolute path inside the container.

Create SandboxClaim. The example YAML file dynamic-mount-claim.yaml is shown below.

apiVersion: extensions.agents.x-k8s.io/v1alpha1
kind: SandboxClaim
metadata:
name: dynamic-mount-claim # SandboxClaim name.
namespace: default # Namespace of the SandboxClaim.
annotations:
vke.volcengine.com/dynamic-volume-mount-config: '[{"volumeName":"data-vol","pvName":"pv-tos-shared","mountPath":"/mnt/data","subPath":"/tenant-a"}]' # Declare the volume mount information to be added for each pod.
spec:
sandboxTemplateRef:
name: dynamic-mount-template # The name of the SandboxTemplate used to claim sandboxes.

Execute the following command to create SandboxClaim.

kubectl create -f dynamic-mount-claim.yaml

Validate results

Confirm that the PV has been mounted to the specified container directory as expected and can be read from and written to normally.

Confirm that /mnt/data is an independent mount point.

kubectl exec dynamic-mount-claim -c app -n default -- df -h /mnt/data

The expected result is as follows. In the "Mounted on" column, it is /mnt/data, and the Filesystem is "TOS mount", indicating that TOS has been mounted to the target directory.

Filesystem Size Used Avail Use% Mounted on
tosfs 10T 0 10T 0% /mnt/data

Confirm that the subdirectory content can be read.

kubectl exec dynamic-mount-claim -c app -n default -- ls -la /mnt/data

The expected result is as follows. It lists the files and directories under the tenant-a subdirectory in the TOS bucket.

total 4
drwxr-xr-x 2 root root 0 Sep 15 16:20 .
drwxr-xr-x 3 root root 0 Sep 15 16:20 ..
-rw-r--r-- 1 root root 26 Sep 15 16:20 hello-from-tos.txt

Volcengine Kubernetes Engine