Docs

Volcengine Kubernetes Engine

Mounting PVs

PersistentVolume mounting overview

This topic describes how to use the PersistentVolume (PV) mounting capability in the VCI Agent Sandbox scenario to implement pre-mounting and dynamic mounting of EFS and TOS PVs in Sandbox warm pool instances. The purpose is to meet the diverse storage requirements of different businesses and tenants.

Feature introduction

VCI Agent Sandbox usually leverages a warm pool (SandboxWarmPool) to accelerate instance creation. The system provisions a batch of sandboxes in the Running state based on the SandboxTemplate. When a SandboxClaim initiates a claim request, it directly hits an instance in the warm pool. This reduces the startup time from tens of seconds to less than a second. However, warm pool instances are mass-produced based on the same SandboxTemplate, and it is impossible to predict the diverse PV requirements of different businesses and users, for example:

In multi-tenant scenarios, Sandboxes of different tenants need to mount different subdirectories of the same PV.

Some businesses can only determine the mount directory inside the container when creating sandboxes. They cannot declare it in the template in advance.

For businesses deployed across multiple clouds, the existing annotation already contains mountPath configuration, and they require a low-cost migration approach.

To address these requirements, VCI Agent Sandbox provides the PV mounting capability and offers two modes for different scenarios: pre-mounting and dynamic mounting.

Based on whether the PV-mounting-related annotations contain the mountPath field, i.e., the mount directory in containers, the system automatically distinguishes the two mutually exclusive modes.

Node	Trigger condition	Core capabilities
Pre-mounting (PreMount)	No entries contain mountPath.	Based on predefined PVs in the template, SandboxClaim dynamically replaces the PV and subdirectory (subPath) when claiming the sandbox instance. mountPath is fixed in the template.
Dynamic mounting (Dynamic)	All entries contain mountPath.	There is no need to predefine PVs in the template. SandboxClaim dynamically specifies mountPath when claiming the sandbox instance and supports adding new PVs and volumeMount.

tip

In summary, the PV mounting capability of VCI Agent Sandbox follows the all-or-nothing principle of mountPath, i.e., all entries in the same annotation must either contain mountPath or not contain it at all. Partial inclusion will lead to direct rejection, and the entire dynamic mounting configuration will not take effect.

Technical architecture

The two modes have essential differences in mounting add-on form and execution chain, which directly determine their respective security boundaries and performance.

Architecture of pre-mounting mode

In pre-mounting mode, Volume and volumeMount are already declared in the template, and mountPath is fixed when the container is created. At the warm pool stage, the system injects a non-privileged VCI Sidecar container. This container only exposes limited internal interfaces. When SandboxClaim claims a sandbox instance, the controller executes commands in the Sidecar, and VCI at the underlying layer directly replaces the endpoint of the storage backend. The mount directory inside the business container remains unchanged.

The key features of this solution are as follows:

The Sidecar container runs as a non-root user, with a read-only root file system, all Linux capabilities removed, and privilege escalation prohibited.

It does not involve host directory sharing and does not use bidirectional mount propagation.

The mounting is performed by the VCI trusted add-on. It is not allowed inside business containers.

The credential is transferred once. The Sidecar automatically exits after the call is completed.

Architecture of dynamic mounting mode

The dynamic mounting mode requires running a CSI storage mounting add-on in the sandbox. VCI adopts the Hidercar (shadow container, a kind of hidden Sidecar) solution, injecting the mounting add-on through the VCI control plane without the need to provision a volume. After the system recognizes the annotation during the warm pool stage, VCI injects the storage mounting add-on as a Hidercar. The add-on is completely invisible to the Virtual Kubelet and end users, minimizing the attack surface while preserving the flexibility of dynamic mounting. The detailed operations are as follows:

Inject the VCI scheduling annotation configuration.

Inject an emptyDir shared volume. It is used to transfer CSI executable files, Socket, and the root mount directory between containers.

Inject the Hidercar shadow container to run the CSI driver in privileged mode.

Inject an Init container to distribute the executable files required for mounting to the agreed directory.

When SandboxClaim claims a sandbox instance, the controller executes commands through the Hidercar channel. After the CSI driver completes the storage mounting, the mount point is propagated to the shared directory through bidirectional mount propagation, and then mapped to the specified mountPath through a symbolic link. The Hidercar container is invisible and inaccessible to end users throughout the entire process.

The security comparison between the Sidecar and Hidercar solutions is as follows:

Item	Sidecar solution	Hidecar solution (adopted by VCI)
Visibility	Visible to kubectl, and exec commands can be used to enter the container.	Completely invisible to the Virtual Kubelet and end users. The exec cannot be used.
Attack surface	The privileged container is exposed to users, resulting in a relatively large attack surface.	Privileged add-ons are hidden from users, significantly reducing the attack surface.
Credential security	Credentials such as AK/SK are transmitted after Base64 encoding, and business containers can probe the Sidecar process to some extent.	Hidercar independently manages storage credentials, and business containers cannot access its process and filesystem.

Feature comparison

The feature comparison of the two mounting modes is shown in the following table.

Dimension	Pre-mounting	Dynamic mounting
mountPath	Fixed in the template and cannot be modified.	Dynamically specified. It can be any absolute path inside the container.
New PV	Adding new PVs is not supported. Replacing the PVs already declared in the templates is the only supported method.	Supports adding new PVs and volumeMount to runtime.
subPath	Supported	Supported
Mounting add-on form	VCI Sidecar container (non-privileged, exits after completion).	Hidercar shadow container (privileged, but invisible to users).
Supported storage type	EFS	TOS
Execution chain	Replace the endpoint with a gRPC direct connection, making a short chain.	CSI mounting + bidirectional propagation + symbolic link. The chain is relatively long.
Multi-container support	Supports multiple containers declared in the template.	Currently only supports a single primary container.

Risk comparison

The risk comparison of the two mounting modes is shown in the following table.

Risk dimensions	Pre-mounting	Dynamic mounting
Container escape risk	No risk. The Sidecar runs in non-privileged and non-root mode, actively drops all capabilities (capabilities.drop=ALL), and has no host directory exposure risk.	Hidercar runs in privileged mode and uses bidirectional mount propagation. The bottom layer still has privileged operation capability, but Hidercar is completely invisible to end users and cannot execute exec commands. End users cannot directly access the target container or its host's directory, thus significantly reducing the attack surface on the service side. The risk mainly comes from potential vulnerabilities in the container runtime itself. We recommend pairing this with runtime security monitoring.
Cross-tenant data isolation	Strong isolation. The mount point is specified by the platform control plane, and business containers cannot determine the mount target.	The system performs path traversal validation on subPath. Storage credentials are independently managed by Hidercar and cannot be accessed by business containers. We recommend pairing this with storage directory-level permissions (such as ObjectSet/access point policy) to achieve minimal privilege isolation.
Mount point reliability	Relatively high reliability. The mount point is established when the container starts, and the Sidecar exits after completing the replacement, with no subsequent dependency.	Mounting depends on the continuous operation of the CSI add-on and the symbolic link mechanism. The current version does not support using dynamic mounting together with container in-place updates and writable layer persistence at the same time. After the container restarts, a new Sandbox needs to be created.
Performance overhead	Low overhead. Uses gRPC to directly replace the endpoint, enabling a single operation to be completed in milliseconds.	Relatively high overhead. It needs to execute the CSI mounting process (including FUSE file system mounting), which takes longer. In addition, the warm pool also needs to run Hidercar and Init containers.
Fault impact scope	Mounting failure only affects the current sandbox claim. After the sandbox instance is automatically cleaned up, a cold start is applied.	Mounting failure automatically cleans up the Sandbox instance and leads to a cold start to downgrade. Hidercar is invisible to users, avoiding fault spread caused by user misoperations or malicious access. tip When the dynamic mounting mode uses the Hidercar solution, users cannot access privileged add-ons, making it significantly more secure than the visible Sidecar solution. However, the permission granularity of storage credentials (AK/SK) is still the security baseline. We recommend limiting credential permissions to specific mount points or directory levels, avoiding using over-privileged credentials, and pairing this with runtime security monitoring (such as detection of abnormal system calls for mounting) for in-depth defense.

Mode selection recommendation

The mode selection recommendation for the two mounting modes is shown in the following table.

Application scenarios	Recommended solution
The mount directory can be determined at the SandboxTemplate stage. It is only necessary to switch to a different PV or subdirectory when SandboxClaim claims the sandbox instance.	By using the pre-mounting mode, performance and security are excellent.
Multiple tenants share the same storage instance, and different tenant directories are isolated through subPath.	Use the pre-mounting mode + subPath mode. One PV serves multiple subdirectories, eliminating the need to create a PV for each tenant.
Multi-cloud deployment. Existing business pod annotations contain mountPath and cannot be modified in the short term.	Use the dynamic mounting mode. Compatible with the Kruise protocol. Hidercar ensures security.
The mount directory can only be determined when the sandbox is running and cannot be declared in advance in SandboxTemplate.	Use the dynamic mounting mode.
The new PV not declared in the SandboxTemplate needs to be added when the sandbox is running.	Use the dynamic mounting mode.

Security boundary description

The data security boundary description for using the PV mounting capability in the VCI Agent Sandbox scenario is as follows:

Prewarming pool isolation: Although prewarming pool sandboxes that have not been claimed already have PVCs mounted, their volumes are not actually accessible (access returns Transport endpoint is not connected). Only after pre-mounting is complete can data at the new access point be accessed, thereby preventing tenants from accessing pre-mounted directories.

Data security: Data security: In multi-tenant scenarios, strictly prevent tenants from accessing or using one another's directories. If any step of pre-mounting fails, the system cleans up the sandbox awaiting claim to prevent intermediate-state data contamination or leakage.

Last updated: 2026.09.21 17:08:51