Docs

Volcengine Kubernetes Engine

LLM applications

Deploying general-purpose LLM template applications

This topic describes how to deploy inference services by using the general-purpose LLM deployment template in AI application templates.

Usage instructions

To ensure the successful deployment of the text model, which involves multiple add-ons that depend on each other and have complex configurations, you need to strictly follow the instructions in this topic.

To achieve the expected results and conform to the limits of Volcengine Kubernetes Engine (VKE), you need to perform operations by following the solution described in this topic or on the recommended resources. If you need to change your solution, you can consult the Account Manager of the Volcano Engine.

Do not use mGPU on GPU resources.

The complete deployment process may depend on the following Volcano Engine beta capabilities. In your deployment, apply for beta capabilities that are necessary in your actual business scenarios.

Beta feature/resource	Related service	Description
High Performance Computing Optimized instance family	GPU Compute Service	The instance families of GPU Compute Service, which provide computing power for AI inference applications. In GPU Compute Service, some instance types of the High Performance Computing Optimized instance family and the instance types of the GPU Accelerated instance family are at the beta stage. For example, the ecs.hpcpni3ln instance type, which is required for deploying most AI application templates for ServingKit, is currently at the beta stage.
GPU Accelerated instance family	GPU Compute Service
Capability of mounting nodes to local disks	Volcengine Kubernetes Engine	You can configure the local disk mounting in the VKE cluster.
EIC	EIC	A high-performance distributed KV Cache service launched by the Volcano Engine storage team for large language model (LLM) inference. Only some AI application templates support KV Cache. Apply to use this service based on your business requirements and the capabilities supported by the template.

Preparations

You can create a managed heterogeneous computing cluster of VKE in advance or create a cluster when you deploy the model. For details, see Creating managed heterogeneous computing clusters.

Prepare storage for storing model data and application data. Supported cloud storage services includes TOS buckets, Cache FileNAS instances, and EFS instances. You can also use cloud disks on nodes of the VKE cluster where your AI application is deployed.

(Optional) If you want to expose the service by using API Gateway (APIG), you need to prepare an APIG instance. You can create an APIG instance in advance or when you deploy the model. For details, see Creating instances.

Procedure

In the left navigation bar, click ServingKit > AI application templates.

On the ServingKit AI application templates page, move the pointer over LLM General Template and click Deploy.

On the Deploy page, specify the parameters of the model.

Parameter	Description
Select framework and orchestration
Selected templates	It displays the LLM General Template that you have selected.
Image	The image of the inference engine. The following image types are supported: Preset: Select a ServingKit preset image and use the Advanced filter feature to view different inference engine models that ServingKit provides for different production environments. Select an image based on your business requirements. Custom: Enter the third-party image address or CR image address. Subsequently, you need to configure the start command, model path configuration, and other settings for the custom image.
Inference engine	This parameter is displayed after Custom is selected for the Image parameter. Select an inference engine for your AI application. The following inference engines and frameworks are supported: sglang: Open-source LLM inference framework. It is applicable to complex control‑flow scenarios such as agents, multi‑step references, and JSON/regex‑constrained generation. It provides an efficient and reliable service deployment mode for text‑to‑text models. vllm: Open-source LLM inference engine. It is applicable to online services requiring high throughput (such as chatbots and content summarization), enterprise-level LLM service deployment, real-time conversation systems, and large-scale text generation scenarios. comfyui-openvefuser: The self-developed AIGC inference acceleration add-on of Volcano Engine. It seamlessly integrates with the ComfyUI ecosystem and is applicable to text-to-image and text-to-video tasks in the AIGC field. For details about the add-on, see Volcano Engine Enhanced Edition of the ComfyUI add-on. openvefuser: The self-developed general-purpose AIGC inference add-on of Volcano Engine. It can be deployed independently or integrated as an add-on with ComfyUI to provide efficient inference services for multimodal models. For details about the add-on, see Openvefuser add-on.
Model configuration	Configure the model settings for the AI application: StorageClass type: Select the storage service used to store model data. The following options are supported: TOS: Use TOS to store model data. You also need to specify a bucket. For details, see Creating buckets. Data volume: Use a data volume on a node of the VKE cluster to store model data. If the node has local disks, local disks are prioritized. Otherwise, the model data will be downloaded to the EBS data disk. Cached NAS: [Beta] Use Cache FileNAS to store model data. You also need to specify a file system and access point. For details, see Cache FileNAS. EFS: [Beta] Use EFS to store model data. You also need to configure a specific file system and access point. For details, see Creating file systems and Managing access points. tip Both Cache FileNAS instance types and EFS are in the beta stage. To use them, contact your Volcano Engine account manager to apply for beta access. We recommend using EFS for storing ComfyUI models data. Select Models: Based on your actual requirements, click Add predefined model or Add custom model and configure the Storage path for the model. The data at the Container mounting path of the model will be stored in this storage path. The description of the preset model and the custom model is as follows: Preset model: The preset model provided by ServingKit. Select the target preset model from the drop-down list, and configure a storage path for the model. After using a preset model, you can use the model in your current application without manually downloading the model on the node. Custom model: Use your prepared model to create an application, and configure a storage path for the custom model. Subsequently, you need to download your custom model to your node. For details, see FAQ. AK & SK: This setting is required for using TOS to store data. Enter the Volcano Engine access key, which is used by VKE to access your TOS resources. It is recommended to use the access key of an IAM user with minimal permissions required to access the involved files. For information on how to obtain an access key ID and secret access key, see Managing access keys.
Application configuration
Application name	Set an application name that complies with the on-screen naming requirements and a description. You can also use the automatically generated name.
Inference mode	Select the inference mode for the AI application. You can select the PD co-location or PD disaggregation mode. For details, see Terms.
Instance configuration warning If PD disaggregation is selected for Inference mode, apply the following instance configuration respectively to Prefill and Decode. In PD disaggregation inference mode, you can also configure instances for model-gateway, which functions as a high-performance request router and load balancer in the PD disaggregation architecture by clicking the plus sign (+).
Instance specification	Instances are the entities that implement inference models. Set the Number of instances and Single instance GPUs for running the inference model as needed.
RDMA settings	You can select Enable RDMA network to achieve interconnection between instances through high-speed RDMA networks and improve communication efficiency. After enabling the RDMA network, you can click the icon to enter the number of RDMA cards, or leave it blank. The system will automatically specify the number based on the value of Resource configuration. We recommend automatic calculation by the system. tip In the Prefill and Decode phases, the RDMA network is enabled, and the model-gateway is disabled by default. You can modify the configurations based on your business requirements.
Resource configuration	Select a managed VKE cluster for heterogeneous computing to deploy the model application. Prepare the resources in a preferred way: Select existing: Select a created managed heterogeneous computing cluster of VKE and select the resources of the cluster, such as the namespace and the node pool, based on on-screen prompts. Automatically create: When you deploy the model, you can create a custom managed heterogeneous computing cluster of VKE. For a detailed parameter description, see Creating managed heterogeneous computing clusters. warning If you select Enable RDMA network, you need to select a node pool with the GPU Accelerated specification. Only this node type supports interconnections through a high-speed RDMA network. If you use existing managed clusters for heterogeneous computing and use TOS storage for the current model application, make sure that csi-fsx and fsx-service add-ons are installed in the clusters. TOS storage can be mounted to containers only via FSX. You can follow the on-screen prompts to navigate to the Addons page of the corresponding cluster and install the add-ons. For details, see csi-fsx and fsx-service.
Start command	Set the start script and port number for the LLM application.
More configurations	Health check: Configure health checking for the AI application as needed. You can specify the Liveness check, Ready check, and Start check parameters. You can test whether workloads are normal by using an HTTP, TCP, gRPC, or command check. Environment variables: Create environment variables. Environment variables are injected into the container during application deployment. Privileged container: After you enable this feature, privileged containers deployed on Linux systems can control the network stack or access devices on hosts. KV Cache: The data caching mechanism of the AI application. ServingKit supports high-performance distributed KV cache based on EIC.
Service configuration
API Gateway	If you want to configure Ingress by using APIG to process inbound traffic to the model, enable APIG and select the target APIG instance. Select existing: Use an existing APIG instance as the route for the current application. You can also click Create API Gateway instances to go to the APIG console to create a custom instance. For details, see Creating instances. Automatically create: When you deploy the AI application, an APIG instance with fixed instance specifications and configurations is automatically created by ServingKit. A newly created APIG instance will incur fees. For details, see APIG service billing. tip When you use an existing APIG instance to expose the model service in the cluster, take note of the following parameter or feature configurations: Region: It needs to be consistent with the region of the AI application. Circuit breaking: When some GPU instances malfunction, the common method is to take the faulty instances offline to eliminate failures. At the same time, it is necessary to retain the faulty instances for troubleshooting later. You can isolate the faulty instances by configuring a circuit breaking rule for the APIG instance. For details, see Circuit breaking policies.
Intelligent routing	Enable it to use AI Gateway to distribute inference traffic. The AI Gateway provides a standardized interface for calls to different models, improving the availability and observability of model calls.
More configuration
Update policy	The update policy is enabled by default. It follows the InPlaceUpdate policy, which performs in-place updates. The InPlaceUpdate policy allows you to directly update the existing RoleSet. During the update, additional GPU resources are not used.
Model download	Select the model download method. The model will be automatically downloaded to all nodes in the node pool for the application. For the descriptions of related terms, see Terms.

Click Confirm to complete the configuration.

Please wait patiently as the application creation takes approximately 15 to 20 minutes. You can also click Back to AI application list to view the application status, or click Go to AI application template to view AI application template details or create another AI application.

What to do next

After you deploy the AI application, you can manage, view, and use the AI model application on the List of AI applications page. For details, see the following documentation:

Managing and using AI applications

AI application observability

Last updated: 2026.10.16 20:33:48