Docs

Code Pipeline

AI application

Create and deploy AI applications (customized)

The AI application is a one-stop platform for development, deployment, and operations, launched for large model online inference scenarios. Custom creation provides users with a relatively flexible method for creating AI applications. By specifying images, models, code, start commands, and third-party libraries, users can deploy large AI models to accommodate the vast majority of users' custom scenarios. This article introduces how to deploy AI applications using a custom creation method.

Prerequisites

Volcengine Kubernetes Engine (VKE)

The VKE cluster has been created. For specific operations, refer to Create a cluster.

Optional: If you need to mount object storage TOS, please install the csi-tos component in advance. For specific operations, refer to Install components.

If you want to deploy using the Volcengine Container Instance Method, note that this is only supported for VKE clusters with the VPC-CNI network model, and the vci-virtual-kubelet component must be installed in the cluster in advance. For specific operations, refer to Install components.

(Optional) If you need to use GPU resources, please install the nvidia-device-plugin component in advance. For specific operations, refer to Install components.

(Optional) If you need to mount a ConfigMap, create the ConfigMap in advance in the namespace to be deployed within the VKE cluster. For specific operations, refer to Create parameter.

Code Pipeline CP

The deploy resource has been connected. Deploy resource must be a Volcengine Kubernetes Engine (VKE) cluster. For specific operations, refer to Integrate deploy resources.

The trained model has been obtained.

(Optional) If you need to use a custom image, please prepare the image address in advance or upload the image to the Volcano Engine image repository. For specific operations, refer to Getting started with pushing and pulling images.

(Optional) If you need to mount model files, please complete the following preparations. You only need to select one storage method.

Torch Object Storage (TOS)

The model has been uploaded to TOS Bucket storage. For specific operations, refer to Object Storage Getting Started.

The access keys AK & SK for the current account have been obtained. It is recommended to use access keys for IAM users with least privilege. Ensure that the user has already been authorized to access the corresponding file.

FileNAS

The model has been uploaded to NAS storage. NAS and the VKE cluster to be deployed must be in the same VPC. For specific operations, refer to File Storage getting started.

Artifact warehouse

The model has been uploaded to the artifact repository. For specific operations, refer to Getting started with uploading and downloading AI format artifacts.

The access keys AK & SK for the current account have been obtained. It is recommended to use the access key of an IAM user with minimal permissions. This IAM user has permission to mount the repository.

(Optional) If you need to configure an external access method, complete the following preparations and select any one method.

API Gateway: The gateway instance has been created. The gateway instance and the VKE cluster to be deployed must be in the same VPC. For specific operations, refer to Create an instance.

CLB load balancer: A load balancer instance has been created. The load balancer instance must be in the same VPC as the VKE cluster to be deployed. For specific operations, refer to Create a load balancing instance.

Procedure

Select AI application in the left navigation bar.

On the AI application page, click create application.

Select the Custom create template, and click Next: Application configuration.

Fill in the relevant configuration information for the application as required. After the configuration is complete, click Confirm, and the application will begin to be created and deployed.

Basic Information

Configuration Item	Descriptions
Application ID	Enter the Application ID as prompted on the interface. The Application ID is the unique identifier for the application and cannot be changed after it is created.
Application name	Display name of custom application.
Description	Enter remarks for the current application. This field is optional.

Deploy a cluster

Configuration Item	Descriptions
deploy resource	Select the VKE deployment clusters included in the prerequisites. warning Deployment using Volcengine Container Instance is only supported for VKE clusters with the VPC-CNI network model, and the vci-virtual-kubelet component must be installed in the cluster in advance.
Namespace	Select or enter the namespace to be deployed. If the namespace you enter does not exist, it will be created automatically.
Environment ID	Customize the unique identifier for the current environment. After the Environment ID is created successfully, it cannot be modified.

Model configuration

Configuration Item	Descriptions
Image selection	Supports selecting the container image used by AI applications through the following methods. Prebuilt image: CP offers prebuilt official images based on different machine learning frameworks, helping you quickly build the runtime environment for AI applications. You can select appropriate images and versions based on the model you are using. The system currently supports Triton, TensorFlow, PyTorch, Stable Diffusion WebUI, and Llama-Factory. Custom image: If preset images do not meet your business needs, you can select your own container image from your Volcano Engine image repository. Please select the Region, Instance, Namespace, OCI Repository, and Tag where the target image is located. To authorize a private image repository, check the Authorization switch, and enter the Username and Password. Image address: If the default image does not meet your business requirements, you can directly enter an external image address. The format of the image address is host/namespace/repo:tag. To authorize a private image repository, check the Authentication switch and enter the Username and Password.
Model selection	(Optional) Click add model mount, and configure model files. An AI application supports mounting multiple models. The following methods are provided to configure model files. Please select as needed. Mount TOS: Mount model files using TOS through Volcano Engine Object Storage. Select the Region and Bucket where the model file is located, and enter the specific path. AK & SK: Enter the access keys AccessKey ID (AK) and Secret Access Key (SK) for the current account. It is recommended to use IAM user access keys with the least privilege, and ensure that permission to access the corresponding files has been granted. Mount path: The target path mounted in the AI application, used to read model files from the TOS path. Mount artifact repository: Mount model files through the Volcano Engine artifact repository. Artifact repository: Select the Region, instance, repository, namespace, model name, and version where the model file is located. Only the AI artifact repository is supported for selection. AK & SK: Enter the AccessKey ID (AK) and Secret Access Key (SK) for the current account. It is recommended to use the access key of an IAM user with minimal permissions. This IAM user must be granted permission to mount the repository. Mount path: The target path mounted in the AI application, used to read model files from the artifact repository. NAS mount: Mount model files to NAS using Volcano Engine File Storage. File system type: Select the file system type. Supports Performance type and Capacity type. NAS and the VKE cluster to be deployed must be in the same VPC. Mount point: Select the file system and mount point. Subdirectory: Enter the subdirectory information of the file system to be mounted; the default is the root directory./. Mount path: The target path mounted in the AI application, used to read model files from the NAS path.
Startup command	Set the application's startup command and the container port to listen on. Start command: For different preset images, the system provides default start commands. For example, when using the trition image, the default start command is tritonserver --model-repository=/models. Support modifying the start command based on business requirements. Port: Set the container port that the application listens on. warning --model-repository is used to specify the model path and must be consistent with the model selection section's mount path. Lifecycle command configuration: To configure commands to be executed after the container starts or before it stops, please check Lifecycle command configuration. After the container starts: Enter the command to execute after the container starts. Before container stops: Enter the command to execute before the container stops.
Advanced configurations	Optional: Supports adding input/output configuration, third-party library configuration, environment variables, and health checks to the application as needed. For details, refer to the table below.

Advanced configuration (optional)

Configuration Item	Descriptions
Input and output configuration	Support storage configuration for code, parameters, plugins, outputs, and more. The following configuration methods are available. Please select as needed. TOS mount: Select the target TOS Region, Bucket, and enter the specific path. AK & SK: Enter the access key AccessKey ID (AK) and Secret Access Key (SK) for the current account. It is recommended to use the access keys of IAM users with the least privilege. Mount path: The target path to mount in the AI application. Parameter mounting Name: Please enter the name of the parameter (ConfigMap). Requirement: Parameters must belong to the namespace to be deployed. Mount path: The target path mounted in the AI application. NAS mount: File system type: Select the file system type. Performance type and Capacity type are supported. NAS and the VKE cluster to be deployed must be in the same VPC. Mount point: Select the file system and mount point to be mounted. Subdirectory: Enter the subdirectory information for the file system to be mounted; the default is the root directory./. Mount path: The target path to be mounted in the AI application.
Third-party library configuration	Configure some third-party library dependencies required during application runtime. PIP: Download the required Python dependencies from the Volcano Engine PIP mirror source. You only need to enter the file name; to specify multiple files, you can use,Or use as a separator.
Environment variable	Configure environment variables as needed. Environment variables are injected into the application's running environment, allowing the application to be dynamically configured at runtime. Support adding multiple environment variables. Key: Enter the name of the environment variable. Value: Enter the value of the environment variable.
Health check	Currently, this is a liveness check: verifies whether the container is running, and restarts the container if the probe fails. Inspection method HTTP request check: Using the HTTP or HTTPS protocol, send an HTTP GET request to the container and confirm the container status by checking the response. You need to perform the following configuration checks: Protocol: HTTP or HTTPS. Custom request headers in HTTP requests (HTTP Headers). Click add, enter the key and value of the request header. Path: Enter the access path in the business code that is used for health checks. Default is/Supports modification. Port: Enter the access port for the application. The default is 8080 and can be changed. TCP port check: Attempt to establish a TCP connection to the application to detect whether a connection can be established successfully. You need to perform the following configuration checks: Port: Enter the application access port. The default value is 8080 and can be changed. Time settings Time parameters for configuration check include: Initial wait time: The time to wait after the application starts before the first probe is executed. The default value is 1 second. Timeout period: The timeout period for waiting for a response after sending a check request. If this time is exceeded, the application is considered unresponsive. The default value is 1 second. Inspection interval: The time interval between two consecutive inspections. The default is 10 seconds. Threshold value settings Result threshold values for configuration checks, including: Success threshold value: After a check request is sent, this value indicates the number of normal system responses. The default is 1. This item cannot be configured. Failure threshold value: The number of non-responses indicating a system exception after a check request is sent. The default is 3 times. That is, if the application does not respond to a request three times, its status is considered abnormal.
Custom tags and annotations	The application supports adding labels (Label) and annotations (Annotation).
Distributed inference	Support is currently available only when the pre-installed image is vLLM or SGLang. Number of machines: Enter the number of machines required to deploy a single instance. For example, a large model service requires 4 instances, and each instance needs 2 machines for deployment. Therefore, a total of 4 ✖️ 2 = 8 machines (nodes) are needed to deploy the service. If the cluster has fewer than 8 nodes, deployment will fail. For information on adding nodes in VKE, refer to Cluster expansion. RDMA configuration: Whether to turn on the RDMA feature in the resource. RDMA (Remote Direct Memory Access) is a high-performance network protocol that reduces CPU usage and memory bandwidth bottlenecks, as well as improves bandwidth utilization. Currently, only high-performance computing GPU instances support RDMA. To turn on the use of RDMA resources in Volcengine Kubernetes Engine, please refer to the documentation: Using RDMA resources in VKE clusters. Number of RDMA cards: Number of cards with the RDMA feature turned on.
Rolling Update	Once enabled, the service will be updated according to the rolling update strategy to avoid service interruption during the update process. Max Unavailable: The maximum number of instances that must not be exceeded during a rolling update. Max Surge: The maximum number of instances allowed to temporarily exceed the desired total number of instances during a rolling update.

Service specifications

Check Volcengine Container Instance method deployment

Configuration Item	Descriptions
Number of instances	Set the number of application replicas based on business scale.
Volcengine Container Instance (VCI)	Keep Deploy using the Volcengine Container Instance method checked. Using this method to deploy applications, there is no need to manage underlying cloud servers or other infrastructure. You only need to provide an image to run containers, and pay for the resources actually consumed. For billing details, refer to Elastic Container billing description. tip If you have applied for the image caching beta feature on the VCI side, image caching will be turned on by default when an AI application is created to accelerate image pulling.
Log collection	Indicates whether VCI container logs are collected through the Volcano Engine Log Service TLS module and stored persistently. Check this box to enable log collection. Do not check the box, and do not turn on log collection. warning After you turn on log collection, make sure that you have completed the relevant log collection rule configuration in the log center of the VKE cluster to ensure that logs can be properly delivered to your configured project and topic. For specific operations, refer to Collect container logs.
Resource configuration type	Select the instance specifications to use based on business requirements. Virtualization capabilities vary by computing specification. For details, refer to Introduction to VCI instance specifications. General-purpose: A general-purpose specification family offering various vCPU and memory combinations. GPU Compute type: GPU compute instance specification families based on various NVIDIA Tesla graphics cards.
System disk	Set the system disk capacity. The system disk is primarily used to cache images; its capacity must be greater than the image size. tip If you have applied for the beta feature on the VCI side that allows manual specification of system disk size, you can customize the system disk capacity here.

Do not check Deploy using the Volcengine Container Instance method

Configuration Item	Descriptions
Number of instances	Set the number of application replicas based on the scale of the business.
Resource configuration type	CPU: Defines the CPU specifications for each instance. Where CPU request ≤ CPU limit. Memory: Defines the memory specification for each instance. where the memory request ≤ the memory limit. GPU Resource type: Whether GPU resources are used. Not turned on: Uses traditional CPU computing resources. Nvdia: Uses GPU computing resources. Currently provides Nvdia card models and supports custom GPU computing power. mGPU: mGPU (multi-container GPU), also known as shared GPU, is based on kernel-level virtualization to isolate GPUs and incorporates a self-developed scheduling framework. It supports sharing GPUs among multiple containers, enabling flexible scheduling and isolation of computing power and video memory, reducing costs and ensuring resource utilization. mGPU supports single-card and multi-card sharing modes. In multi-card mode, configuration of the computing power allocation policy is required. Fixed quota: Each container has a fixed quota of computing power. During container operation, even if the GPU has remaining computing power, it cannot exceed the fixed quota of computing power. Guaranteed burst: Each container has a guaranteed minimum compute quota, but as long as the GPU has idle compute resources, the container can use them. Burstable mode: You can set the compute quota. The higher the value you set, the greater the quota that can be competed for. However, it is not guaranteed that each container will be able to compete for a quota equal to the configured value.

Optional access configurations

Configure the access method for the AI application. For detailed information, refer to Access settings.

Result validation

Verify successful application deployment

On the application's Basic information > Resource configuration tab, view the application's deployment progress. You can view detailed startup and runtime information for the application through the instance's running status, real-time logs, and events.

View real-time logs: Click Real-time logs to view the application's runtime information, which facilitates operations, maintenance, and troubleshooting.

View events: Click Events to view the application's startup information. You can view event information from at most the past hour.

Obtain the application access domain name

After the application is deployed successfully, you can copy the application's access address in the access settings section. Access the AI application using this address.

Verify whether the application has started successfully

Different models use different validation methods. The following figure takes an AI application that uses the Triton inference engine as an example. You can use the following command to check whether a specific model in the application has started successfully.

Last updated: 2026.06.17 16:54:30