When deploying applications using Volcengine Kubernetes Engine (VKE) application templates, use this method to test the performance of an Elastic Instant Cache (EIC) instance.

Prerequisites

A large language model (LLM) application has been deployed through an application template in the application center of VKE. For detailed operations, refer to AI cloud-native best practices.

Procedure

Step 1: Stop the GDKV Server and enable EIC

After you deploy an application based on SGLang or Dynamo by using an application template in the application center of VKE, update the configuration file by using the application template to stop the GDKV Server and enable EIC. Then, deploy the application again.

Log in to the VKE console.

In the left navigation bar, click Application center > Helm.

On the Helm page, filter the SGLang- or Dynamo-based application by Cluster and Namespace and click the application name.

Click Update in the upper-right corner of the Helm details page.

On the Update Helm page, click the edit icon next to values.yaml in the Parameter configuration section.

Find the configurations related to GDKV and EIC, modify the configurations based on the following code block and annotations, and then click Confirm to save the update.

Modify the following configurations for the SGLang-based application.

eic:
enabled: true # Set the value to true to enable EIC KV Cache.
# The ID of the EIC instance. Log in to the EIC console and click the name of the target EIC instance to view its ID on the Instance Information tab.
instanceID: "eicydx97nk******"
# The endpoint of the EIC instance. Log in to the EIC console and click the name of the target EIC instance to view its endpoint on the Instance Information tab.
remoteURL: "eic://192.168.xx.xx:12500;192.168.xx.xx:12500;192.168.xx.xx:12500"
# The EIC node NIC. Log in to the EIC console and click the name of the target EIC instance to view the NIC on the Node List tab.
netLocalInterfaceNames: "eth1;eth2;eth3;eth4;eth5;eth6;eth7;eth8"
# The ID of the VKE node pool to which the EIC-deployed node belongs. Log in to the VKE console. In the left navigation bar of the cluster management page, click Nodes > Node pool to view the ID.
machinePool: "pd0ve352******"
gdkv:
enabled: false # Change the value to false to disable the model loading feature of the GDKV Server.

Modify the following configurations for the Dynamo-based application.

gdkv:
enabled: false # Change the value to false to disable the model loading feature of the GDKV Server.
kvcacheConfig:
eic:
enabled : true # Change the value to true to enable EIC KV Cache.
# The ID of the EIC instance. Log in to the EIC console and click the name of the target EIC instance to view its ID on the Instance Information tab.
instanceId: "eicydx97nk******"
# The endpoint of the EIC instance. Log in to the EIC console and click the name of the target EIC instance to view its endpoint on the Instance Information tab.
endpoint: "192.168.xx.xx:12500;192.168.xx.xx:12500;192.168.xx.xx:12500"
# The EIC node NIC. Log in to the EIC console and click the name of the target EIC instance to view the NIC on the Node List tab.
ClientMultiNetLocalInterfaceNames: "eth1;eth2;eth3;eth4"
modelConfig:
local:
enable: true # Change the value to true to enable local model caching.

After you are redirected to the Update Helm page, click Confirm in the lower-right corner to deploy the application.

Step 2: Uninstall the existing nodes of the EIC instance

Log in to the EIC console.

In the left navigation bar, click Instances. On the page that appears, find the target EIC instance and click the instance name to go to the details page of the instance.

On the instance details page, click the Node List tab to view the existing nodes of the EIC instance.

Select the check boxes before all nodes and click Uninstall below the node list to uninstall the nodes.

Step 3: Scale out the nodes of the EIC instance

On the details page of the EIC instance in the EIC console, click the Instance information tab.

On the Instance information tab, click Expand next to Node count in the Configuration information section.

On the Expand page, select the nodes that you want to scale out. If the number of nodes of the instance is 0, modify configurations such as the memory and disk capacity used by the nodes, and then click Expand instance in the lower-right corner to scale out nodes.

We recommend that you set at least 1,024 GiB of DRAM for each GPU to deliver higher inference performance.

Step 4: Verify performance

Generate the benchmark test script.

Download the vLLM or SGLang benchmark test script.

https://github.com/vllm-project/vllm/tree/main/benchmarks

Run the following command to start the script.

bash test.sh >test.log 2>&1 &

The following sample code shows the script content.

# Define the length of the input and output.
OUTPUT_LEN=1500
# Create a logs directory (if it does not exist)
model=Deepseek
# When GPU and EIC performance is tested, replace dir with the log storage directory related to the model application.
dir=logs_vllm_Deepseek
mkdir -p $dir
# Define request-rate and max-concurrency.
#declare -a input_lens=(200 500 1024 2048 4096 8192 16000 32000)
#declare -a request_rates=(1 16 32 48 64 80 128 160)
#declare -a max_concurrency=(1 16 32 48 64 80 128 160)
#declare -a num_prompts=(4 64 128 192 256 320 512 640)
declare -a max_concurrency=(2 4 8 16)
declare -a input_lens=(3500 3500 3500)
declare -a num_prompts=(16 32 64)
SEED=1
# Run benchmark test.
for i in "${!input_lens[@]}"; do
for j in "${!max_concurrency[@]}"; do
INPUT_LEN=${input_lens[$i]}
RATE=${max_concurrency[$j]}
CONCURRENCY=${max_concurrency[$j]}
PROMPTS=${num_prompts[$i]}
SEED=$((SEED + 1))
echo "Running benchmark with input len: $INPUT_LEN, output len $OUTPUT_LEN, request rate $RATE, concurrency $CONCURRENCY, prompts $PROMPTS, first round"
python3 benchmark_serving.py --backend vllm \
--tokenizer /models/deepseek \
--model $model \
--base-url http://127.0.0.1:8080 \
--endpoint /v1/completions \
--metric_percentiles '50,90,95,99' \
--goodput ttft:5000 tpot:250 \
--num-prompts $PROMPTS \
--request-rate $RATE \
--max-concurrency $CONCURRENCY \
--random-input-len $INPUT_LEN \
--random-prefix-len 0 \
--random-output-len $OUTPUT_LEN \
--dataset-name random \
--ignore-eos \
--seed $SEED \
--save-result \
--result-dir ./$dir \
>> $dir/${model}_${CONCURRENCY}_${PROMPTS}_${INPUT_LEN}_${OUTPUT_LEN}_firstround.txt 2>&1
sleep 1
echo "Running benchmark with input len: $INPUT_LEN, output len $OUTPUT_LEN, request rate $RATE, concurrency $CONCURRENCY, prompts $PROMPTS, second round"
python3 benchmark_serving.py --backend vllm \
--tokenizer /models/deepseek \
--model $model \
--base-url http://127.0.0.1:8080 \
--endpoint /v1/completions \
--metric_percentiles '50,90,95,99' \
--goodput ttft:5000 tpot:250 \
--num-prompts $PROMPTS \
--request-rate $RATE \
--max-concurrency $CONCURRENCY \
--random-input-len $INPUT_LEN \
--random-prefix-len 0 \
--random-output-len $OUTPUT_LEN \
--dataset-name random \
--ignore-eos \
--seed $SEED \
--save-result \
--result-dir ./$dir \
>> $dir/${model}_${CONCURRENCY}_${PROMPTS}_${INPUT_LEN}_${OUTPUT_LEN}_secondround.txt 2>&1
sleep 1
echo "Running benchmark with input len: $INPUT_LEN, output len $OUTPUT_LEN, request rate $RATE, concurrency $CONCURRENCY, prompts $PROMPTS, finished"
done
done

(Optional) Disable EIC to compare and verify performance.

If you need to compare the performance before and after EIC is enabled, you can perform the steps described in Step 1 to find the YAML configuration file of the application and disable EIC based on the following code and annotations. Then perform testing again to compare performance.

SGLang-based application

eic:
enabled: false # Change the value to false to disable EIC KV Cache.

Dynamo-based application

kvcacheConfig:
eic:
enabled : false # Set the value to false to disable EIC KV Cache.