This topic describes how to deploy an optimized Retrieval-Augmented Generation (RAG) framework on ECS instances of the ecs.g4i type to enhance RAG inference efficiency. The RAG framework consists of core components such as OCR, embedding models, and vector databases.

Features

This practice uses the 4th-generation ECS instances of VolcanoEngine, specifically the General-purpose g4i type. This instance type adopts the latest self-developed DPU architecture of VolcanoEngine and is equipped with the 6th-generation Intel® Xeon® Scalable processor (Granite Rapids). This instance type also leverages next-generation hardware AI acceleration instructions AMX-BF16 (Intel Advanced Matrix Extensions) with IPEX framework optimizations based on this technology. These features deliver system-level optimization and accelerate the inference of each core component of the RAG framework.

Background

RAG

RAG is a technology that integrates information retrieval and text generation. It enhances the accuracy and reliability of generative AI models, such as large language models (LLMs), while mitigating model hallucinations.

Information retrieval: Before generating an answer, RAG uses a retrieval system to search extensive text datasets, including local documents, Internet sources, and enterprise databases, for information relevant to the input question or prompt.

Text generation: After obtaining the relevant retrieval results, RAG combines the retrieved document fragments with the user question as input to a generative AI model (for example, an LLM). The model incorporates this retrieved information as supplementary context, processes it alongside the input question or prompt, and then generates the corresponding answer or content.

OCR

Optical character recognition (OCR) is a technology that converts printed or handwritten text into machine-readable text. The technology identifies and extracts characters from images or documents, and then converts them into encoded text that can be processed by computers. This topic uses the deployed and optimized OCR tool MinerU as an example. This tool recognizes PDF text through the following steps, where the models used in each step are sourced from PDF-Extract-Kit and have all been fine-tuned using MinerU. The following table describes all steps and the models used in each step.

Step	Model
Layout detection	LayoutLMv3: Developed by Microsoft Research Asia, LayoutLMv3 is a general-purpose document AI pre-trained model capable of efficiently processing text and image multimodal information. This capability is achieved through simplified image representation, unified masked modeling, and token-aligned pre-training. You Only Look Once (YOLO): An object detection algorithm that leverages convolutional neural networks. YOLO is one of the fast object detection algorithms.
Formula detection	You Only Look Once (YOLO): An object detection algorithm that leverages convolutional neural networks. YOLO is one of the fast object detection algorithms.
Table recognition	TableMaster: This model breaks down a table recognition task into four subtasks: table structure recognition, text line detection, text line recognition, and box assignment. StructEqTable: This model performs table recognition tasks in an end-to-end manner, demonstrating stronger recognition performance and delivering favorable results even when handling complex tables.
Formula recognition	UniMERNet: A deep learning model specializing in mathematical formula recognition.
OCR	Paddle-OCR: An open-source OCR toolkit built on the deep learning framework PaddlePaddle, offering end-to-end capabilities such as text detection, recognition, and direction classification.

Embedding & Rerank

In a RAG system, Embedding and Rerank are two core stages for enhancing retrieval quality. Each serves a distinct function, and together they ensure the model can efficiently and accurately acquire relevant knowledge.

Operation	Description
Embedding	Embedding, which converts text such as questions and document segments into high-dimensional vectors (Embeddings). It measures the semantic relevance of text by calculating the "distance" (such as cosine similarity) between vectors, thereby quickly filtering out candidate content relevant to the question from a large volume of documents.
Rerank	Re-ranking, which performs a secondary sorting on the candidate documents (Top N) returned in the Embedding stage to further improve relevance.

Vector database

In the RAG framework, the vector database is a core component that connects information retrieval and text generation. It handles the efficient storage, management, and retrieval of semantic vector representations for unstructured data, directly influencing the accuracy, response speed, and scalability of the RAG system.

OCR

Install the optimized OCR

Purchase an ecs.g4i instance. In the example provided in this topic, the following instance configurations are used.

Instance type: ecs.g4i.8xlarge

Image: Ubuntu 24.04 64 bit

Network: an assigned EIP

Log in to the target instance.

Execute the following command to install dependencies.

apt update
apt install docker.io

Execute the following command to pull the OCR image optimized for the ecs.g4i instance type.

docker pull cr-for-diting-nodelete-cn-beijing.cr.volces.com/practice-images/ai:ocr

Execute the following command to start the Docker image.

docker run -it cr-for-diting-nodelete-cn-beijing.cr.volces.com/practice-images/ai:ocr bash

Execute the following command to run the demo pre-installed in the image.

cd MinerU/demo/ && python3 demo_new.py

Perform related operations.

View OCR results:

The results of OCR operations performed on PDF files using the MinerU tool are stored in the /MinerU/demo/pdfs directory. You can check the corresponding results in this directory.

Replace the OCR target:

Execute the following command to enter the directory containing the target PDF file for OCR processing.

cd /MinerU/demo/pdfs

Execute the following command to run the OCR demo.

python3 demo_new.py

Performance data

This section presents the duration data for OCR operations before and after enabling AMX acceleration, using an optimized OCR model deployed on an ECS instance of the ecs.g4i.8xlarge type.

Instance type	Not accelerated	AMX acceleration
ecs.g4i.8xlarge	49.42 s	23.73s

Embedding & Rerank

Install the optimized Embedding and Rerank

Deploy Embedding & Rerank and start a Docker image.

Use an image of a preset model for deployment
Use a standard image for deployment
Purchase an ecs.g4il instance. In the example provided in this topic, the following instance configurations are used.
Instance type: ecs.g4i.16xlarge
Image: Ubuntu 24.04 with LLM Knowledge Base 64 bit
Network: an assigned EIP
Log in to the target instance.
Execute the following command to enable AMX-accelerated Embedding & Rerank operations.

sed -i 's|ghcr.io/huggingface/text-embeddings-inference:cpu-1.6|iaas-cpu-inference-benchmark-cn-beijing.cr.volces.com/rag/tei-cpu:bytedance_fix|g' /root/opea_chatqna/compose.yaml

Execute the following command to start the knowledge base Docker image.

cd opea_chatqna
docker compose up -d

Once all services show an available status (marked in green), your AI knowledge base is started successfully.
Purchase an ecs.g4i instance. In the example provided in this topic, the following instance configurations are used.
Instance type: ecs.g4i.16xlarge
Image: Ubuntu 24.04 64 bit
Network: an assigned EIP
Log in to the target instance.
Execute the following commands to install and configure Docker.

if ping -W 2 -c 2 mirrors.ivolces.com &> /dev/null; then
docker_url="mirrors.ivolces.com"
else
docker_url="mirrors.volces.com"
fi

sudo apt update
sudo apt install ca-certificates curl gnupg lsb-release -y

sudo mkdir -p /etc/apt/keyrings
curl -fsSL https://$docker_url/docker/linux/ubuntu/gpg | sudo gpg --dearmor -o /etc/apt/keyrings/docker.gpg --yes

echo "deb [arch=$(dpkg --print-architecture) signed-by=/etc/apt/keyrings/docker.gpg] https://$docker_url/docker/linux/ubuntu $(lsb_release -cs) stable" | sudo tee /etc/apt/sources.list.d/docker.list > /dev/null

sudo apt update

sudo apt install docker-ce docker-ce-cli containerd.io docker-compose-plugin -y

Execute the following command to install the runtime environment.

pip install click \
deepdiff \
deepeval==1.4.0 \
evaluate \
flask \
jieba \
kubernetes \
langchain_community \
langchain_huggingface \
lm-eval==0.4.3 \
locust \
numpy \
pytest \
pyyaml \
ragas==v0.1.19 \
requests \
rouge_score \
sseclient-py \
prometheus_client \
docker
pip install gradio protobuf google
pip install cmake==3.26.1 sentencepiece==0.1.99 tokenizers==0.19.1 accelerate==0.26.0
pip install xfastertransformer==1.8.2
pip install transformers==4.44.0
if ! command -v oniond &> /dev/null; then
wget -qO - https://mirrors.ivolces.com/extra-tools/debian/GPG-KEY-system | apt-key add -
echo 'deb http://mirrors.ivolces.com/extra-tools/debian buster main' > /etc/apt/sources.list.d/volctools.list
apt update
apt install -y onion-ai-data
fi

Run the following command to create a configuration file.
Execute the following command to create and enter the configuration file directory.

mkdir opea_chatqna
cd opea_chatqna

Execute the following command to create the compose.yaml file.

vim compose.yaml

Press the i key to enter the edit mode, and copy the following content to the file.
tip
The following script uses the ecs.g4i.16xlarge instance type as an example. Modify the following CPU pinning parameters based on your needs to achieve optimal performance.
cpuset: This parameter specifies the IDs of CPU cores and restricts the service to run exclusively on these specific cores. In the example provided in this topic, this parameter is set to 32,34,36,38,40,42,44,46,48,50,52,54,56,58,60,62.
OMP_NUM_THREADS: This parameter specifies the number of threads utilized by the OpenMP parallel computing framework, directly governing the thread concurrency level in multithreaded programs. In the example provided in this topic, this parameter is set to 16.

services:
redis-vector-db:
image: cr-for-diting-nodelete-cn-beijing.cr.volces.com/practice-images/redis-stack:7.2.0-v9
container_name: redis-vector-db
ports:
- "6379:6379"
- "8001:8001"
healthcheck:
test: ["CMD", "redis-cli", "ping"]
interval: 5s
timeout: 3s
retries: 3
dataprep-redis-service:
image: cr-for-diting-nodelete-cn-beijing.cr.volces.com/practice-images/dataprep:${TAG:-latest}
container_name: dataprep-redis-server
depends_on:
redis-vector-db:
condition: service_healthy
tei-embedding-service:
condition: service_started
ports:
- "6007:5000"
volumes:
- "./weights:/data"
environment:
REDIS_URL: redis://redis-vector-db:6379
REDIS_HOST: redis-vector-db
INDEX_NAME: ${INDEX_NAME}
TEI_EMBEDDING_ENDPOINT: http://tei-embedding-service:80
HUGGINGFACEHUB_API_TOKEN: ${HUGGINGFACEHUB_API_TOKEN}

tei-embedding-service:
image: cr-for-diting-nodelete-cn-beijing.cr.volces.com/practice-images/tei-cpu:bytedance
container_name: tei-embedding-server
ports:
- "6006:80"
volumes:
- "./weights:/data"
shm_size: 1g
cpuset: '0,2,4,6,8,10,12,14,16,18,20,22,24,26,28,30'
environment:
OMP_NUM_THREADS: 16
HF_ENDPOINT: https://hf-mirror.com
command: --model-id ${EMBEDDING_MODEL_ID} --auto-truncate
retriever:
image: cr-for-diting-nodelete-cn-beijing.cr.volces.com/practice-images/retriever:${TAG:-latest}
container_name: retriever-redis-server
depends_on:
redis-vector-db:
condition: service_healthy
ports:
- "7000:7000"
ipc: host
environment:
REDIS_URL: redis://redis-vector-db:6379
REDIS_HOST: redis-vector-db
INDEX_NAME: ${INDEX_NAME}
TEI_EMBEDDING_ENDPOINT: http://tei-embedding-service:80
HUGGINGFACEHUB_API_TOKEN: ${HUGGINGFACEHUB_API_TOKEN}
LOGFLAG: ${LOGFLAG}
HF_ENDPOINT: https://hf-mirror.com
RETRIEVER_COMPONENT_NAME: "OPEA_RETRIEVER_REDIS"
restart: unless-stopped
tei-reranking-service:
image: cr-for-diting-nodelete-cn-beijing.cr.volces.com/practice-images/tei-cpu:bytedance
container_name: tei-reranking-server
ports:
- "8808:80"
volumes:
- "./weights:/data"
shm_size: 1g
cpuset: '32,34,36,38,40,42,44,46,48,50,52,54,56,58,60,62'
environment:
OMP_NUM_THREADS: 16
HUGGINGFACEHUB_API_TOKEN: ${HUGGINGFACEHUB_API_TOKEN}
HF_HUB_DISABLE_PROGRESS_BARS: 1
HF_HUB_ENABLE_HF_TRANSFER: 0
HF_ENDPOINT: https://hf-mirror.com
command: --model-id ${RERANK_MODEL_ID} --auto-truncate
vllm-service:
image: cr-for-diting-nodelete-cn-beijing.cr.volces.com/practice-images/vllm-xft:0.5.5.0
container_name: vllm-service
ports:
- "9009:80"
volumes:
- "/data00/models:/data00/models"
shm_size: 1g
environment:
MODEL_PATH: "/data00/models"
PORT: 80
MODEL_NAME: DeepSeek-R1-Distill-Qwen-7B
healthcheck:
test: ["CMD-SHELL", "curl -f http://localhost:80/version || exit 1"]
interval: 10s
timeout: 10s
retries: 100
command: sh -c "numactl -C $(lscpu -p=CPU,CORE,NODE|grep -v '^#'|awk -F, '$3 == 0 && !seen[$2]++ {print $1}' | paste -sd \",\" -) python -m vllm.entrypoints.openai.api_server --model ${LLM_MODEL_ID} --tokenizer ${LLM_MODEL_ID} --dtype bf16 --kv-cache-dtype fp16 --served-model-name ${LLM_MODEL_ID} --port 80 --trust-remote-code"
chatqna-xeon-backend-server:
image: cr-for-diting-nodelete-cn-beijing.cr.volces.com/practice-images/chatqna:${TAG:-latest}
container_name: chatqna-xeon-backend-server
depends_on:
redis-vector-db:
condition: service_healthy
tei-embedding-service:
condition: service_started
retriever:
condition: service_started
tei-reranking-service:
condition: service_started
vllm-service:
condition: service_healthy
ports:
- "8888:8888"
environment:
- MEGA_SERVICE_HOST_IP=chatqna-xeon-backend-server
- EMBEDDING_SERVER_HOST_IP=tei-embedding-service
- EMBEDDING_SERVER_PORT=${EMBEDDING_SERVER_PORT:-80}
- RETRIEVER_SERVICE_HOST_IP=retriever
- RERANK_SERVER_HOST_IP=tei-reranking-service
- RERANK_SERVER_PORT=${RERANK_SERVER_PORT:-80}
- LLM_SERVER_HOST_IP=vllm-service
- LLM_SERVER_PORT=${LLM_SERVER_PORT:-80}
- LLM_MODEL=${LLM_MODEL_ID}
- LOGFLAG=${LOGFLAG}
ipc: host
restart: always
chatqna-xeon-ui-server:
image: cr-for-diting-nodelete-cn-beijing.cr.volces.com/practice-images/chatqna-ui:${TAG:-latest}
container_name: chatqna-xeon-ui-server
depends_on:
- chatqna-xeon-backend-server
ports:
- "5173:5173"
ipc: host
restart: always
chatqna-xeon-nginx-server:
image: cr-for-diting-nodelete-cn-beijing.cr.volces.com/practice-images/nginx:${TAG:-latest}
container_name: chatqna-xeon-nginx-server
depends_on:
- chatqna-xeon-backend-server
- chatqna-xeon-ui-server
ports:
- "${NGINX_PORT:-80}:80"
environment:
- FRONTEND_SERVICE_IP=chatqna-xeon-ui-server
- FRONTEND_SERVICE_PORT=5173
- BACKEND_SERVICE_NAME=chatqna
- BACKEND_SERVICE_IP=chatqna-xeon-backend-server
- BACKEND_SERVICE_PORT=8888
- DATAPREP_SERVICE_IP=dataprep-redis-service
- DATAPREP_SERVICE_PORT=5000
ipc: host
restart: always

networks:
default:
driver: bridge

Press the Esc key to exit the edit mode, enter :wq, and then press Enter to save and exit the file.
In the opea_chatqna directory, execute the following command to start the Docker container.

echo "TAG=1.2
EMBEDDING_MODEL_ID=\"/data/bge-base-zh-v1.5\"
RERANK_MODEL_ID=\"/data/bge-reranker-base\"
LLM_MODEL_ID=\"/data00/models/DeepSeek-R1-Distill-Qwen-7B\"
INDEX_NAME=\"rag-redis\"
HUGGINGFACEHUB_API_TOKEN=\"YourOwnToken\"
LOGFLAG=\"\"" > .env
mkdir weights
cd weights
export BUCKET=iaas-public-model-cn-beijing
if [ ! -d "bge-base-zh-v1.5" ]; then
oniond download model bge-base-zh-v1.5
fi
if [ ! -d "bge-reranker-base" ]; then
oniond download model bge-reranker-base
fi
mkdir -p /data00/models
if [[ ! -d "/data00/models/DeepSeek-R1-Distill-Qwen-7B" ]]; then
pushd /data00/models
oniond download model DeepSeek-R1-Distill-Qwen-7B
python3 -c 'import xfastertransformer as xft; xft.Qwen2Convert().convert("/data00/models/DeepSeek-R1-Distill-Qwen-7B","/data00/models/DeepSeek-R1-Distill-Qwen-7B")'
popd
fi
cd ..
docker compose --compatibility up -d

Start and run the demo that is pre-installed in the Docker container and used to demonstrate the two core operations of Embedding and Rerank.

Click here to download the test PDF file and upload the test PDF file to the target instance.

In the directory where the nke-10k-2023.pdf file is located, execute the following command to upload the nke-10k-2023.pdf file to the dataprep service.

curl -X POST -H "Content-Type: multipart/form-data" -F "files=@./nke-10k-2023.pdf" http://localhost:6007/v1/dataprep/ingest

If the following output is displayed, the operation is successful.

{"status":200,"message":"Data preparation succeeded"}

Create a benchmark test file.

Execute the following command to create the benchmark.yaml file.

vim benchmark.yaml

Press the i key to enter edit mode, and then copy the following content into the file.

*Script content *tip *You can modify the following parameters based on your needs: *user_queries: Used to specify the concurrency. *llm_model: Used to specify the LLM model path. * *
Script content

Press the Esc key to exit the edit mode, enter :wq, and then press Enter to save and exit the file.

Execute the following command to start the performance testing demo.

China (Beijing)
China (Shanghai)
China (Guangzhou)
Asia Pacific (Johor)

TOS_BUCKET="https://iaas-debug-beijing.tos-cn-beijing.ivolces.com"
if [ ! -d 'GenAIEval' ]; then
wget $TOS_BUCKET/tool/cpu-serving/GenAIEval.tar
tar -xf GenAIEval.tar
rm -f GenAIEval.tar
fi
cp benchmark.yaml GenAIEval/evals/benchmark
cd GenAIEval/evals/benchmark
python3 benchmark.py

Execute the following command to obtain performance data.

cat /root/virt-benchmark-dev/ubuntu24.04/bin/rag/ragbench_out/chatqnafixed_20250730_173320/1_metrics.json

Sample output:

{
"chatqna-xeon-backend-server_average_latency_per_request": 13.670678080748985,
"tei-embedding-server_average_latency_per_request": 0.029958314499999972,
"vllm-service_average_latency_per_request": 13.525004148483276,
"tei-reranking-server_average_latency_per_request": 0.10077269599999998,
"retriever-redis-server_average_latency_per_request": 0.002257339499919908
}

Performance data

This section presents the duration data for Embedding and Rerank operations before and after enabling AMX acceleration, by using an ECS instance of the ecs.g4i.16xlarge type.

Request count	Embedding duration (ms)			Rerank duration (ms)
Request count		Not accelerated	AMX acceleration	Not accelerated	AMX acceleration
1	314.25	41.19	345.42	106.23
2	312.39	31.87	335.5	111.36
4	305.6	32.94	366.95	108.77
8	377.13	52.33	383.89	159.54
16	721.81	65.13	586.86	233.23
32	1909.94	104.76	882.99	431.02
64	4600.33	169.6	915.75	884.31
128	9475.15	228.16	912.41	1913.32

Vector database

Install an optimized vector database

Purchase an ecs.g4i instance. In the example provided in this topic, the following instance configurations are used.

Instance type: ecs.g4i.4xlarge

Image: Ubuntu 24.04 64 bit

Network: an assigned EIP

Log in to the target instance.

Execute the following command to install dependencies.

apt update
apt install docker.io

Execute the following command to pull the vector database image optimized for the ecs.g4i instance type.

docker pull cr-for-diting-nodelete-cn-beijing.cr.volces.com/practice-images/vectordb:amx

Execute the following command to start the Docker image.

docker run -itd --name vectordb --privileged --network host cr-for-diting-nodelete-cn-beijing.cr.volces.com/practice-images/vectordb:amx bash

Execute the following commands to enter the container.

docker exec -it vectordb bash

Test operator performance using ann-benchmark.

Execute the following command to download test resources.

cd ~/ann-benchmarks/data
wget http://ann-benchmarks.com/nytimes-256-angular.hdf5

Execute the following command to run the test.

conda activate hnswlib && cd /root && bash bench-hnswlib.sh
conda activate faiss && cd /root && bash bench-faiss.sh

Performance data

This section presents data on the number of query requests a vector database can process per second before and after enabling AMX acceleration, by using an ECS instance of the ecs.g4i.4xlarge type.

Metric	Not accelerated	AMX acceleration
hnswlib Build an index using the Hierarchical Navigable Small World (HNSW) algorithm.
qps-recall-95 The number of query requests that can be processed per second at a 95% recall rate.	12921.64	18725.88
qps-recall-99 The number of query requests that can be processed per second at a 99% recall rate.	3177.78	4974.79
faiss-indexflatip Build an index using the Facebook AI Similarity Search (FAISS) algorithm.
qps The number of query requests that can be processed per second at a 100% recall rate.	6296.99	12285.91

Test conclusion

When you deploy the RAG architecture on General-purpose g4i instances equipped with the 6th-generation Intel® Xeon® Scalable processor (Granite Rapids) and enable the Advanced Matrix Extensions (AMX) instruction set, all core components of the RAG architecture achieve significant performance improvements. In scenarios where cost-effectiveness and abundant instance resources are prioritized, you can use instance types equipped with the 6th-generation Intel® Xeon® Scalable processor (Granite Rapids), such as the General-purpose g4i instances, as an alternative to GPU instances. This way, you can obtain the most cost-effective RAG inference solution.

Elastic Compute Service