Docs

Elastic Compute Service

AIGC practices

Building system environment based on RAG with Intel® TDX for text generation inference

This topic describes how to build a Retrieval-Augmented Generation (RAG) framework and deploy an inference model on an instance with Intel® TDX capabilities. In this solution, a hardware-level trusted confidential environment is provided for inference data by using TDX, and the answers provided by the inference model are enriched by using RAG.

Background

Intel® TDX

Intel® Trust Domain Extensions (Intel® TDX) is a technology introduced by Intel to enhance the security of virtual machine environments. TDX leverages hardware-based features to establish an isolated trusted execution environment for virtual machines at the physical hardware layer. This delivers enhanced security protection to virtual machines and ensures sensitive data and critical applications are shielded from potential security threats within complex cloud computing environments. Learn more.

RAG

RAG is a technology that integrates information retrieval and text generation. It enhances the accuracy and reliability of generative AI models, such as large language models (LLMs), while mitigating model hallucinations.

Information retrieval: Before generating an answer, RAG uses a retrieval system to search extensive text datasets, including local documents, Internet sources, and enterprise databases, for information relevant to the input question or prompt.

Text generation: After obtaining the relevant retrieval results, RAG combines the retrieved document fragments with the user question as input to a generative AI model (for example, an LLM). The model incorporates this retrieved information as supplementary context, processes it alongside the input question or prompt, and then generates the corresponding answer or content.

Llama-2-7b-chat-hf model

Llama 2 is an open-source suite of pretrained and fine-tuned generative text models developed by Meta. In this practice, we use the Llama-2-7b-chat-hf model, a variant within the Llama 2 suite optimized specifically for dialogue tasks. This model enhances security and conversational fluency by using two key techniques: supervised fine-tuning (SFT) and reinforcement learning from human feedback (RLHF). These improvements make it well-suited for interactive scenarios such as customer service chatbots and virtual assistants. For more information, visit meta-llama/Llama-2-7b-chat-hf.

Procedure

Step 1: Prepare the environment

Create an ECS instance with Intel® TDX capabilities. For more information, see Purchasing ECS instances.

When you create an instance, take note of the following configurations:

Instance type: In the example provided in this topic, the Security Enhanced General-purpose g3ilt (ecs.g3ilt.8xlarge) type is used.

Volume: The recommended system volume capacity is greater than or equal to 100 GiB.

Image: In the example provided in this topic, an Ubuntu 24.04 64 bit image is used.

Network: You need to associate an EIP with the instance. For more information, see Associating EIPs.

Modify the security group configuration.

Communication with the inference model requires port 8502, which must be allowed in the inbound rules of the security groups associated with the ECS instance.

In the top menu bar, select the region and project where the target instance resides.

Click the name of the target instance to enter the instance details page.

Select the "Security Group" tab.

Click the "Configure rule" button.

Add an inbound rule as shown in the following figure, and then click the "Confirm" button.

Install required tools and software for the target instance.

Execute the following command to update the APT package repository.

apt update

Execute the following command to install the required tools and software.

apt install docker.io git cryptsetup -y

Execute the following command to download the confidential-computing-zoo code.

git clone https://github.com/intel/confidential-computing-zoo.git

tip

If your instance cannot access Git, you can also download confidential-computing-zoo and then upload it to the target instance. After you upload the code, execute the unzip confidential-computing-zoo-main.zip command to decompress the package, and then execute the mv confidential-computing-zoo-main confidential-computing-zoo command to rename the directory.

(Optional) Create an image proxy.

tip

In this practice, the Container Registry service provided by Volcano Engine and a remote proxy repository are used to enhance the efficiency of cross-border image pulling. If you can pull Docker images across borders by using other methods, you may skip this step.

In the directory tree on the left, select "Instance list > Proxy cache".

Select "Create Micro Edition".

Click the "Create proxy cache registry" button and configure the following information.

tip

This practice only describes necessary parameters. Retain the default values for parameters not mentioned in this practice. For more information, see Create proxy cache registry.

Parameter	Description	Sample value
Domain name	The domain name you want to specify for the proxy repository. In the example provided in this topic, the domain name is set to tdxdocker.	tdxdocker
Billing type	Select Pay-as-you-go: Postpayment mode. You can start and stop the resource at any time, with charges based on your actual usage duration. For more information, see Billing.	Pay-as-you-go
Specifications	Select Micro Edition.	Micro Edition

View the comparison of instance types and the fees. Then, click the "Confirm order" button.

Carefully read and select "I have read and agree to Container Registry Product Terms of Service" and then click the "Confirm" button to submit the order and start creating the remote proxy repository.

After the remote proxy repository is created, record the domain name of this remote proxy repository.

Click the name of the proxy repository you created to go to the details page.

In the directory tree on the left, select "Namespace".

Click the "Create namespace" button and configure the following information.

tip

This practice only describes necessary parameters. Retain the default values for parameters not mentioned in this practice. For more information, see Create a namespace.

Parameter	Description	Sample value
Name	The name you want to specify for the namespace. In the example provided in this topic, the name is set to library.	library
Default repository type	Set the default type of artifact repositories in this namespace to Public.	Public

Click "Confirm" to complete the operation.

Step 2: Build an image

You can use any method to build a Docker image for running the inference model based on your business requirements.

tip

If your ECS instance cannot access Git, use the official Docker image provided by Volcano Engine.

Use the official Docker image

Build your own Docker image

Download the official Docker image provided by Volcano Engine.

Execute the following command to download the backend image.

docker pull cr-for-diting-nodelete-cn-beijing.cr.volces.com/practice-images/ai:tdx-rag-backend

Execute the following command to download the frontend image.

docker pull cr-for-diting-nodelete-cn-beijing.cr.volces.com/practice-images/ai:tdx-rag-frontend

Modify the image name and tags.

Execute the following command to modify the backend image name and tags.

docker tag cr-for-diting-nodelete-cn-beijing.cr.volces.com/practice-images/ai:tdx-rag-backend intelcczoo/tdx-rag:backend

Execute the following command to modify the frontend image name and tags.

docker tag cr-for-diting-nodelete-cn-beijing.cr.volces.com/practice-images/ai:tdx-rag-frontend intelcczoo/tdx-rag:frontend

Execute the following command to delete unnecessary image tags.

docker rmi cr-for-diting-nodelete-cn-beijing.cr.volces.com/practice-images/ai:tdx-rag-backend
docker rmi cr-for-diting-nodelete-cn-beijing.cr.volces.com/practice-images/ai:tdx-rag-frontend

Execute the following command to enter the confidential-computing-zoo directory.

cd confidential-computing-zoo/cczoo/rag

tip

Replace confidential-computing-zoo/cczoo/rag with the actual download path of confidential-computing-zoo.

Modify the confidential-computing-zoo configuration file.

Modify the backend/configs/etc/sgx_default_qcnl.conf file

Execute the following command to open the backend/configs/etc/sgx_default_qcnl.conf file.

vim backend/configs/etc/sgx_default_qcnl.conf

Press the i key to enter edit mode. Modify the following parameter values:

PCCS_URL: https://tdx-dcap-server-cn-beijing.ecs.volces.com/sgx/certification/v4/

USE_SECURE_CERT: TRUE

Example after modification:

Press the Esc key to exit edit mode, enter :wq, and then press Enter to save and exit the file.

Modify the frontend/chatbot-rag/ra_configs/etc/sgx_default_qcnl.conf file.

Execute the following command to open the frontend/chatbot-rag/ra_configs/etc/sgx_default_qcnl.conf file.

vim frontend/chatbot-rag/ra_configs/etc/sgx_default_qcnl.conf

Press the i key to enter edit mode. Modify the following parameter values:

PCCS_URL: https://tdx-dcap-server-cn-beijing.ecs.volces.com/sgx/certification/v4/

USE_SECURE_CERT: TRUE

Example after modification:

Press the Esc key to exit edit mode, enter :wq, and then press Enter to save and exit the file.

Modify the Dockerfile.

Execute the following command to delete existing files.

rm /root/confidential-computing-zoo/cczoo/rag/frontend/chatbot-rag/frontend.dockerfile /root/confidential-computing-zoo/cczoo/rag/backend/backend.dockerfile

Download the following files and upload them to the target instance. Learn more.

frontend.dockerfile

backend.dockerfile

Execute the following command to move the downloaded Dockerfile to the appropriate directory.

mv frontend.dockerfile /root/confidential-computing-zoo/cczoo/rag/frontend/chatbot-rag/
mv backend.dockerfile /root/confidential-computing-zoo/cczoo/rag/backend/

(Optional) Modify proxy configuration.

If you use the Container Registry service provided by Volcano Engine, and pull Docker images by using a remote proxy repository, you need to modify the proxy configuration by performing the following steps. Otherwise, skip this step.

Modify the backend.dockerfile file

Execute the following command to open the backend.dockerfile file.

vim /root/confidential-computing-zoo/cczoo/rag/backend/backend.dockerfile

Press the i key to enter edit mode. Change the value of the YOUR_IMAGE_PROXY parameter to the domain name for accessing the remote proxy repository you create.

Example before modification:

Example after modification:

Press the Esc key to exit edit mode, enter :wq, and then press Enter to save and exit the file.

Modify the frontend.dockerfile file

Execute the following command to open the frontend.dockerfile file.

vim /root/confidential-computing-zoo/cczoo/rag/frontend/chatbot-rag/frontend.dockerfile

Press the i key to enter edit mode. Change the value of the YOUR_IMAGE_PROXY parameter to the domain name for accessing the remote proxy repository you create.

Example before modification:

Example after modification:

Press the Esc key to exit edit mode, enter :wq, and then press Enter to save and exit the file.

Build an image.

Execute the following command to download Grpc.

git clone --recurse-submodules -b v1.38.1 https://github.com/grpc/grpc

tip

In the example provided in this topic, Grpc V1.38.1 is used. If your instance cannot access Git, you can download Grpc and then upload it to the target instance. Learn more.

After you upload Grpc, execute the tar -xzvf grpc-1.38.1.tar.gz command to decompress the package, and then execute the mv grpc-1.38.1 grpc command to rename the directory.

Execute the following command to download the cmake-linux.sh script.

wget -q -O cmake-linux.sh https://github.com/Kitware/CMake/releases/download/v3.19.6/cmake-3.19.6-Linux-x86_64.sh

tip

In the example provided in this topic, CMake script V3.19.6 is used. If your instance cannot access Git, you can download the CMake script and upload it to the target instance. Learn more.

After you upload the CMake script, execute the mv cmake-3.19.6-Linux-x86_64.sh cmake-linux.sh command to rename the script.

Execute the following command to move Grpc to the specified directory.

cp -r /root/grpc /root/confidential-computing-zoo/cczoo/rag/backend/
mv /root/grpc /root/confidential-computing-zoo/cczoo/rag/frontend/

Execute the following command to delete the existing confidential-computing-zoo directory.

rm -rf /root/confidential-computing-zoo/cczoo/rag/frontend/confidential-computing-zoo /root/confidential-computing-zoo/cczoo/rag/backend/confidential-computing-zoo

Execute the following command to copy confidential-computing-zoo to the specified directory.

cp -r confidential-computing-zoo /root/confidential-computing-zoo-cp
cp -r /root/confidential-computing-zoo-cp /root/confidential-computing-zoo/cczoo/rag/backend/confidential-computing-zoo
mv /root/confidential-computing-zoo-cp /root/confidential-computing-zoo/cczoo/rag/frontend/confidential-computing-zoo

Execute the following command to move the CMake script to the specified directory.

cp /root/cmake-linux.sh /root/confidential-computing-zoo/cczoo/rag/backend
mv /root/cmake-linux.sh /root/confidential-computing-zoo/cczoo/rag/frontend

Execute the following command to confirm the instance DCAP version.

dpkg -l | grep -E 'tee|tdx'

Sample output:

As shown in the following figure, the DCAP version of the instance used in this topic is 1.22.

Execute the following command to build an image.

cd /root/confidential-computing-zoo/cczoo/rag/
./build-images.sh --dcap-version 1.22

tip

Replace 1.22 with the actual DCAP version.

Step 3: Create an encrypted partition

Execute the following command to enter the luks_tools directory.

cd /root/confidential-computing-zoo/cczoo/rag/luks_tools

Execute the following command to create an encrypted partition.

./create_encrypted_vfs.sh 50G /home/vfs

tip

The encrypted virtual file system created in this topic has a 50 GB capacity, and its mount point is set to /home/vfs. Modify the size and mount point based on your business requirements.

Enter YES, and press Enter to confirm the creation.

Enter the password for accessing the encrypted partition, and press Enter to confirm the password.

Sample output:

Obtain the loop device number from the output, which is LOOP_DEVICE=/dev/loop0 in the example provided in this topic, as shown in the following figure.

Execute the following command to specify the environment variables for the loop device.

export LOOP_DEVICE=/dev/loop0

tip

Replace dev/loop0 with the actual loop device number.

Execute the following command to create a directory for storing encrypted data.

mkdir /home/encrypted_storage

Execute the following command to format the loop device with the ext4 file system.

./mount_encrypted_vfs.sh ${LOOP_DEVICE} format

Enter the password for accessing the encrypted partition as instructed, and then press Enter to confirm the password.

Sample output:

Step 4: Download the Llama-2-7b-chat-hf model

Obtain download permission.

Visit the official Hugging Face page for the meta-llama/Llama-2-7b-chat-hf model.

Read the usage license agreement of the model, fill in the required information, and click the "Submit" button to submit your application.

After your application is approved, log in to the Hugging Face Token page to obtain tokens with download permissions. For more information, see User access tokens.

Execute the following command to enter the encrypted partition.

cd /home/encrypted_storage

Execute the following command to install the Hugging Face download tool.

pip install -U huggingface_hub --break-system-packages

Execute the following command to configure environment variables.

export HF_ENDPOINT=https://hf-mirror.com

tip

In this topic, the model is downloaded from the Hugging Face mirror site. You can also go to the Hugging Face official website to download the model.

Download the model file.

Execute the following command to download the Llama-2-7b-chat-hf model.

huggingface-cli download --resume-download --local-dir-use-symlinks False meta-llama/Llama-2-7b-chat-hf --local-dir Llama-2-7b-chat-hf --token <your_token>

tip

Replace <your_token> in the command with the token obtained in the previous step.

Execute the following command to download the ms-marco-MiniLM-L-12-v2 model.

huggingface-cli download --resume-download --local-dir-use-symlinks False cross-encoder/ms-marco-MiniLM-L-12-v2 --local-dir ms-marco-MiniLM-L-12-v2

Execute the following command to download the dpr-ctx_encoder-single-nq-base model.

huggingface-cli download --resume-download --local-dir-use-symlinks False facebook/dpr-ctx_encoder-single-nq-base --local-dir dpr-ctx_encoder-single-nq-base

Execute the following command to download the dpr-question_encoder-single-nq-base model.

huggingface-cli download --resume-download --local-dir-use-symlinks False facebook/dpr-question_encoder-single-nq-base --local-dir dpr-question_encoder-single-nq-base

Step 5: Configure the database

Execute the following command to pull the official MySQL database image.

docker pull <your_remote_proxy>/library/mysql

tip

Replace <your_remote_proxy> with the domain name for accessing the remote proxy repository.

If you do not use the Volcano Engine remote proxy repository, execute the docker pull mysql command to pull the image.

(Optional) Modify the startup script.

If you use the Container Registry service provided by Volcano Engine, and pull Docker images by using a remote proxy repository, you need to modify the proxy configuration by performing the following steps. Otherwise, skip this step.

Execute the following command to enter the rag directory.

cd /root/confidential-computing-zoo/cczoo/rag

tip

Replace /root/confidential-computing-zoo/cczoo/rag with the actual path.

Execute the following command to open the startup script.

vim run.sh

Press the i key to enter edit mode, and change mysql:latest to <your_remote_proxy>/library/mysql:latest.

tip

Replace <your_remote_proxy> with the domain name for accessing the remote proxy repository.

Press the Esc key to exit edit mode, enter :wq, and then press Enter to save and exit the file.

Step 6: Run the Llama-2-7b-chat-hf model

Start the backend inference service.

Execute the following command to enter the rag directory.

cd /root/confidential-computing-zoo/cczoo/rag

tip

Replace /root/confidential-computing-zoo/cczoo/rag with the actual path.

Execute the following command to start the inference backend service.

./run.sh db && ./run.sh backend ra 127.0.0.1

Enter 127.0.0.1 and press the Enter key.

Enter the MySQL database username and password you configured and press Enter. The default username is root and the default password is 123456.

Start the inference frontend service.

Keep the terminal that is running the backend service open. In the browser, open a new page and log in to the target instance again.

In a new terminal, execute the following command to enter the rag directory.

cd /root/confidential-computing-zoo/cczoo/rag

In a new terminal, execute the following command to start the inference frontend service.

./run.sh frontend ra 127.0.0.1

After both the frontend and backend inference services are started, press ctrl +c in the terminals where the frontend and backend services are running, respectively, to stop service running.

Frontend service startup completed

Backend service startup completed

On any terminal, configure the dynamic_config.json file by performing the following steps.

Configure the frontend dynamic_config.json file.

Execute the following command to obtain the parameter values.

docker exec -it tdx_rag_backend bash -c "cd /usr/bin && ./tdx_report_parser"

Sample output:

Record the value of each parameter in the output.

Execute the following command to open the frontend dynamic_config.json file.

vim /root/confidential-computing-zoo/cczoo/rag/frontend/chatbot-rag/dynamic_config.json

Press the i key to enter edit mode, and fill in the parameter values obtained in Step 1 into their respective positions.

Example before configuration:

Example after configuration:

Press the Esc key to exit edit mode, enter :wq, and then press Enter to save the changes and exit the file.

Configure the backend dynamic_config.json file.

Execute the following command to obtain the parameter values.

docker exec -it tdx_rag_frontend bash -c "cd /usr/bin && ./tdx_report_parser"

Sample output:

Record the value of each parameter in the output.

Execute the following command to open the frontend dynamic_config.json file.

vim /root/confidential-computing-zoo/cczoo/rag/backend/pipelines/dynamic_config.json

Press the i key to enter edit mode, and fill in the parameter values obtained in Step 1 into their respective positions.

Example before configuration:

Example after configuration:

Press the Esc key to exit edit mode, enter :wq, and then press Enter to save the changes and exit the file.

Follow the previous steps 2 and 3 to start the inference backend and frontend services in sequence, and then retrieve the Internet access address for the inference page (corresponding to the External URL) from the terminal running the frontend inference service.

On any local PC with access to the Internet, you can interact with the AI by visiting the Internet access address of the inference page in your browser.

If the web page displays the information shown in the red box in the following figure, a secure connection has been established between the frontend and backend inference services based on Intel® TDX.

Last updated: 2026.06.18 17:48:26