In large-scale scenarios, dedicated ENIs (elastic network interfaces) improve the network performance between pods and let you bind a separate security group to to the ENI of each pod to meet pod-level network isolation requirements. This topic describes how to configure a dedicated ENI for a pod.
Background
The VKE supports enabling Trunk ENI mode in clusters using the VPC-CNI network model. On Trunk ENI nodes, exclusive ENI resources are provided for pods. In addition, subnets and security groups can be specified for pods.
In the scenarios of large-scale applications, the dedicated ENI (elastic network interface) can provide better communication performance between pods. Meanwhile, it allows each pod to be bound to a security group, satisfying the need for network isolation between pods. Trunk ENI has the following features:
- It can be exclusively assigned to a pod, providing security-group-based traffic isolation at the network-interface level.
- It allows IPs from custom subnets assigned to pods, meeting the requirement for fine-grained traffic isolation.
The following CRD objects are involved when configuring dedicated ENI for pods:
- VPCENIDefinition object: Used to define the VPC IPAM configuration of the ENI. including VPC ID, IP allocation mode, security groups, and subnet ID.
- PodNetworkDefinition object: Used to define the network configuration of the pod, including the information of IPAM and pod selection.
Usage limits
- Only supported in clusters using the VPC-CNI network model.
- Once Trunk ENI is enabled, each node reserves one ENI as the Trunk interface. As a result, the number of shared IPs available to pods is reduced accordingly.
- With Trunk ENI, network interfaces cannot be pre-allocated. As a result, creating and deleting network resources takes longer than when using shared IPs.
- The pod name can be up to 63 characters in length.
Procedure
Step 1: Enable Trunk ENI
Enable when creating clusters
Enable for the existing clusters
tip
This topic only covers the key configurations for enabling Trunk ENI when creating a cluster, including installing the vpc-cni-controlplane add-on. For more details about creating a cluster, see Creating clusters. - On the cluster management page, select Cluster in the left navigation bar.
- Click Create cluster in the upper left corner to go to the Cluster configuration page. Finish the configurations on the page and click Next: Node pool configuration in the lower-right corner.
- The key parameters are as follows. Configure other parameters as needed.
- Finish the necessary configurations on the Node pool configuration page, and click Next: Addon configuration in the lower right corner.
- The vpc-cni-controlplane add-on is selected by default on the page. You can select more add-ons if necessary and then click Next: Configuration confirmation to finish.
- Ensure that all cluster information and billing details are correct. Read the "VKE Dedicated Service Terms" and click the checkbox. Then click Confirm on the page to start creating the cluster.
tip
To enable Trunk ENI in an existing cluster, ensure the cluster meets the following conditions:
- When creating a cluster, set the network mode to Shared ENI.
- The vpc-cni-controlplane addon has been installed and upgraded to v1.4.5 or later. For details, see Installing add-ons.
- In the left navigation bar, click Cluster. On the right page, click the target cluster name to open the management page of this cluster.
- Click Overview in the left navigation bar, and then select the Overview tab on the right page.
- In Cluster network configuration section, check Enable Trunk ENI. In the pop-up window, click Confirm.
- In the left navigation bar of the cluster management page, click Addons. On the page that appears, click the Network tab.
- Locate the vpc-cni add-on, hover the mouse over it, and click Configuration.
- There is no need to make any changes to the add-on configuration. Click Confirm.
- If the vpc-cni add-on is v1.9.6 or later, the system will automatically restart the add-on after you click Confirm to apply the configuration.
- If the vpc-cni add-on version is earlier than v1.9.6, you need to connect to the cluster using kubectl and execute the following command to perform a rolling restart of the Cello pods to apply the configuration.
warning
To ensure business stability, we recommend performing the above add-on restart operation during low-peak hours.
Step 2: Create a VPCENIDefinition object
Create a VPCENIDefinition object. The configuration example and key parameter description are as follows:
The parameters under spec are listed below.
Step 3: Create a PodNetworkDefinition object
Create a PodNetworkDefinition object. Configuration examples and key parameter descriptions are as follows:
The parameters under spec are listed below.
warning
- podSelector and namespaceSelector are optional parameters that can be configured as needed. At least one of them must be configured so that the feature can take effect.
- The selectors is a list that may contain multiple namespaces and pod labels. Each individual member in the list can have labels for both namespaces and pods simultaneously.
- Within each list member, the selectors are combined using AND logic. If both a namespace selector and a pod selector are configured, an ENI is assigned to a pod only when both the pod labels and the labels of its namespace match the corresponding selectors.
- If the list contains multiple members, the members are combined using OR logic. A pod that matches the selectors of any member in the list is assigned an ENI.
Step 4: Configure a dedicated ENI for a pod
There are several methods to do it:
- Namespace label: Use namespace labels to assign ENIs to all the pods under the namespaces.
- Pod label: Use pod labels to assign ENIs to specific pods.
Configuring namespace labels
Creating pods and adding labels
Creating workloads and adding labels
You can add the label defined in the PodNetworkDefinition object (e.g., foo: bar) when creating a namespace, so that all the pods in that namespace can have dedicated ENIs The example is as follows:
You can add the label defined in the PodNetworkDefinition object (e.g., foo: bar) when creating a pod, so that the pod has a dedicated ENI. The example is as follows:
You can add the label defined in the PodNetworkDefinition object (e.g., foo: bar) when creating a pod, so that the pod has a dedicated ENI. Here is an example of stateless workload:
Validating results
When the configurations are saved, the system adds necessary annotations (including ENI request/limit) to pods matching the labels in the PodNetworkDefinition object. It then relocates these pods to Trunk-enabled nodes and provides them with dedicated ENIs.
As shown in the figure below, the annotation k8s.volcengine.com/network-status added automatically by the system indicates that the pod already has a dedicated ENI.
The information about ENI can be found in the Network interface tab on the pod details page.