Volcengine Kubernetes Engine allows static IPs for pods to address requirements such as IP-based access control, service registration, service discovery, and audit logging. This topic describes how to configure static IPs for pods.
Background
Volcengine Kubernetes Engine allows the Trunk ENI mode enabled when creating clusters with the VPC-CNI network model. The pods running on a Trunk ENI node can have ENI (elastic network interface) of their own with the specified subnets and security groups.
A pod can be assigned a static IP if it has a dedicated network interface. It facilitates configurations such as access control, service registration, service discovery, and audit logging based on the IP of a pod. Here are the CRD objects to be used when configuring static IPs:
- VPCENIDefinition object: This object defines the VPC IPAM configuration of the ENI, including VPC ID, IP allocation mode, security groups, and subnet ID.
- PodNetworkDefinition object: This object is used to define the network configuration of a pod, including IPAM and Pod selector.
Usage limits
- Trunk ENI is applicable only to clusters using the VPC-CNI network model.
- The pod name must not exceed 63 characters.
Prerequisites
- The vpc-cni add-on has been installed and upgraded to v1.6.0 or later.
- The vpc-cni-controlplane add-on has been installed and upgraded to v1.6.0 or later.
Procedure
Step 1: Enable Trunk ENI
Enable when creating clusters
Enable for existing clusters
tip
This topic only covers the key configurations for enabling Trunk ENI when creating a cluster, including installing the vpc-cni-controlplane add-on. For more details about creating a cluster, see Creating clusters. - On the cluster management page, select Cluster in the left navigation bar.
- Click Create cluster in the upper left corner to go to the Cluster configuration page. Finish the configurations on the page and click Next: Node pool configuration in the lower-right corner.
- The key parameters are as follows. Other parameters can be configured as needed.
- Finish the necessary configurations on the Node pool configuration page, and click Next: Addon configuration in the lower right corner.
- The vpc-cni-controlplane add-on is selected by default on the page. You can select more add-ons if necessary and then click Next: Configuration confirmation to finish.
- After confirming that the cluster information and billing details are correct, read the Volcengine Kubernetes Engine Specific Terms of Service, select the checkbox to accept them, and then click Confirm to start creating the cluster.
tip
To enable Trunk ENI in an existing cluster, ensure that the cluster meets the following conditions:
- When creating a cluster, set the network mode to Shared ENI.
- The vpc-cni-controlplane add-on has been installed and upgraded to v1.4.5 or later. For details, see Installing add-ons.
- In the left navigation bar, click Cluster. In the right part of the page, click the target cluster name to open the management page of this cluster.
- In the left navigation bar, click Overview, and then select the Overview tab on the right page.
- In Cluster network configuration section, check Enable Trunk ENI. In the pop-up window, click Confirm.
- In the left navigation bar of the cluster management page, click Addons. On the page that appears, click the Network tab.
- Locate the vpc-cni add-on, hover the mouse over it, and click Configuration.
- There is no need to make any changes to the add-on configuration. Click Confirm.
- If the vpc-cni add-on is v1.9.6 or later, the system will automatically restart the add-on after you click Confirm to apply the configuration.
- If the vpc-cni add-on version is earlier than v1.9.6, you need to connect to the cluster using kubectl and execute the following command to perform a rolling restart of the Cello pods to apply the configuration.
warning
To ensure business stability, we recommend performing the above add-on restart operation during low-peak hours.
Step 2: Enable static IP for pods
- Click the cluster name you just created to enter the cluster management page.
- Select Addons in the left navigation bar.
- On the Network tab, click Configuration on the vpc-cni-controlplane add-on and enable Static IP for pods in the pop-up window.
- Click Confirm to finish.
tip
If this option does not appear in your cluster or is grayed out and cannot be configured, it indicates that Trunk ENI is not enabled in the cluster or the add-on is not at the required version. For more information, see the previous content.
Step 3: Create a VPCENIDefinition object
Create a VPCENIDefinition object. The configuration example and key parameter descriptions are as follows:
The parameters under spec are listed below.
Step 4: Create the PodNetworkDefinition object
Create a PodNetworkDefinition object. The configuration example and key parameter descriptions are as follows:
The parameters under spec are listed below.
warning
- podSelector and namespaceSelector are optional parameters that can be configured as needed. At least one of them must be configured so that the feature can take effect.
- The selectors is a list that may contain multiple namespaces and pod labels. Each individual member in the list can have labels for both namespaces and pods simultaneously.
- The selectors contained in each member apply AND logically with each other. It means only the pod with labels that match the two selectors simultaneously can be assigned with an ENI and a static IP if both selectors are configured.
- The members in the list apply OR logically with each other if there is more than one member in the list. It means a pod that matches the selectors of any member in the list can be assigned an ENI and a static IP.
Step 5: Configure an ENI dedicated to a pod
There are several methods to do it:
- Namespace label: Use namespace labels to assign ENIs to all the pods under the namespaces.
- Pod label: Use pod labels to assign ENIs to specific pods.
tip
You can use either one label type or both. If both types are used, pods need to match them simultaneously to get ENIs.
Configuring namespace labels
Creating pods and adding labels
Creating workloads and adding labels
You can add the label defined in the PodNetworkDefinition object (e.g., foo: bar) when creating a namespace, so that all the pods in that namespace can have dedicated ENIs The example is as follows:
You can add the label defined in the PodNetworkDefinition object (e.g., foo: bar) when creating a pod, so that the pod has a dedicated ENI. The example is as follows:
You can add the label defined in the PodNetworkDefinition object (e.g., foo: bar) when creating a pod, so that the pod has a dedicated ENI. The example is as follows:
Validating results
When the configurations are saved, the system adds necessary annotations (including ENI request/limit) to pods matching the labels in the PodNetworkDefinition object. It then relocates these pods to Trunk-enabled nodes and provides them with dedicated ENIs.
tip
For pods running with static IPs, the AZs (availability zones) of ENIs are determined when pods are first created. If pods are rebuilt later, they will automatically relocate to the AZs of the nodes where the original pods were first created. This achieves automatic affinity.
As shown in the following figure, automatically added annotations (such as k8s.volcengine.com/eni-ip-status) indicate that the pod has been configured with a dedicated ENI.
The information about ENIs can be found on the Network interface tab of the Pod Details page.
After you delete and recreate the Pod, the Pod IP address remains unchanged, indicating that the static IP has been successfully configured for the Pod.