The Gateway is primarily used to submit tasks to the compute cluster and provide security isolation. You can use the Gateway node group (software stack version 3.23.0 and above) or EMR-CLI (software stack versions prior to 3.23.0) to customize the deployment of the Gateway environment.

Use Gateway node group #

The software stack version 3.23.0 and above provides the Gateway node group feature. It is recommended to use the Gateway node group feature. For details, see Node group management.

Use EMR-CLI to customize the deployment of the Gateway environment #

Volcano Engine E-MapReduce (EMR) provides the EMR-CLI tool for deploying the Gateway environment based on Volcano Engine ECS. You can create ECS instances as needed and quickly deploy the Gateway environment using EMR-CLI.

Prerequisites #

A cluster of type Hadoop 3x, Hadoop 2x has already been created in EMR, and the compute cluster is running. For details, see Create clusters (3.18.0 and earlier).

Limits #

• This document applies only to Gateway environment deployment for data lake clusters. EMR 3.x version must be greater than or equal to EMR-3.5.0, EMR 2.x version must be greater than or equal to EMR-2.4.0, and Kerberos authentication must not be enabled.
• The ECS instances within the EMR cluster have already integrated the relevant client. Do not install again to avoid affecting the cluster runtime environment.
• When deploying the Gateway client with EMR-CLI, an overwrite installation mode is used. If Gateway has already been deployed on the ECS, redeploying will install a new client in the same directory.
• Currently, the services supported for Gateway deployment using this method include HDFS, YARN, HBASE, MAPREDUCE2, HIVE, SPARK, FLINK, SQOOP, PRESTO, TRINO, TEZ, PHOENIX, DORIS, STARROCKS. Since version 3.8.0, deployment support has been added for the following components: IMPALA, KERBY, HUDI, ICEBERG, DELTALAKE.

First-time deployment of the Gateway environment #

1. Log in to the ECS console and create an instance. For operation details, see Purchase instance .
   The recommended parameters are as follows.

   Parameter	Note
   Region and availability zone	Must be consistent with the region and availability zone of the EMR cluster.
   Image	Use the Velinux operating system (not CentOS-compatible), or CentOS 7.9.
   System disk	Use PL0 or FlexPL cloud disk, with a capacity of no less than 60 GiB.
   Network	Must be consistent with the EMR cluster VPC.
   Security group	Must be consistent with the security group of the EMR cluster's primary instance group to ensure network connectivity between the ECS instance and the EMR cluster.

2. Log in to the newly created ECS instance via SSH. For login methods, see Log in to instance .

3. Run the following command to install EMR-CLI.

   regionId=`curl http://100.96.0.96/latest/region_id`; curl https://emrcli-${regionId}.tos-${regionId}.ivolces.com/emrcli.sh -o emrcli.sh; chmod 755 emrcli.sh; ./emrcli.sh install ${regionId}
   

   Upon successful installation, the following information will be returned.

   install emrcli success
   

4. Optional: Configure the meta.json file
   You can maintain the identity authentication AK and SK required for deploying the client in the meta.json file located in the /usr/local/emrcli/ directory before running commands, to avoid entering them each time.
   The content of the meta.json file is as follows.

   {
       "access-key-id": "xxx",
       "access-key-secret": "xxx"
   }
   

   You need to modify the following parameters according to your actual situation.

   Parameter	Required	Note
   access-key-id	Yes	The Access Key of the Volcano Engine account.
   access-key-secret	Yes	The Access Secret of the Volcano Engine account.

   Run the following command to deploy the EMR Gateway client.

   tip

   With a 10M public bandwidth, this step takes approximately 3 minutes.

   • If the meta.json file is configured:

     emrcli gateway deploy \
       --clusterId <ClusterId> \
       --appNames <ApplicationName>
     

   • If the meta.json file is not configured:

     emrcli gateway deploy \
       --clusterId <ClusterId> \
       --appNames <ApplicationName> \
       --access-key-id <AccessKeyId> \
       --access-key-secret <AccessKeySecret>
     

   You need to modify the following parameters according to your actual situation.

   Parameter	Required	Description
   clusterId	Yes	The cluster ID of an EMR cluster that has already been created.
   appNames	No	Application name. If there are multiple applications, separate them with commas (,), for example: HDFS,YARN. If this parameter is not specified, all supported client applications of the cluster are used by default, such as Hive and HDFS.

   Upon successful deployment, the following information will be returned.

   deploy emr gateway sucess
   

   warning

   After Gateway installation, the JAVA_HOME environment variable will be set to /usr/lib/jvm/java-8-openjdk-velinux-amd64. You can modify it in the /etc/profile.d/jdk.sh file, but changes may affect Gateway functionality, so proceed with caution.

5. Log in again to the ECS instance or run the following command:

   source /etc/profile
   

   To make the system environment variables take effect.

Manage the Gateway environment #

After the Gateway is created, if the computing cluster associated with the Gateway adds new services or changes service configurations, you can use the following commands to update client components or synchronize the latest configuration.

Update client components

When new services are added to the computing cluster, use the EMR CLI to add the corresponding client on the Gateway.
The command is basically the same as the initial deployment; you only need to specify the name of the newly added application in the appNames parameter. Installed applications will not be affected.

• meta.json file configured

  emrcli gateway deploy \
    --clusterId <ClusterId> \
    --appNames <ApplicationName>
  

• meta.json file not configured

  emrcli gateway deploy \
    --clusterId <ClusterId> \
    --appNames <ApplicationName> \
    --access-key-id <AccessKeyId> \
    --access-key-secret <AccessKeySecret>
  

The following information is returned after a successful update.

deploy emr gateway success

Synchronize the modified configurations of the EMR cluster

Use the EMR CLI to synchronize the modified application configuration of the EMR cluster (such as core-site.xml and so on) to the Gateway client.

• meta.json file configured

  emrcli gateway refreshConfigs \
    --clusterId <ClusterId> \
    --appNames <ApplicationName>
  

• app_info.json file not configured

  emrcli gateway refreshConfigs \
    --clusterId <ClusterId> \
    --appNames <ApplicationName> \
    --access-key-id <AccessKeyId> \
    --access-key-secret <AccessKeySecret>
  

The following information is returned after a successful synchronization.

refresh emr gateway configs success

Authentication process after enabling Kerberos #

If the associated cluster is Hadoop type version 3.8.0 or later and Kerberos is enabled, after the above deployment process, the client node can execute the kinit command. Identity authentication before submitting tasks can be performed in the following two ways:

• Method 1: Download a user's keytab from the cluster user management interface to the Client Node, then execute the following command.

  kinit -kt (keytab路径) 用户名
  

• Method 2: Directly execute the following command.

kinit （用户）
(输入密码)

Enter the user password to authenticate.

Additional procedures for out-of-the-box Presto and Trino #

If the cluster contains Trino or Presto services, to submit tasks on the Client node, you need to additionally copy the key file from the node where the Coordinator is installed in the cluster to the Client Node. The detailed steps are as follows:

Trino

1. Log in to any node in the Master node group of the cluster via SSH, assuming it is the master-1 node, and check whether the keystore.jks key file exists.

   tip

   The key file path is usually /etc/emr/trino/conf/keystore.jks or /etc/emr/trino/keystore/keystore.jks.

2. Log in to the Client node as root and execute the following scp command to copy the key file:

   密钥登录时:
   scp -i 密钥.pem(保证登录集群节点所需要的密钥在 Client node 上存在) -r root@emr-master-1:/etc/emr/trino/keystore /etc/emr/trino/ 
   
   非密钥登录时:
   scp -r root@emr-master-1:/etc/emr/trino/keystore /etc/emr/trino/
   

3. Modify the key file by executing the following command as root:

   chmod -R a+x /etc/emr/trino/keystore
   

4. Run trino --user xxx --password to verify whether it is valid.

5. If Kerberos is enabled, additional authentication steps are required as follows:

   • Download a user's keytab from the cluster user management interface to the Client Node.
   • Execute the command

trino --krb5-principal=(用户名) --krb5-keytab-path=(keytab路径)

Presto

1. Use SSH to log in to any node in the Master node group of the cluster, assuming it is the master-1 node, and check whether the /etc/emr/presto/keystore/keystore.jks key file exists.

2. Log in to the Client node as root and execute the following scp command to copy the key file:

   密钥登录时:
   scp -i 密钥.pem(保证登录集群节点所需要的密钥在Client node上存在) -r root@emr-master-1:/etc/emr/presto/keystore /etc/emr/presto/ 
   
   非密钥登录时:
   scp -r root@emr-master-1:/etc/emr/presto/keystore /etc/emr/presto/
   

3. Modify the key file by executing the following command as root:

   chmod -R a+x /etc/emr/presto/keystore
   

4. Run presto --user xxx --password to verify whether it is valid.

5. If Kerberos is enabled, additional authentication steps are required as follows:

   • Download a user's keytab from the cluster user management interface to the Client Node.
   • Execute the command

   presto --krb5-principal=(用户名) --krb5-keytab-path=(keytab路径)
   

Note #

For Velinux system images, after deploying the EMR cluster client, a MySQL service will be installed with the root directory located at /usr/local/mysql. You can use the mysql client to access Doris and StarRocks services within the cluster.

The default root password is located in /root/mysql/password.dat and is ready to use out of the box.