Volcano Engine LAS Catalog Java SDK is a development tool for interacting with the LAS Catalog service. It helps you manage metadata through code and improve data governance efficiency. It is suitable for scenarios such as data integration and batch metadata operations.
This article will guide you through installing and using the SDK.
It is recommended to use Maven to install dependencies for the Volcano Engine Java SDK.
First, you need to download and install Maven on your operating system. After installation, simply add the corresponding dependencies to the pom.xml file of your Maven project.
<repositories> <!-- Custom repository --> <repository> <id>bytedance-las-repo</id> <url>https://artifact.bytedance.com/repository/data_compute_engine_service</url> <snapshots> <enabled>false</enabled> </snapshots> </repository> </repositories>
When specifying dependencies, select a specific version number; otherwise, unforeseen issues may occur during the build process.
<dependency> <groupId>javax.annotation</groupId> <artifactId>javax.annotation-api</artifactId> <version>1.3.2</version> </dependency>
Add the lf-client-2 or lf-client-3 SDK dependency package as needed.
Compatible with the open-source Hive 2 protocol. For the specific structure, refer to the official Thrift definition.
<dependency> <groupId>org.apache.hive</groupId> <artifactId>lf-client-2</artifactId> <version>1.6.0-RELEASE</version> </dependency>
Apache Hive, the names of other Apache projects, and the ASF logo are either registered trademarks or trademarks of the Apache Software Foundation in the United States and/or other countries.
tip
The endpoint and region in the configuration specify the service access point and region; refer to Link address for the corresponding values.
export VOLCENGINE_AK=your ak export VOLCENGINE_SK=your sk export VOLCENGINE_ENDPOINT=endpoint export VOLCENGINE_REGION=region
tip
There is a significant security risk in writing the authentication AK and SK directly in the code. It is recommended to store them encrypted in a configuration file or environment variable and decrypt them when used to ensure security.
String ak = System.getenv("VOLCENGINE_AK"); String sk = System.getenv("VOLCENGINE_SK"); String endpoint = System.getenv("VOLCENGINE_ENDPOINT"); String region = System.getenv("VOLCENGINE_REGION"); HiveConf conf = new HiveConf(); conf.set(HiveConf.ConfVars.METASTOREURIS.varname, endpoint); conf.set(ServerlessHiveConf.ConfVars.HIVE_CLIENT_LAS_REGION_NAME.varname, region); conf.set(ServerlessHiveConf.ConfVars.HIVE_CLIENT_LAS_AK.varname, ak); conf.set(ServerlessHiveConf.ConfVars.HIVE_CLIENT_LAS_SK.varname, sk); conf.setBoolean(ServerlessHiveConf.ConfVars.METASTORE_IS_PUBLIC_CLOUD.varname, true); conf.setBoolean(HiveConf.ConfVars.METASTORE_EXECUTE_SET_UGI.varname, false); conf.set(ServerlessHiveConf.ConfVars.HIVE_CLIENT_PUBLIC_CLOUD_TYPE.varname, "catalog-service"); // Set the catalog of LAS to be accessed by this request conf.set("hive.metastore.catalog.default", "hive"); HiveMetaStoreClient hmsClient = new HiveMetaStoreClient(conf); // ak/sk can also be set via hmsClient.setAccessKeyIdAndSecretAccessKey(ak,sk); // You can use retryingmetastoreclient for automatic retries: IMetaStoreClient hmsClient = RetryingMetaStoreClient.getProxy(conf, null, HiveMetaStoreClient.class.getName());
List<String> allDatabases = hmsClient.getAllDatabases(); System.out.println(allDatabases); Table table = hmsClient.getTable("db_test","tbl_test"); System.out.println(table);
import org.apache.hadoop.hive.metastore.api.*; Table table = new Table(); table.setDbName("db_test"); table.setTableName("tbl_test"); // Field definitions List<FieldSchema> cols = new ArrayList<>(); cols.add(new FieldSchema("user_id", "bigint", "用户ID")); cols.add(new FieldSchema("user_name", "string", "用户名")); cols.add(new FieldSchema("age", "int", "年龄")); cols.add(new FieldSchema("email", "string", "邮箱")); table.setSd(new StorageDescriptor()); table.getSd().setCols(cols); // Partition fields List<FieldSchema> partCols = new ArrayList<>(); partCols.add(new FieldSchema("dt", "string", "分区日期")); table.setPartitionKeys(partCols); // Storage format table.getSd().setInputFormat("org.apache.hadoop.mapred.TextInputFormat"); table.getSd().setOutputFormat("org.apache.hadoop.hive.ql.io.HiveIgnoreKeyTextOutputFormat"); table.getSd().setSerdeInfo(new SerDeInfo()); table.getSd().getSerdeInfo().setSerializationLib("org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe"); // Table properties Map<String, String> tblProps = new HashMap<>(); tblProps.put("creator", "hive_demo"); table.setParameters(tblProps); // Create a table via the HMS client hiveMetastoreClient.createTable(table);
Apache Hadoop, the names of other Apache projects, and the ASF logo are either registered trademarks or trademarks of the Apache Software Foundation in the United States and/or other countries.