AI Data Lake Service (Lake AI Service, LAS), as a core component of the integrated lakehouse architecture, provides unified metadata construction and enterprise-level data permission control capabilities. It supports seamless integration with multiple compute engines, including Hive and Spark, enabling users to focus on their own business and accelerate the release of data value.

Prerequisites #

LAS has been enabled, and TOS storage is prepared.

Configure the data catalog #

• Log in to the LAS console, click Metadata in the left navigation panel, and select Data Catalog (catalog).
• The system provides a default data catalog for users. Here, you only need to configure the storage location for this data catalog.
• After completing the data catalog configuration, you can view the corresponding details in the console.

Operate database tables #

You can currently use the LAS SDK to operate databases and data tables. For detailed configuration, see the SDK page.

Configure the client #

public HiveMetaStoreClient createClient() {
    // ak sk of the access account
    String ak = getAk();
    String sk = getSk();
    HiveConf conf = new HiveConf();
// Only supported for Volcengine internal access
    conf.set(HiveConf.ConfVars.METASTOREURIS.varname, "thrift://lakeformation.las.cn-beijing.ivolces.com:48869");
    HiveMetaStoreClient client = new HiveMetaStoreClient(conf);
    client.setRegion("cn-beijing");
    client.setAccessKeyIdAndSecretAccessKey(ak, sk);
}

Create a database #

Create the test_database database using the SDK.

public void createDatabase(HiveMetaStoreClient client) throws TException {
    Database database = new Database();
    database.setName("test_database");
    // By default, automatically uses the form of catalog storage location + db name
    database.setLocationUri("TOS://bucket/warehouse/test_db.db");
    database.setDescription("test database description");
    client.createDatabase(database);
}

After the database is created, it will be visible in the console interface.

Create a data table #

public void createTable(HiveMetaStoreClient client) throws TException {
    Table table = new Table();
    table.setDbName("test_database");
    table.setTableName("test_table");
    StorageDescriptor sd = new StorageDescriptor();
    // Optional; by default, it will be configured as db location uri + table name
    sd.setLocation("TOS://bucket/warehouse/test_db.db/test_table");
    List<FieldSchema> columns = new ArrayList<>();
    columns.add(new FieldSchema("id", "BIGINT", "主键"));
    columns.add(new FieldSchema("name", "STRING", "名字"));
    columns.add(new FieldSchema("age", "INT", "年龄"));
    sd.setCols(columns);
    table.setSd(sd);
    // Set partition columns; for non-partitioned tables, this can be omitted
    List<FieldSchema> partitionKeys = new ArrayList<>();
    partitionKeys.add(new FieldSchema("ds", "STRING", "日期"));
    table.setPartitionKeys(partitionKeys);
    // Advanced table configuration
    Map<String, String> tableParams = new HashMap<>();
    tableParams.put("comment", "test table description");
    hiveMetaStoreClient.createTable(table);
}

After the data table is created, it will be visible in the console interface.

Delete a database #

public void dropDatabase(HiveMetaStoreClient client) {
   client.dropDatabase("test_databse");
}

Delete a data table #

public void dropTable(HiveMetaStoreClient client) {
   client.dropTable("test_database", "test_table");
}

• Data permissions can be configured in the console. Log in to the LAS console and go to the Metadata Management > Permission Management page.
• Click Add Authorization. On the Add Authorization page, select the database created in the previous step, specify the authorization principal, choose the desired permissions to grant, and then click Authorize.
• After authorization is completed, you can view it on the Data Permissions page.

EMR Serveless Spark integration with LAS #

For details about EMR Serveless Spark, see the Serveless Spark Product Introduction.

Environment preparation #

Before integrating EMR Serveless Spark with LAS, confirm the following information in advance:

• LAS has already been enabled. If not, refer to Enable LAS.
• The data catalog configuration has been completed in LAS.
• The queue has been created.

After confirming all the above information, you can start using the EMR Serveless Spark interface.

Spark SQL #

Click Create Job, specify the development type as Spark SQL, and specify the job queue.

Create a database #

Run the database creation DDL in the Spark SQL interface.

CREATE DATABASE IF NOT EXISTS test_database SET location 'TOS://bucket/warehouse/test_database'

After execution is complete, you can view the details of test_database on LAS.

Create a data table

Run the table creation DDL in the Spark SQL interface.

use test_database;

CREATE TABLE IF NOT EXISTS test_table(id BIGINT, name STRING, age INT);

Create a partitioned table

use test_database;

CREATE TABLE IF NOT EXISTS test_partition_table(id BIGINT, name STRING, age INT) PARTITIONED BY (ds STRING);

After execution is complete, you can view the details of the newly created table on LAS, and you can also see that the table directory has been created under the TOS database path.

Data writing and querying

Run DML in the Spark SQL interface.

use test_database;

INSERT INTO `test_table` VALUES (1, 'tom', 12);

use test_database;

INSERT INTO `test_partition_table` PARTITION(ds='20240412') VALUES (1, 'tom', 12);

After data insertion is complete, for non-partitioned tables, data files have been created in the table directory; for partitioned tables, data files have been created in the partition directory. On LAS, you can see that the table information has been updated.
After data writing is complete, run DQL to view the data just written.

use test_database;

SELECT id, name, age FROM test_database;

use test_database;

SELECT id, name, age FROM test_database WHERE ds = '20240412';

Spark Jar #

Upload the job jar package

Before creating a Spark Jar job, you need to upload the file to be executed to the TOS path.

Create a Spark Jar

• Click Create Job and specify the development type as Spark Jar.
• Configure Spark Jar information in the popup window. The resource file is the jar package uploaded in the first step, Main Class is the class name where the job entry function resides, and after configuration is complete, click Run.

View job execution status

After the job is running, you can click the job name to view job details and logs.