Apache Paimon is a real-time Data Lake solution for stream and batch processing. It combines efficient storage with flexible computation and is designed for large-scale, real-time, and streaming data. Paimon provides comprehensive Catalog interface support for the Flink engine, allowing you to easily manage real-time Data Lake metadata.

1. Usage Limits #

• The data catalog service supports Paimon metadata based on file systems and LAS. If you need to use other catalogs, such as Hive, refer to the developer reference at Paimon uses EMR Hive to manage metadata.
• The Paimon data catalog can only be used with engine version Flink 1.17-volcano or higher.

2. Steps #

2.1 Create a Catalog #

2.1.1 Create a FileSystem Catalog

Before creating a catalog, you need to create a relevant TOS bucket and catalog folder. For related documentation, see Create a bucket and Create a folder.

Note: Flink and TOS must be in the same Region, because Flink does not currently support cross-Region access to TOS buckets.

Select Data Directory - Catalog List - Create Catalog - Select Paimon Catalog:
Enter the Catalog name and TOS address.

2.1.2 Create an LAS Catalog

Create the required TOS buckets and Catalog folders before creating a Catalog. For related documentation, see Create a bucket and Create a folder.
Additionally, you must:

1. Use API Key Management to create an AccessKey/AccessKeySecret for a sub-account.
2. Grant the sub-account LASFullAccess permission in Permission Management.

Authorize the LAS Catalog in the permission management module. Refer to Permission Management for permission assignment. Because Flink will be used to create database tables and write data. Therefore, we recommend granting Flink developers Admin permissions for the Catalog. Ensure you can perform the following tasks.

After meeting the prerequisites, select the Paimon and LAS Catalog options in Data Catalog and fill in the parameters above:
Click OK to complete the creation.

2.1.3 Parameter Description

2.2 View Paimon Catalog Content #

Go to Data Catalog > Catalog List, select a catalog, and then select the Sync Catalog Metadata feature.

You can view the contents of all Paimon databases and tables in the Catalog:

2.3 Read and Write Paimon Catalog #

2.3.1 Create a Database

Create a database in the Catalog to organize and manage tables.

CREATE DATABASE IF NOT EXISTS ${catalog_name}.${db_name};

• ${catalog_name}: The name of the Catalog. Use the same name as the Paimon Catalog created previously.
• ${db_name}: A custom name for the Database.

2.3.2 Create a Data Table

Create a table in the Database to define its structure and related configurations.

CREATE TABLE IF NOT EXISTS `${catalog_name}`.`${db_name}`.`${table_name}` (
    word varchar, -- Example field
    cnt bigint,
    PRIMARY KEY (word) NOT ENFORCED
) WITH (
    'bucket' = '4',  -- Controls the number of buckets. It is recommended that each bucket stores around 1 GB of data.
    'changelog-producer' = 'input' -- Generates a changelog for downstream stream reading.
);

• ${table_name}: A custom name for the table.
• bucket: The number of buckets. We recommend that each bucket stores approximately 1 GB of data.
• changelog-producer：
  • Set to input to generate data based on new upstream data for downstream stream reading. For details, see Changelog output mechanismDetailed Selection. If a changelog is not needed, use the none option to save storage and write resources.

2.3.3 Querying Paimon Tables

Paimon query SQL supports both streaming and batch reads. Simply select the corresponding mode during the Flink runtime:

INSERT INTO `print_table`
SELECT * FROM `paimon_test`.`default`.`doc_result`;

Additionally, the ad-hoc query feature enables data exploration with Flink SQL:

2.3.4 Streaming and Batch Reads and Writes

The Flink INSERT statement supports both streaming and batch write semantics. Simply select the appropriate execution mode at runtime:

INSERT INTO `paimon_test`.`default`.`doc_result`
select
  t.word,
  count(1)
from
  doc_source t
GROUP BY
  t.word;

2.3.5 Overwriting

In addition to INSERT statements, Paimon also supports batch overwrites on tables and partitions:

-- Overwrites a non-partitioned table.
INSERT OVERWRITE my_table SELECT ...

-- Overwrite partitioned table
INSERT OVERWRITE my_table PARTITION (key1 = value1, key2 = value2, ...) SELECT ...