To access LAS metadata, such as Paimon on LAS or Iceberg on LAS, with Volcano Flink Datastream, add the required LAS dependencies to your open-source Paimon and Iceberg Connector dependencies. Then, package the LAS dependencies into the uber jar.

The steps are as follows:

1. Modify pom.xml #

Add the following pom configuration, which includes the LAS dependency, to your existing Flink job dependencies:

<?xml version="1.0" encoding="UTF-8"?>
<project xmlns="http://maven.apache.org/POM/4.0.0"
         xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
         xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
    <modelVersion>4.0.0</modelVersion>
    
    <!-- Custom Volcano Maven repository -->
    <repositories>
        <repository>
            <id>bytedance-custom-repo</id>
            <url>https://artifact.bytedance.com/repository/data_compute_engine_service</url>
            <snapshots>
                <enabled>false</enabled>
            </snapshots>
        </repository>
    </repositories>

    <dependencies>
        <!-- LF client dependency must be placed before hive-metastore to prevent dependency conflicts -->
        <dependency>
            <groupId>org.apache.hive</groupId>
            <artifactId>lf-client-3</artifactId>
            <version>1.5.0-RELEASE</version>
        </dependency>
        <!-- Use the Hive version that matches your environment -->
        <dependency>
            <groupId>org.apache.hive</groupId>
            <artifactId>hive-metastore</artifactId>
            <version>3.1.2</version> 
            <scope>provided</scope>
        </dependency>
        ...
        Other dependencies, such as Hive, Iceberg, etc.
        ...
    </dependencies>
    
    <build>
        <plugins>
            <plugin>
                <groupId>org.apache.maven.plugins</groupId>
                <artifactId>maven-shade-plugin</artifactId>
                <executions>
                    <execution>
                        <id>shade-flink</id>
                        <phase>package</phase>
                        <goals>
                            <goal>shade</goal>
                        </goals>
                        <configuration>
                            <artifactSet>
                                <includes>
                                    <!-- Package the LAS dependencies into the final job Jar： -->
                                    <include>org.apache.hive:lf-client-3</include>
                                    <include>org.apache.hive:lf-client-common-3</include>
                                    <include>org.apache.hive:lf-common-3</include>
                                    <include>org.apache.thrift:libfb303</include>
                                </includes>
                            </artifactSet>
                        </configuration>
                    </execution>
                </executions>
            </plugin>
        </plugins>
    </build>

</project>

2. Package the job Jar #

The packaging command is as follows:

mvn clean package -DskipTests

Once the job Jar is packaged, upload it to Volcano Flink.

3. Other Notes #

• The provided LAS dependencies consist of multiple independent Maven libraries. If a Maven library version conflicts with one you already have, you can easily resolve it. For instance, you can use your own version for the Parquet dependency.