Data catalog

Add new data catalog feature

Support Paimon type Catalog

Add Beta

Job development

File management feature

File management has been greatly optimized to provide customers with better management capabilities

1. Supports private file types and stricter permission management capabilities
2. Supports multi-version management, allowing file updates, viewing history versions, and rollback of history versions
3. Supports reference management to view all tasks referencing a file

New

Temporary query feature

Supports temporary query capability for user convenience

1. Supports running temporary query tasks on selected Session clusters, currently supporting DQL and some DDL SQL types
2. Supports formatting, running, and saving query scripts for easier code reuse

New Beta

job development management

1. support graphical management of state TTL and backend state (default rocksdb) capabilities
2. support selecting specified file versions or fixed historical versions in dependency files

optimization

Flink engine

Flink CDC version iteration

Flink CDC 3.4 full release; customers can choose Flink CDC version for architecture upgrade

upgrade

file system

use Proton as the default TOS file system access, supporting high-performance TOS read and write

Upgrade

Job operations and maintenance

Support job prioritization

Serverless Flink tasks support tagging priorities:

1. Support high, normal, and test priority tasks, and update real-time task levels without restarting, enabling better task management.

Optimization

Alarm optimization

Cloud monitoring alarms include Serverless Flink project details to prevent interference from same-name tasks across different projects.

Optimization

Resource pool

Hybrid Elastic Resource Pool

In addition to pay-as-you-go and monthly subscription resource pools, Serverless Flink supports hybrid elastic resource pools to better provide computing services:

1. Fixed Part: Paid on an annual or monthly subscription basis for resident computing resources
2. Elastic Part: Paid on a pay-as-you-go basis for temporary and burst computing resources

Add

DNS Optimization

DNS input is optimized to avoid omissions during multiple consecutive entries.

Optimization

DNS supports private network resolution

Serverless Flink supports Volcano Engine private network resolution (PrivateZone), enabling the following scenarios:

• Resolve domain names to IP addresses within the VPC or public IP addresses
• Forward DNS query requests between the VPC and self-built IDC/third-party cloud services

Add

Project overview

New UI style optimization

Product navigation bar optimized to support improved visuals and interaction

Optimization

Project Overview Optimization

Support displaying used resources separately for pay-as-you-go and monthly resource pools

Optimization

Data Catalog

Added Data Catalog feature

Support visual management of metadata such as databases and tables, making Serverless Flink development and metadata management more convenient.

1. Support integration with EMR-Hive, LAS-Hive, and other catalogs
2. Support database table metadata synchronization
3. Support Serverless Flink task write operations without table creation statements

Add beta

job development

variable management optimization

greatly optimize variable management to provide clients with better control

1. support variable value editing
2. support encrypted variables for safer management of keys and sensitive data
3. support referencing variable values in Flink-CDC, JAR, PyFlink, and custom parameters

add

support deep inspection

continuous optimization of deep inspection

1. Supports deep inspection of CDC tasks
2. Supports diagnosing issues of disorder caused by uncertainty in SQL tasks
3. Provides performance recommendations for SQL

Added

Flink engine

Flink CDC version iteration

Supports compatible versions of Flink CDC 3.4+, enabling richer downstream data

Upgraded

Flink CDC supports BH automatic table creation

Flink CDC syncs ByteHouse CDW tables, supporting automatic table creation based on upstream table DDL events

Optimize

Upgrade ByteHouse CE/CDW Connector

Upgrade ByteHouse CE/CDW connector
Upgrade from 1.16 to 1.27.131-1.16
Upgrade from 1.17 to 1.27.131-1.17

Upgrade

Job Operations

Add levels to runtime events

Runtime events now include INFO / WARN / ERROR levels to help users view and filter events by level.

Optimize

resource pool

DNS input optimization

Optimize DNS input to avoid omissions during multiple consecutive entries.

optimize

job development

JAR task supports KV mode

JAR tasks support KV mode, improving input handling for multiline and complex parameters.

Add

Optimize network connectivity testing

Allow selecting Session clusters directly without choosing a version first.

Optimize

Support deep inspection

Support phase one SQL deep inspection capabilities, including syntax and semantic checks.

1. Support testing Connector connectivity, permission verification, and syntax and semantic correctness
2. Enable quickly locating syntax errors that cause connectivity issues upstream or downstream during verification

Add

Flink engine

Built-in support for HNS buckets

Flink 1.17 and above versions built-in support for HNS hierarchical buckets

Added

Support JSON parsing optimization

Support object reuse during JSON parsing, significantly improving JSON parsing performance.

Added

Paimon job stability improvement

Flink 1.17 and above versions disable multiple-input optimization by default to prevent checkpoint failure during Paimon multi-stream join

Optimized

Configure template

Support template duplication

Support template duplication. For templates with many configuration parameters, enable one-click duplication to avoid repeated input.

Add new

Optimize template display style

Optimize the style of configuration templates that rely heavily on files for easier management

Optimize

Resource pool

Support PLGW network optimization

Significantly reduce customer IP resource usage by optimizing from one IP per POD to one IP for the entire resource pool.

invitation testing

job development

network connectivity test

supports network connectivity testing for session cluster resource pools, data sources, and Flink engines to facilitate job development and troubleshooting.

add

Flink CDC variable reference

Flink CDC supports referencing environment variables or project variables to enable dynamic configuration.

Added

Job operation and maintenance

0.5 CU container stability optimization

Optimized small-scale container operation logic to reduce memory risks and improve stability of long-running tasks.

Stability

Enhancements to data lake ingestion and performance metrics

Added monitoring metrics such as data lake ingestion delay and task throughput, optimized alarm strategies, and improved operational efficiency.

Added

Log relative time selection

The log module supports relative time (such as the last 15 minutes) and time range filtering for faster issue identification.

Optimization

Connector

Built-in LAS Catalog support

No need to upload custom Connectors; with the built-in LAS Catalog, metadata connection is streamlined.

New

Resource Pool

IAM tag dropdown filter

Supports filtering existing IAM tags via dropdown when creating resource pools, improving tag management efficiency.

Optimization

Configure Template

Default Template Settings

Support saving common configurations as default templates that load automatically when creating new tasks, reducing repeated selections.

Add

Job Development

Variable Management

The new job development supports variable management, template configuration, connector management, and other modules, providing more convenient support

Added

Network parameters

The product has built-in network parameters enabled, so no additional custom parameters are needed for cross-VPC and external network access

Added

Session debugging

Supports session debugging with Flink 1.17 cluster version selection

Added

Job type

Job development supports pyflink job development (only supports Flink versions 1.17 and later)

Add

Flink Connector

Flink CDC

1. Flink MongoDB CDC data source supports ignoring delete events.
2. Flink CDC supports Paimon 0.8.2 byte-enhanced version.
3. Flink CDC supports Paimon-LAS data downstream.

Optimize

ByteHouse Connector

Upgrade the built-in Connector version of ByteHouse CE/cloud data warehouse edition to 1.27.109_patch5-1.16, the latest stable version

Optimize

Official website integration

Project selection

Supports product top leader project selection space

Add

Project management

Supports managing Flink product-related resources by tags and other dimensions in project management

Add

Project billing

Supports billing and other operations by project and tag dimensions

Add

Flink version

Flink 1.17

Flink 1.17 engine version continues to optimize, supports Session debugging, stored procedure syntax, and includes the built-in Paimon 0.8.2 enhanced byte version.

Beta release

Resource pool management

DNS settings

Supports setting DNS information in the resource pool interface to meet private domain access requirements.

Added

Job operation and maintenance

Job restart

In addition to basic functions like job stop and start, it also supports one-click job restart to reduce operation complexity in job maintenance

Added

File management

File upload

Using file chunk upload significantly improves efficiency and product stability for files over 100MB

Added

Job development

New job development features with a brand-new UI and delivery style. The current version fully supports SQL, JAVA, and CDC task development capabilities.

Beta release

Job operation and maintenance

The new job operation and maintenance startup process supports BMQ/Kafka data sources starting based on time points.

Added

Log archiving

Flink jobs support exporting runtime logs to TLS by log level for more comprehensive log analysis and alert functions.

Added

Flink Connector

Supports Kafka cross-cluster synchronization, enabling multi-cloud and multi-Kafka cluster data synchronization. Supports specifying topic whitelists, wildcards, and seamless migration of MirrorMaker traffic from existing consumer group offsets

New

Flink version

Flink 1.17 engine version continuously optimized, supporting full upstream and downstream, native engine, and other enterprise enhancements

Beta release

Flink CDC

The new job development supports the latest Flink CDC 3.x, enabling job development through YAML configuration, supporting full database synchronization, database and table level synchronization, schema change synchronization, and data transformation capabilities

Beta release

Job development

The new job development feature offers a refreshed UI and delivery style, providing developers with a better Flink task development experience. Currently supports Flink CDC development.

Beta release

Job locking

Job development supports locking to prevent conflicts during concurrent operations by multiple developers.

Add

File management

An independent file management module enables clearer and more accurate management of all dependency files in Flink projects.

Add

Session cluster

An independent session cluster management module enables easier management of session clusters for debugging purposes.

Add

Flexible resource allocation

Current job submissions support flexible CPU and memory resource allocation. Supports customers in precisely controlling resource usage.

Optimize

Business latency

Cloud monitoring metrics optimization: business latency metrics now support multi-metric composition with Kafka Lag at the Kafka dimension, improving the accuracy of business latency alerts.

Optimize

Job operation and maintenance

After exiting task details, search criteria remain unchanged.

Optimize

Version selection

When submitting a task, the main version of the current streaming Flink engine is selected by default.

Optimize

Paimon Catalog

Support Paimon Catalog.

Add

Enable task skip verification for release

The SQL task supports skipping deep SQL code verification to allow forced task release.

Add

Flink Python task

Support developing Flink Python type tasks.

Add

Task parameter configuration

Added hints to task configuration parameters to improve user readability.

Optimize

Scheduling duration

Added hints to the scheduling duration parameter when launching a task to improve user readability.
Scheduling duration indicates the interval before retrying scheduling, meaning the task will retry scheduling if it fails to start.

Optimize

Task resource usage

On the task list page, display CPU and memory usage of launched tasks, and support sorting in ascending or descending order.

Optimize

Task logs

Task log supports full-screen viewing and downloading for local viewing.

Optimization

Project integration

Integration of projects in the Flink Console and Volcano Engine allows grouping and management of cloud resources used by different businesses or projects.

• IAM authorization based on projects helps maintain resource independence and data security.
• Resource consumption bills can be viewed from the project perspective, facilitating calculation of cloud resource usage costs.

Optimization

add Flink Connector

add ByteHouse connector, including cloud data warehouse version (bytehouse-cdw) and enterprise version (bytehouse-ce)

add

support external EMR Hive Catalog

Flink console includes the Default Hive Catalog by default and now supports adding external EMR Hive Catalog

add

distinguish stream and batch task configurations

Automatically distinguish task configuration parameters based on task type.

Optimize

Optimize front-end page styles.

Optimize the product overview page.

• Provide quick access to commonly used projects.
• Provide product application scenario descriptions to enhance cloud product integration.
• Provide links to common documents such as best practices and quick start guides.

Optimized

Support small-scale resource configurations

Support small-scale resource configurations when configuring Task Manager and Job Manager resources; currently, the minimum setting is 0.5 cu.

Optimized

Flink CDC Connector

Added CDC connectors, including MySQL CDC, MongoDB CDC, Postgres CDC, and SQLServer CDC.

Added

Support auto restart on task failure

During task execution, if an exception causes the entire task to fail, the system will attempt to restart the task according to the configured retry on failure policy.

• Maximum retry attempts
• Retry interval

Add

Resource pool supports yearly and monthly billing types

Support creating general VCI resource pools with yearly and monthly billing types.
Yearly and monthly billing requires prepayment and offers better pricing compared to pay-as-you-go.

Add

Flink image cache snapshot

Flink image cache snapshots are supported by default to improve container startup speed.

Add new

Centralized architecture optimization

The product's centralized architecture has been adjusted to optimize tenant service quantity, significantly reducing product initialization time.

Optimization

Monitoring metric optimization

Added Flink task monitoring metrics to better understand task status and troubleshoot issues.

Optimize

Task checkpoint support disable

Checkpoint is disabled by default for tasks.
To manually create snapshots for task failure recovery, enable checkpoint. When enabled, you can set parameters such as checkpoint interval and timeout duration.

Optimize

Task restart policy

Provides various task restart strategies to choose the appropriate one based on business needs.

• No restart strategy
• Fixed interval restart strategy
• Failure rate restart strategy
• Aggregation failure rate restart strategy

Add

Supports Flink Batch type tasks

Add Flink Batch type tasks to support more business scenarios.

Add

Metadata overview

Provides a metadata overview page to view metadata table storage location, recent update time of table structure, table structure, partition columns, and other information.

Add

Flink accesses the public network

In special scenarios, Flink needs to access public network resources in other cloud domains.
Declare the use of an external network in the Flink parameter configuration and set up SNAT rules to enable public network access.

Add

Flink reads and writes TOS

Flink provides read and write capabilities for object storage.

Add

multi-engine versions

supports Flink V1.11 and Flink V1.16 engine versions, allowing switching of the task runtime engine.

new

data lineage

on the task development side, you can configure upstream and downstream data source information, and later view data lineage in the task details.

new

task version management

when the task is deployed to the operation side, the system records task versions, supporting version comparison and rollback of historical versions.

Add

Project Role

Add a new project regular user (Project_Member) role, with read-only access to most project resources.

Optimize

Product Initialization

Update the product initialization process to include association with TOS and VPC.

Update

服务初始化

Project Overview Architecture Optimization

• Projects are independent and provide a unique portal entry point.
• Project members, roles, and policy management are built-in for easier fine-grained permission control.

Update

• 项目管理
• 项目权限管理

IAM User Management

Supports importing and removing IAM users; imported IAM users can serve as members of Flink projects.

Add

用户管理

Project Management

Supports creating, editing, and deleting projects.

Add

项目管理

Project Access Control

Projects provide a unified space to manage resource pools, tasks, and other resources. Different projects are isolated and independent from each other.
The main account has permissions for all projects and resources under it; project members only have relevant permissions for the current project.
Projects have independent access control policies, using a complete “policy-role-member” control chain to ensure minimal member permissions and enable precise permission management within projects.

Add

• 权限概述
• 成员管理
• 角色管理
• 策略管理

Resource pool management

A resource pool manages computing resources in a project. The computing resources within a resource pool are isolated and independent from each other.
Tasks submitted to the resource pool can only use the computing resources within that pool.
Supports creation and deletion of resource pools.

Add

资源池管理

product initialization

Before using the Flink product, complete product initialization first. This process requires associating TOS and VPC.

add

快速入门

workspace management

support creating, expanding, and deleting workspaces.

add

工作空间管理