Docs

Torch Log Service

Consumption and delivery

Synchronize TLS data to EMR Hive in real time

This topic describes how to use the dataset integration feature of Volcengine DataLeap to synchronize logs from TLS to Volcengine EMR Hive in real time.

Application scenarios

This practice uses the data analysis system of a food delivery platform as an example. The main function of the food delivery platform data analysis system is to analyze user logs and business data to display dashboards in real time, perform real-time recommendations, search result ranking, business analysis, business monitoring, and other functions. Therefore, the food delivery platform data system has high requirements for data timeliness, and a real-time data warehouse can better meet this need. You can build a real-time data warehouse using Volcengine cloud products such as TLS, DataLeap, and EMR Hive. For user logs, this practice uses TLS for log collection. Once TLS collects logs, the logs are synchronized in real time to EMR Hive via DataLeap's data integration feature.

Sample log fields are shown below. You can create EMR Hive tables based on the following data types.

tip

All TLS fields are of String type; for EMR Hive data types, please refer to Configuring Hive data sources—supported field types.

Field	Data types (TLS)	Data types (Hive)
id	String	Int
name	String	String
address	String	String
create_time	String	Timestamp
date_info	String	Date
price	String	Decimal(15,5)

Prerequisites

The TLS project and log topic have been created. For detailed instructions, refer to Create log project and Create log topic.

Logs have been sent to TLS. For detailed instructions, refer to Quick Start.

EMR Hive

The EMR cluster has been created. For detailed instructions, refer to Create a cluster.

DataLeap

If you use an IAM user to create a new data source, you must assign the administrator role of the DataLeap project to the IAM user. For detailed instructions, please refer to Manage members.

A DataLeap project has been created in the region where the TLS project is located, and an EMR cluster was bound during creation. For detailed instructions, please refer to Create project.

EMR Hive table has been created. For detailed instructions, refer to Manage EMR Hive tables.

A dedicated integration resource group has been created. For detailed instructions, please refer to Dedicated resource group management.

The VPC used in the dedicated integration resource group must be the same as the VPC used by the EMR cluster.

1. Register a TLS data source

In the region dropdown menu at the top of the page, select the target region.

On the Project Management page, click the Configuration Information for the target project.

In the left navigation bar of the project console, click Data Source Management.

On the Custom Data Source page, click New Data Source.

In the Add Data Source panel, configure as follows.

Note: The important parameters are explained as follows. For detailed instructions, please refer to Configure a TLS data source – data source registration.

Parameter	Description
Basic configuration
Data source type	Select Volcengine TLS.
Access method	Select TLS.
Data source name	Data source name.
Parameter configuration
TLS project ID	Enter the target TLS project ID.
Access Key ID	Enter the Access Key ID of the VolcEngine main account or sub-account. On the Key Management page in the Volcengine console, follow the prompts on the page to view and copy the Access Key ID. If you use a sub-account's Access Key, you must grant the sub-account the relevant permissions for the corresponding log project and log topic, including DescribeProject, DescribeTopics, DescribeShards, DescribeCursor, and ConsumeLogs.
Secret Access Key	Secret Access Key of the VolcanoEngine main account or sub-account. You can view and copy the Secret Access Key on the Volcano Engine console Key Management page by following the instructions on the page. If you use the Access Key of a sub-account, you must grant the sub-account the relevant permissions for the corresponding project and log topic, including DescribeProject, DescribeTopics, DescribeShards, DescribeCursor, and ConsumeLogs.
Resource group connectivities	Select the target dedicated integration resource group, and click Test. The data source and the dedicated integration resource group network must be able to communicate with each other. For details, see Network connectivity solutions.

Click OK.

2: Register Hive data sources

On the Custom Data Source page, click New Data Source.

In the Add new data source panel, configure as follows.

The important parameters are described as follows. For detailed instructions, refer to Configure Hive data source - Data source registration.

Parameter	Description
Basic configuration
Data source type	Select Hive.
Access method	Select EMR Hive.
Data source name	The data source name.
Parameter configuration
Authentication methods	SelectNone.
EMR cluster ID	Select the target EMR cluster name.
Database name	Select the target Hive database.
Hive version number	Based on the selected EMR cluster, automatically display the Hive versions included in the cluster.
Username	Enter a username with permission to access the database.
Password	Enter the user password.
Resource group connectivity	Select the target dedicated integration resource group, and click Test. Ensure that the data source and the dedicated integration resource group network are mutually accessible. For details, see network connectivity solution.

3 Create a data synchronization task

This example uses visual mode to create a data streaming synchronization task. If you want to use DSL mode and batch data reading, refer to Configure TLS data source—data synchronization task development and Configure Hive data source—data synchronization task development.

Return to the Project Management page and click the name of the target project.

In the left navigation bar, click New Task.

Select task types

Parameter
Description
Category
Select Data Integration.
Select task
Select streaming integration.

Set the basic information of the task.

Parameter
Description
Task name
The name of the data synchronization task.
Save to
Select the target directory for storing the task.

Click OK.

Select a data source.

Configure the data source.

Parameter	Description
Data source type	Select TLS.
Data source name	Select the TLS data source you registered.
Log topic	Select the TLS log topic where the logs to be synchronized are located.

Set the target data source.

Parameter	Description
Target type	Select Hive.
Data source name	Select the Hive data source you registered.
Data table	Select the target EMR Hive table for storing synchronized logs.
Shard frequency	Select day-level or hour-level shard frequency according to business requirements. tip Data is produced according to the shard generation time in Hive, rather than being written in real time.
Shard settings	Partition fields are automatically obtained from EMR Hive tables. You can set the specific time granularity for partitioning shard content, and use parameter variables such as ${date} and ${hour} to indicate the partitioning granularity. tip It is recommended that the number of dynamic shards in a shard type be less than 10; otherwise, the consumption rate will decrease.

Set up field mapping between source and target fields

Specify the field mapping relationships between the source endpoint and the target endpoint based on TLS logs and EMR Hive tables. Based on the field mapping relationship, the data integration task writes the data from the fields on the source side into the corresponding fields on the target side. For detailed operational steps, please refer to Stream data synchronization - task configuration.

Add a source field.

Add fields to the target

Set the task run parameters.

The important parameters are described in the following table. Other parameters can remain at their default settings. For detailed instructions, refer to Streaming data synchronization - task configuration.

Parameter	Description
Default start of consumption	Select the start time for consuming TLS logs. Latest: Start consuming from the latest log. Timestamp: Start consumption from the specified timestamp.

4 Use data

EMR supports online data updates, and Doris/StarRocks provide services to the data application layer, supporting online and offline query analysis. After synchronizing logs to EMR Hive, you can build a real-time data warehouse based on EMR Doris or StarRocks. A real-time data warehouse enables rapid data processing and analysis, allowing for real-time analysis of data from the food delivery platform and quick access to information such as user profiles, report analysis, decision support, and other information.

Last updated: 2025.10.11 17:07:35

Parameter	Description
Category	Select Data Integration.
Select task	Select streaming integration.

Parameter	Description
Task name	The name of the data synchronization task.
Save to	Select the target directory for storing the task.