Docs

Torch Log Service

Consumption and delivery

Consume logs via ByteHouse

ByteHouse (Cloud Data Warehouse Edition) supports streaming data via Kafka. This document describes how to consume log data from TLS into ByteHouse using the Kafka protocol.

Background information

TLS supports consuming log data from specified log topics via the Kafka protocol. For example, you can deliver log data to ByteHouse (cloud data warehouse edition), where it can be further analyzed and processed. After creating a Kafka data import task in ByteHouse, you can stream data directly via Kafka. The data import task will run automatically, continuously reading log data from the log topic and writing it into the specified database table.

When consuming logs, you can choose to consume only certain fields and configure settings such as the maximum message size. You can also stop the data import task at any time to reduce resource usage, and resume the task whenever necessary. ByteHouse will record the offset internally to ensure that no data is lost during stop and resume operations.

Billing description

When consuming logs via ByteHouse, TLS read traffic fees are incurred. It is recommended to use the private network service address to consume log data. For the corresponding internal network read traffic fees, please refer to Billing items and pricing.

After logs are ingested into ByteHouse, storage fees will be incurred in ByteHouse. If real-time query and analysis are performed on the data, this will also incur computational costs. For detailed pricing, please refer to Billing overview.

Prerequisites

TLS has been enabled, a log project and log topic have been created, and log data has been successfully collected. For detailed instructions, please refer to Quick Start.

The Kafka protocol consumption feature has been enabled for the specified log topic. For instructions on how to enable the Kafka consumption feature, refer to Enable the Kafka consumption feature.

It is recommended to use an IAM account to perform operations. Before performing any operations, ensure that the currently logged-in account has permission to consume via the Kafka protocol, specifically the Action ConsumeLogs permission. For more details, refer to Authorized actions.

ByteHouse (Cloud Data Warehouse Edition) has been activated. Please refer to Service activation for the operational steps.

Notes

When consuming logs through ByteHouse, the maximum log size that can be ingested into ByteHouse is 128KB. Log data that exceeds this limit cannot be successfully ingested into ByteHouse. This limitation is specified by the Max Block Size configuration of the data import task. The value range for this configuration is 65,536 to 131,072 bytes, that is, the maximum is 128KB.

Import logs into ByteHouse

The ByteHouse console provides both new and old versions of the data loading interface. This document uses the old data loading console interface as an example to demonstrate the steps for creating a new Kafka data source and a new data import task.

Create a database.

In the upper left corner of the database page, click + New Database.

Enter the basic information for the database.

Configuration
Description
Database name
The database name can only contain letters, numbers, and underscores (_included).
Database description
Database description.

Click create.

Create a data table.

On the details page of the specified database, click the Data Table tab.

Click + Data Table at the top right corner of the page.

Select SQL or visual page creation.

Set column names and data types in the SQL or visualization tab.

You need to design a new table structure based on the source log structure in TLS. It is recommended to create only the columns that need to be saved or used for subsequent analysis.

Click create.

Create a Kafka data source.

Click Data loading in the top navigation bar.

Click Data Source at the top left corner of the page.

On the data source page, click + Connect new source.

Select Kafka as the data source type. data flow.

Enter the basic information for the Kafka source.

Configuration	Description
Source name	Kafka data source name.
IP addresses in the Kafka proxy list	A cluster address for initial connection The format is service address:port number; for example, tls-cn-beijing.ivolces.com:9093. Details: The service address is the TLS service address in the current region. Select the correct service endpoint based on region and network type. For details, see Service address. It is recommended to use a private network service address to save on data charges. The port number is fixed at 9093.
Authentication modes	The authentication mechanism. The default settings should be maintained here PLAIN.
SSL	Enable SSL? The default enabled state should be maintained here.
Username	This parameter should be configured as the project ID of TLS.
Password	Use VolcanoEngine account key. The format is ${access-key-id}#${access-key-secret}, where: ${access-key-id} should be replaced with your AccessKey ID. ${access-key-secret} should be replaced with your AccessKey Secret. tip If the currently logged-in user is an IAM user, you should configure the IAM user's key here.

Click Connect.

Create a data import task.

Click Data loading in the top navigation bar.

Click the + New data load button at the top-left corner of the page.

Enter basic information.

Configuration
Description
Task name
Name of the data import task. Length must be between 1 and 128 characters.
Task description
Description of the data import task.

Select data source.

Configuration	Description
Category	Choose Kafka data flow.
Data source	Select the Kafka data source created in Step 4.
Topic	Kafka protocol consumption topic ID, in the format out-[topic ID], for example out-0fdaa6b6-3c9f-424c-8664-fc0d222c****. You can also view and copy the Kafka protocol consumption topic ID on the Topic detail page in the TLS console.
Consumer group	A Kafka consumer group is used to consume log data from specified log topics. If you have never created one before, you can click Create consumer group to create one temporarily.
Format	Data formats. This value must be fixed as JSON_KAFKA.

Select target table.

Configuration
Description
The target database
The database name where TLS log data is stored in ByteHouse.
Target table
The name of the table in ByteHouse where TLS log data is stored.

Define Schema mapping. Configure the field mapping between log data in TLS and table fields in ByteHouse. After specifying the mapping relationship, the designated fields in ByteHouse will be stored using the corresponding column names and data types.

tip

If log data already exists in the log topic, you can directly click Parse to automatically parse the data.

If the table structure does not meet your requirements, you can also click Edit target table to modify the structure of the target table.

The source column name and data type of a field must exactly match those of the field in the source log data; otherwise, the field cannot be imported successfully.

Set import task configuration.

Configuration	Description
Load type	Methods of data loading. Currently, only incremental write is supported. That is, during each job execution, log data will be incrementally written to the target table, and existing data will not be overwritten.
Max Block Size	Maximum message size. The unit is bytes. The value range is from 65,536 to 131,072 bytes; that is, the maximum can be set to 128 KB.

Click create. After a data import task is successfully created, it will automatically run and continuously stream log data from TLS using the Kafka protocol. If you need to pause an import task, you can click Stop in the upper right corner of the import task details page.

View import tasks

The Execution plan > Real-time data loading page displays the status information of all existing real-time data import tasks. Click the name of the import task to view detailed information.

Display item			Description
Basic information	Data source	The Kafka data source name
		Topic	The Topic of the Kafka data source, which is the consumption topic ID for TLS using the Kafka protocol.
		Consumer groups	The name of the consumer group in ByteHouse.
		The target data table	The name of the table in ByteHouse where TLS log data is stored.
		Total number of imported records	Number of successfully imported logs.
		Creation date	Import task creation dates
Sync history records			Each time a task is resumed after being paused, the backend creates a new execution record. Synchronization history tab displays all execution records for the task, including Execution ID, Execution start time, Status, and Duration. In addition, changes in import record data for each day over the past 14 days are also displayed together. Among them, for details on the various statuses, please refer to Synchronization status descriptions. Click Execution ID to view detailed information for each execution record, including Status, Start time, End time, Duration, Number of imported records, Number of failed imported records, and more. In addition, Error details, Configuration information, Logs, and Consumption status during execution are also displayed simultaneously.
Schema Mapping			Schema mapping information, that is, the mapping relationships between the source column and the target column, and between the source data type and the target data type.
Consumption status			By default, the consumption metrics for this import task over the past 15 minutes are displayed, namely the current offset, number of unconsumed messages, and consumption rate.
Error log			By default, all error messages from the execution records of this import task are displayed.
Configuration			By default, the system displays the configuration details of this import task in JSON format.

Synchronization status descriptions

The synchronization status of data import tasks is explained as follows.

Status	Description
Resource allocation in progress	After starting the data import task, the backend begins allocating the corresponding computing resources. After resource allocation is completed, it will enter the Running state.
Running	The data import task is running normally.
Stopped	The data import task has been stopped.
Import failed	Data import task failed. Possible causes include configuration conflicts and data anomalies. It is recommended to restart the import task after checking the configuration and data source.

FAQs

Error indicating "unable to connect to Broker" occurs when creating a Kafka data source

Problem description: When creating a new Kafka data source in Step 4, the page reports the error "kafka connectivity test failed, ping failed with err: kafka: client has run out of available brokers to talk to.......".

Possible causes: The log topic has not enabled Kafka protocol consumption, or the Kafka broker list IP address, username, and password are incorrect.

Solution:

To enable the Kafka protocol consumption feature for the specified log topic, refer to Enable the Kafka protocol consumption feature for instructions.

Refer to the operation steps to correctly fill in the relevant parameters. For instructions on the operation steps, see operation steps.

Last updated: 2025.10.11 17:07:57

Configuration	Description
Database name	The database name can only contain letters, numbers, and underscores (_included).
Database description	Database description.

Configuration	Description
Task name	Name of the data import task. Length must be between 1 and 128 characters.
Task description	Description of the data import task.

Configuration	Description
The target database	The database name where TLS log data is stored in ByteHouse.
Target table	The name of the table in ByteHouse where TLS log data is stored.