TLS supports consuming data via consumer groups and provides a range of SDKs to manage consumer groups. This topic describes how to consume data using consumer groups in TLS.

Background information

TLS provides the ConsumeLogs OpenAPI interface for log consumption, supporting real-time access to log data collected on the server. When using the ConsumeLogs interface, you must consume log data based on the shard dimension. Plus, you need to specify the log topic ID, Shard ID, and starting and ending cursors during consumption. Therefore, the progress of log consumption depends on the read-write capability of each Shard. You also need to manually maintain the consumption progress. In scenarios where shards are auto split, the consumption logic and workflow become complex.

TLS offers the ConsumerGroup feature through the SDK. You can consume log data using consumer groups. When consuming logs via a consumer group, TLS automatically balances consumer capacity and progress and allocates shards. You do not need to manage the internal scheduling details, load balancing, or failover between consumers. You can just focus on your business logic.

Basic concepts

Concepts	Description
Consumer Group (ConsumerGroup)	A virtual group composed of multiple consumers. When logs are consumed at the consumer group level, all consumers in the group subscribe to the same log topic and work together to consume data from that topic. Each consumer processes data from one or more shards in the log topic. Data is not consumed redundantly across different consumers.
Consumer (Consumer)	A client that consumes data from TLS and is a part of a consumer group. Within the same consumer group, consumer names are unique. At any given time, a shard in a log topic is assigned to a consumer in the consumer group. One consumer may handle multiple shards. During the running of consumer groups, the relationship between consumers and shards may change.
Consumption offset (Checkpoint)	When a shard is consumed by a consumer, the current consumption checkpoint(cursor progress) of the shard is continuously recorded and reported to the server. This offset becomes the starting cursor for program restarts, ensuring data is not consumed repeatedly.

Limits

Each project supports up to 150 consumer groups.

A consumer group can consume multiple topics within the same project, but cross-project consumption is not supported.

Prerequisites

AccessKey for Volcanoengine has been created and obtained.

The access key (AccessKey) of the Volcanoengine account has full permissions for all APIs. We recommend using an IAM user for API operations and routine O&M. Before using an IAM user, the primary account should grant the IAM user the relevant consumer group permissions. For authorization examples, see Examples of custom permission policy.

TLS SDK has been installed. Currently supported SDKs for consumer groups include Go SDK overview, Java SDK overview and Python SDK overview.

Consume data with consumer group

TLS provides the Consumer asynchronous log consumption library, which supports consuming multiple log topics under the same project. Features include asynchronous consumption, high performance, retry on failure, and graceful shutdown. You can use TLS Go SDK overview, Java SDK overview or Python SDK overview to manage consumer groups and consume log data.

Code examples

The following code uses the Go SDK to demonstrate how to create consumer groups and consumers, and how to consume logs using the SDK.

Configuration instructions

The TLS SDK allows you to configure various consumer group consumption settings, such as enabling sequential consumption and setting the consumer heartbeat interval. You can use these configurations to manage different consumption logics of consumer groups.

For example, in the Go SDK code example above, the log_consumer.GetDefaultConsumerConfig() function returns the consumer group’s default configuration consumerCfg, and shows how to configure basic information like Endpoint, Region, AccessKeyID, AccessKeySecret, as well as project ID and log topic IDs, consumer group name, and consumer name. In addition, you can use other fields in consumerCfg for further custom settings. Supported fields for consumerCfg are as follows.

Parameters	Type	Example value	Description
MaxFetchLogGroupCount	Integer	100	The maximum number of LogGroups a consumer can retrieve in a single log consumption is 100 by default and up to 1000.
HeartbeatIntervalInSecond	Integer	20	The interval for consumer heartbeat reporting, measured in seconds.
DataFetchIntervalInMillisecond	Integer	200	Consumer log time interval; measured in milliseconds.
FlushCheckpointIntervalSecond	Integer	5	Time interval in seconds for a consumer to upload consumption progress.
ConsumeFrom	String	begin	The default consumption checkpoint when starting consumption. It is consistent with the From parameter in DescribeCursor. It is valid only when the consumer has never uploaded a consumption checkpoint.
OrderedConsume	Boolean	FALSE	Enable ordered consumption. After enabling ordered consumption, consumers will process data based on the parent-child relationships of shard splits. For example, Shard0 splits into Shard1 and Shard2, and Shard1 further splits into Shard3 and Shard4. After enabling ordered consumption, data will be processed in the order: (Shard0) -> (Shard1, Shard2) -> (Shard2, Shard3, Shard4).
LoggerConfig	LoggerConfig	/	Log-related configuration. For details, see the table below.

LoggerConfig configuration description:

Parameters	Type	Example value	Description
LogLevel	String	info	Set the log output level. The default value is info. There are four log output levels in consumer: debug, info, warn, and error.
LogFileName	String	50	Log file output path. If not set, logs are output to stdout by default.
IsJsonType	Boolean	true	Determines whether to format the file output. The default is false.
LogMaxSize	Integer	10	The maximum storage size for a single log is 10 MiB by default.
LogCompress	Boolean	true	Whether log compression is enabled; the default is false.

Troubleshooting

TLS SDK consumer group provides automatic retry for failed requests and automatic reporting of consumption progress checkpoints. Therefore, you only need to focus on implementing the business logic for handling each consumed LogGroupList.

View or reset consumption progress

You can check the current consumption progress at any time while consuming log data via the consumer. In offline data processing scenarios, if you need to consume log data from a past period or from a specific position, you can reset the consumption checkpoint for the consumer group to start consuming data from the specified position.

TLS console supports viewing consumption progress, resetting all consumption checkpoints for the consumer group on a subscribed topic, and resetting checkpoints for specific shards via API.

The relevant steps in the console are as follows:

Log in to TLS console.

In the left navigation pane, select data processing > log consumption.

On the SDK tab, find and expand the project containing the target consumer group, and then click the specified consumer group.

View consumption checkpoint.

Click the consumer group name in the left consumer group list to view the most recent consumption time for each consumer in each shard.

Reset consumption checkpoint.

Click the consumer group name in the left consumer group list.

In the upper-right corner, click Reset checkpoint.

Select Reset point.

Reset position	Description
Earliest location	Consume data starting from the earliest entry in the log topic.
Latest location	Skip all historical data and consume directly from the most recent entry written in the log topic.
Specified time point	Consume data starting from a specified point in the past, based on the data retention period of the log topic. When resetting the consumption point to a specified time, you can not set it to a past moment beyond the data retention period or to a future time.

Click Reset.

Related API

API	Description
CreateConsumerGroup	Call the CreateConsumerGroup API to create a TLS consumer group.
DeleteConsumerGroup	Call the DeleteConsumerGroup API to delete a TLS consumer group.
DescribeConsumerGroups	Call the DescribeConsumerGroups API to get information about all consumer groups in a project.
ModifyConsumerGroup	Call the ModifyConsumerGroup API to update the configuration of a TLS consumer group.
ConsumerHeartbeat	Call the ConsumerHeartbeat API to send the heartbeat information of a consumer in a consumer group to TLS.
DescribeCheckPoint	Call the DescribeCheckPoint API to view the current checkpoint of a specified consumer group on a specified shard.
ModifyCheckPoint	Call the ModifyCheckPoint API to reset the checkpoint of a specified shard for a specified consumer group.
ResetCheckPoint	Use the ResetCheckPoint API to reset all checkpoints for the specified consumer group.

Torch Log Service