Docs

Torch Log Service

Use other systems to analyze logs

Visualize and analyze logs via Grafana

TLS provides a data source plugin for rapid integration of log data with Grafana, making it easy to visualize and analyze log data in Grafana. This topic describes how to connect log data to Grafana using TLS data source plugin.

Background information

As an open-source visualization monitoring platform, Grafana offers features like data metrics monitoring and visual analysis. It is commonly used for large-scale metrics data and time series data visualization, especially in scenarios such as network architecture and application performance analysis. Once log data is collected by TLS, you can use the provided data source plugin to integrate log data with Grafana. In Grafana, you can create visual dashboards and charts for log data analysis.

The data source plugin provided by TLS supports using VolcanoEngine access keys to retrieve all log topics under the current account, as well as fetching only specific log topic data. The configuration methods for the two data sources are described in the following table. Select the appropriate method based on your business needs.

Mode	Description	Advantages	Disadvantages
Specify only the access key method.	When using the current data source, you can view all log topic data under the current account. If new log topics are added under the current account, they will be automatically synced to the corresponding data source, enabling dynamic expansion of log topics to Grafana.	High flexibility Easy maintenance	Security risk Reduced performance
Specify the log topic method.	When using the current data source, you can only view the log topic data specified for it.	High security Strong controllability	Large configuration workload Low maintenance cost

Prerequisites

Logs have been collected. For steps, see Overview .

Grafana has been installed. For detailed steps, see the official Grafana documentation.

TLS supports integration with Grafana versions 6.0 to 12.x (dashboard import supports versions 8.0.x to 12.x). This topic uses Grafana 9.5.8 as an example to describe related steps.

Add data source steps

1. Download and install the data source plugin

Download the data source plugin package. For more details, see Github repository.

Grafana version 9.0 and later

wget https://codeload.github.com/volcengine/tls-grafana-datasource/zip/refs/heads/main -O tls-grafana-datasource-main.zip

Grafana versions 6.x–8.x

wget https://codeload.github.com/volcengine/tls-grafana-datasource/zip/refs/tags/v1.2 -O tls-grafana-datasource-main.zip

Extract the data source plugin package to the Grafana plugin directory.

Select the corresponding Grafana plugin directory and run the appropriate command based on your installation method.

Grafana installation methods	Plugin directory	Unzip command example
Install using RPM or YUM	/usr/lib/grafana/plugins	unzip tls-grafana-datasource-main.zip -d /var/lib/grafana/plugins
Install using .tar and .gz files.	{PATH_TO}/grafana-6.6.0/data/plugins:	unzip tls-grafana-datasource-main.zip -d {PATH_TO}/grafana-6.6.0/data/plugins

Edit the permissions in the Grafana configuration file.

Add permission parameters for the data source plugin in the Grafana configuration file to allow Grafana to load this plugin.

Tip

Grafana 6.x by default allows loading unsigned plugins; no additional configuration is required.

Grafana installation methods	Plugin directory	Plugin configuration
Install using RPM or YUM	:/etc/grafana/grafana.ini	allow_loading_unsigned_plugins = tls-grafana-datasource
Install using .tar and .gz files.	:{PATH_TO}/grafana-6.6.0/conf/defaults.ini	allow_loading_unsigned_plugins = tls-grafana-datasource

Restart Grafana.

Run the following command to restart Grafana and apply the plugin configuration.

Run the kill command to terminate the Grafana process.

Restart Grafana.

Select the appropriate command based on your installation method to restart Grafana.

Grafana installation methods
Restart command
Install using RPM or YUM

systemctl restart grafana-server

Install using .tar and .gz files.

./bin/grafana-server

2 Add data source

Description

The Grafana interface may vary by version. Refer to the actual interface for accuracy.

Add TLS as a data source in Grafana.

On the left menu, select Administration > Data sources.

On the Data sources tab, click Add new data source.

On the Add data source page, locate and click Tls-Grafana-Datasource.

Fill in the data source configuration.

Set the data source name.

Select the configuration mode.

Configure the data source information.

Specify only the access key method (Account mode)

Enable AccountMode and configure the following parameters:

Configuration	Description
AccessKeyId	Volcengine account Access Key ID. You can view and copy your Access Key ID on the Key Management page of the Volcengine console. Tip We recommend using an IAM user's secret key, and ensure the IAM user has permission for Action SearchLogs. For more details, see Authorizable operations.
AccessKeySecret	Secret Access Key of Volcengine account. You can view and copy the Secret Access Key on the Key Management page in the Volcengine console by following the instructions on the page.
Endpoint (Optional)	Endpoints refer to the service address of specified TLS under the access account, e.g., https://tls-cn-beijing.ivolces.com. Select the correct service entry based on the region and network type. For details, see Service address. Tip The service address must start with https://. If a private network endpoint is used, verify that the environment where the Grafana project is deployed can access TLS via VolcanoEngine private network.
Region (Optional)	The region ID of the specified TLS under the access account. This region should be the same as the region of endpoint. For details, see Service address.

Specify the log topic method (Single Topic mode)

Enable AccountMode and configure the following parameters:

Configuration	Description
Endpoint	The TLS service address, such as https://tls-cn-beijing.ivolces.com. Select the correct service entry point based on region and network type. For details, see Service address. Tip The service address must start with https://. If you use a private network service address, ensure that the environment hosting the Grafana project supports accessing TLS via VolcanoEngine private network.
Region	The TLS region ID must match the region of the Endpoint. For more information, see Service address.
Topic	The ID of the TLS log topic where log data is stored.
AccessKeyId	Volcengine account Access Key ID. You can view and copy your Access Key ID on the Key Management page of the Volcengine console. Tip We recommend using an IAM user's secret key, and ensure the IAM user has permission for Action SearchLogs. For more details, see Authorizable operations.
AccessKeySecret	Secret Access Key of Volcengine account. You can view and copy the Secret Access Key on the Key Management page in the Volcengine console by following the instructions on the page.

Click Save & test.

Step 2: Add dashboard actions

Import TLS dashboard

1. Obtain the data source UID

In the Data sources tab in Grafana.

Click the data source you added in the previous step, and get the data source UID from the URL bar at the top.

2 Get Grafana-format dashboard

Refer to 2 Get Grafana-format dashboard to generate a Grafana-format dashboard JSON in the TLS console.

3 Import a Grafana-format dashboard

In the left navigation pane in Grafana, click Dashboards.

In the top right corner of the page, select New > Import.

On the Import dashboard tab, upload the Grafana-format dashboard JSON file, or paste the JSON directly into Import via dashboard JSON model, and then click Load.

Add dashboards manually

1 Add a dashboard

In the left navigation pane, click Dashboards.

In the upper right corner of the page, select New > New Dashboard.

In the upper right corner of the New dashboard page, click the Settings icon.

On the General page, set the name of the dashboard.

In the upper right corner of the page, click Save dashboard, and then click Save.

2 (Optional) Add variables

In the upper right corner of the dashboard page, click the Settings icon.

In the left navigation bar, click Variables.

On the Variables page, click New variable.

Set the parameters for the variable. Keep other parameters as default, and then click Apply.

Configuration	Description
Select variable type	Select the variable type. Only the Query type supports TLS data source configuration.
Name	Set the variable name.
Data source	Select the target TLS data source.
Query	Enter the query and analysis statement to dynamically retrieve variable values. Grafana uses the query and analysis results from the selected data source as options for the variable.

Configuration examples of querying variables via specifying access key (Account mode)

Tip

If the added data source uses the specified Access Key method (Account mode), you can configure custom variables such as Region, Topic and Log via query type to implement multi-Topic queries across regions or within the same region.

Custom variables	Parameters configuration
region	This parameter is used to configure TLS region variables. query type: set it to Region.
accesslog	Refers to the variables that are used to configure TLS log topic, e.g., log topic accesslog. query type: set this to Topic. region variable: set this to the custom variable you create in the last step, e.g., $region. projectName: configure log project name, such as internal-log. If no name is configured, all projects within the region will be used by default. topicName:configure log topic name, such as access-log. If no name is configured, all topics within the project will be used by default.
account_id	Refers to the variables that are used to extract the specified value. query type: set this to Logs. region: set this to the custom variable you created in the previous step, e.g., $region (or any specified Region). topic: set this to the topic variable you configured in the previous step, e.g., $access-log (or any specified Topic). Query: set this to SQL statement that are used to extract specified value, e.g., * \| select distinct account_id.

In the upper right corner of the page, click Save dashboard, and then click Save.

After adding variables, you can view variables and their values in the upper right corner of the chart page.

3 Add chart

After adding a data source, you need to create a dashboard in Grafana and add charts to it.

On the dashboard page, click Add visualization.

On the Select data source page, select the TLS data source you created in 2 Add data source.

Fill in the relevant configuration for query and analysis.

Select the region (region) and log topic (topic) for the data source.

Description

When configuring the data source by specifying access key only, you need to select the target region and log topic.

Configuration parameters
Description
region
Select the region where the TLS log topic is located.
topic
Select the TLS log topic where the data source is located.

Enter your query and analysis statement in Query.

The query and analysis statement should follow the syntax of Volcengine TLS query and analysis. For more details, see Query overview and Analysis overview. When filtering chart data with variables, you need to add variable placeholders in the query and analysis statement. The format is Field name: ${Variable name}.

Set xcol and ycol.

Configuration parameters	Description
xcol	Set the display field for the X-axis. When using a table chart, you must set xcol to Table/Log. When a bar chart is used, xcol is required to set as bar. When using a time series chart, you must set xcol to TimeSeries/Custom. With time series charts, you can expand a dimension into multiple metrics and use multiple line charts to display the trends of different metrics over time. For details, see Example 2. When a pie chart is used, xcol is required to set as pie.
ycol	Set the display fields for the Y axis.

Example without Variable

Example with Variable

In the query and analysis statement, insert the variable Method:${Method} and then you can filter chart data using the variable.

Configure the chart.

Select the chart type on the right side of the page.

Configure Panel options based on business requirements.

At the upper right corner of the page, click Save. In the Save dashboard panel, click Save.

If you want to continue adding visualization charts, at the upper right corner of the dashboard, click Add > Visualization.

Chart configuration examples

TLS provides chart configuration examples for common scenarios for your reference.

Bar gauge

Bar charts are typically used to display categorized data, with each category represented by a bar whose height reflects the value for that category. For steps to add a bar chart, see 3 Add chart. The configuration example for a bar chart is as follows:

Scenarios

Use a bar chart to display the PV and UV for each request method.

Parameter configuration

Configuration parameters	Description
region	Select the region where the TLS log topic is located, such as Beijing.
topic	Select the TLS log topic for the data source, such as mytopic.
Query	Enter the following query and analysis statement. *\|select method,count(1) as PV,count(distinct account_id) as UV group by method
xcol	When using a bar chart, set xcol to bar.
ycol	Set to method, PV, UV.

Visual chart

Table

Tables can be used to display or compare values of the same type of fields, such as querying user operation times in access logs and similar scenarios. For steps to add a table, see 3 Add chart. The table configuration example is as follows:

Scenarios

Display PV and UV for every 5 minutes using a table.

Parameter configuration

Parameters	Example value
Data source	Select the TLS data source you added.
region	Select the region where the TLS log topic is located, such as Beijing.
topic	Select the TLS log topic for the data source, such as mytopic.
Query	Enter the following query and analysis statement. * \| select DATE_FORMAT(__time__ - __time__ % 300000, 'Y-M-d HH:mm') as time, count(1) as PV, count(distinct accountid) as UV group by time order by time
xcol	When using a table, xcol must be set to Table/Log.
ycol	No required to configure.

Visual chart

Time series chart

Time series charts show data trends at different time points, such as viewing access trends over a period. For steps to add a time series chart, see 3 Add chart. The following is a configuration example for a sequence diagram:

Example 1

Scenarios

The sequence diagram shows the trend of PV changes within each 5-minute interval.

Parameter configuration

Configuration parameters	Example value
Data source	Select the TLS data source you added.
region	Select the region where the TLS log topic is located, such as Beijing.
topic	Select the TLS log topic for the data source, such as mytopic.
Query	* \| select (__time__ - (__time__ % 30000)) as time,count(1) as PV, count(distinct account_id) as UV group by time
xcol	When using a time series chart, you need to set xcol to TimeSeries/Custom.
ycol	Set as PV, UV.

Visual chart

Example 2

Scenarios

The sequence diagram displays the IOPS changes of each cluster over time.

Parameter configuration

Configuration parameters	Example value
Data source	Select the TLS data source you added.
region	Select the region where the TLS log topic is located, such as Beijing.
topic	Select the TLS log topic for the data source, such as mytopic.
Query	Action: PutLogs AND newArch:true \| select Tls2Cluster, (__time__ - (__time__ % 60000)) as time,sum(ReqSize)/60 as num group by Tls2Cluster, time order by time, num DESC limit 1000000
xcol	When using time series diagrams, set xcol to TimeSeries/Custom and set the specific fields to time,Tls2Cluster. Notice When using a timeline chart, you can expand a dimension into multiple metrics. Configure xcol as time, query field, and set ycol to the corresponding value columns (multiple value columns supported). The system will automatically add the relevant metrics, naming them as query field value * value column name. Each metric corresponds to a line, with a maximum of 100 lines. For example, in this case, xcol is set to time, Tls2Cluster, and ycol is set to num. The metrics generated based on Tls2Cluster field values are newarchnum, tls-newarch-az-bnum, and tls-newarch-az-b-2*num. Then, how these metrics change over time will be displayed.
ycol	Set it to num.

Visual chart

Gauge chart

Gauge charts highlight query and analysis results in rectangular frames. They are ideal for prominently displaying data, such as key information at a specific moment or during a certain period. For steps to add a gauge chart, see 3 Add chart. Configuration examples for gauge charts are as follows:

Scenarios

Display PV and UV using gauge charts.

Parameter configuration

Configuration parameters	Example value
Data source	Select the TLS data source you added.
region	Select the region where the TLS log topic is located, for example, Beijing.
topic	Select the TLS log topic for the data source, such as mytopic.
Query	* \| select count(1) as PV, count(distinct accountid) as UV
xcol	When using a single-value chart, xcol must be set to stat/Gauge.
ycol	Set as PV, UV.

Visual chart

Pie chart

Pie charts show the proportion of different categories and compare differences using arcs, highlighting each category's value relative to the total. For instructions on adding a pie chart, see 3 Add chart. Pie chart configuration examples:

Scenarios

Display the number of requests for each status with a pie chart.

Parameter configuration

Configuration parameters	Description
Data source	Select the TLS data source you added.
region	Select the region where the TLS log topic is located, for example, Beijing.
topic	Select the TLS log topic for the data source, such as mytopic.
Query	status>200 \| select status,count(*) cnt group by status
xcol	When using a pie chart, set xcol to pie.
ycol	Set as status,cnt.

Visual chart

map

Notice

The TLS Grafana plugin does not provide map functionality; maps are supported by Grafana.

The map shows regional distribution. For instructions on adding a map, see 3 Add chart. Currently, you can configure maps through the Echarts plugin and Worldmap Panel. The configuration examples are as follows:

Worldmap Panel

Scenarios

Display the distribution of request quantities across regions using the map.

Parameter configuration

Configuration parameters	Description
Data source	Select the TLS data source you added.
Query	* \| select ip_to_city(clientip) as city, geohash(ip_to_geo(clientip)) as hash, count(*) as count group by city, hash order by hash limit 100000 Note: For the ip_to_geo function description, see IP_TO_GEO function. For the ip_to_city function description, see IP_TO_CITY function. For the geohash function description, see GEOHASH function.
ycol	Set to city, hash, count.

Visual chart

Echarts plugin

Install and configure the Echarts plugin before using it. For more details, see the Echarts plugin.

Scenarios

Display the distribution of request quantities across regions using the map.

Parameter configuration

Configuration parameters	Description
Data source	Select the TLS data source you added.
Query	* \| select count(1) as pv ,ip_to_geo(arbitrary(RemoteAddr)) as geo,IP_TO_PROVINCE(RemoteAddr) as city where IP_TO_COUNTRY(RemoteAddr,'en')='CN' group by city limit 100 For the IP_TO_PROVINCE function description, see IP_TO_PROVIDER function.

Visual chart

Examples of viewing raw log configuration

Using the TLS Grafana plugin, you can view raw logs directly within Grafana dashboards. The steps are shown below:

Configure dashboard Visualization parameters, and select Logs.

Configure dashboard Query parameters, and select Logs in xcol field.

Select Data source and enter query statements. Then you can view raw logs.

For query syntax details, see Query syntax.

For example, you can enter "1e1589926ae14140812ab4f2b6c0ac34" to query raw logs that contain this ID.

Last updated: 2026.08.05 18:23:57

Grafana installation methods	Restart command
Install using RPM or YUM	systemctl restart grafana-server
Install using .tar and .gz files.	./bin/grafana-server