Docs

Torch Log Service

Associate external data source for log query

Associate to MySQL data source

TLS query and analysis feature supports associating external data storage, which means you can associate log data with a MySQL database during log queries. This topic outlines the steps to associate to a MySQL data source.

Background information

In a real production environment, fact tables such as system logs and sales records, which relate to production events and behaviors, and dimension tables storing core business metadata, are often kept separately in different systems or databases. For example, in e-commerce scenarios, user activity and event data—such as page visits, product purchases, payment methods, and network types—are stored as logs in TLS, while user registration and membership information are maintained in database systems for data security and are not included in the logs.

In log analysis scenarios, fact tables and dimension tables can be associated, using the raw data stream as the basis for queries and analysis, and linking numerous external tables to add detailed information and attributes. Based on these scenarios, TLS query and analysis feature provides data source association, supporting joint query and analysis of log topics and external MySQL data sources. When querying data in TLS, you can use the JOIN clause in SQL to connect log data from log topics with data in MySQL databases, expanding the scope of query and analysis.

Description

External data sources are now in closed beta. If you have related needs, contact your account manager for allowlist access.

There is currently no charge for external data source features. We will release related notes on billing time and billing rules in the upcoming product announcement. For details, please stay tuned.

Notice

Category	Description
Data security	For data security, the database account used to access MySQL for TLS should be granted only the minimum necessary permissions. TLS only requires SELECT and INSERT permissions. No other permissions are required. We recommend assigning a dedicated database account for joint queries and keeping account details secure.
MySQL database connection address	When building a MySQL database in Volcanoengine ECS, set the connection address to 0.0.0.0 to allow all servers to access the MySQL database. Then use ECS security groups to manage access permissions and enhance database security.
MySQL database user	By default, the root user of the MySQL database can only log in to local device. Therefore, you need to create a new user for remote access or enable remote access for the root user.
MySQL database allowlist	The allowlist is a database security measure. Only IP addresses on the allowlist can access the database. Before associating with MySQL data source, you need to configure the allowlist on the MySQL data source side to allow TLS-related IP addresses to access it. For more details, see 1 Configure allowlist.

Limits

Items	Description
External data source	MySQL data sources support Cloud database MySQL on Volcanoengine, as well as self-built MySQL databases on Volcanoengine ECS. Plus, Cloud Database MySQL instances or ECS instances should be in the same region as the TLS log project.
MySQL version	Supported MySQL versions include 5.7 and 8.0.
MySQL table size	It is recommended that the amount of data in MySQL tables used for join queries be less than 20 MiB. Large datasets and complex SQL queries from TLS may impact database stability.
External data source quota	You can create up to 50 external data sources under each project.

Steps

1 Configure allowlist

Before associating a MySQL data source, set the allowlist on the MySQL data source to allow access from the IP addresses used by TLS. Add the following IP addresses to the allowlist:

Data source	IP address	Description
Volcanoengine Cloud database MySQL	10.0.0.0/8 33.0.0.0/8 100.64.0.0/10	Add the IP address field to the database allowlist. For steps, see Create allowlist .
Self-built MySQL database in Volcengine ECS	10.0.0.0/8 33.0.0.0/8 100.64.0.0/10	If ECS has a security group configured, you should add a rule to allow access from the specified IP range. For detailed steps, refer to Modify security group access rules.

2 Create MySQL data source association

Select the region where log service is located from the top navigation bar.

On the left navigation bar, select Resource Management > Project.

Click the specified project name. Then on the External data source tab, click Create data source association.

Enter the basic configuration for the data source.

Self-built database

Configuration	Description
Data source name	External data source name. The following requirements must be met: Be unique within the same project. Supports lowercase letters, numbers, Chinese characters, underscores (_), and hyphens (-). Must start and end with a lowercase letter, Chinese character, or number. Length must be between 3 and 64 characters.
Type	Type of external data source. Only supports configuring MySQL, i.e., data source.
Database type	The type of database. In this case, self-built database is used as an example, i.e., the MySQL database set up on Volcanoengine ECS.
Connection address	ECS private IP address
Port	Port number bound to the MySQL service.
VPCId	VPC ID where the ECS instance is located.
Database username	MySQL database username.
Database password	MySQL database user password.
Database	MySQL database name.
Database table	Name of the MySQL database table.

Volcanoengine Cloud database MySQL

Configuration	Description
Data source name	External data source name. The following requirements must be met: Be unique within the same project. Supports lowercase letters, numbers, Chinese characters, and hyphens (-). Must start and end with a lowercase letter, Chinese character, or number. Length must be between 3 and 30 characters.
Type	Type of external data source. Only supports configuring MySQL, i.e., data source.
Database type	The type of database. This example uses VolcEngine RDS for MySQL. That is, VolcEngine Cloud Database MySQL.
Select region	The region where the Cloud database MySQL instance is located. It should match the region of the current project.
Select AZ	Instance availability zone of Cloud database MySQL .
Select instance	Instance name of Cloud database MySQL .
Connection address	Cloud database MySQL instance private network address.
Port	Cloud database MySQL instance private network port number. The corresponding port number will be automatically filled when the connection address is specified.
VPCId	The instance's VPC of Cloud database MySQL . The corresponding VPC is automatically filled in when you specify the instance.
Database username	The instance's database username of Cloud database MySQL .
Database password	The instance's database user password of Cloud database MySQL .
Database	The instance's database name of Cloud database MySQL .
Database table	The instance's database table name of Cloud database MySQL .

Click Test connection.

TLS automatically tests connectivity between the server and the MySQL data source.

After the test is done, click Create Data Source Association.

Related operations

After the data source association is created, you can also view, modify, or delete it. For details, see the table below.

Related operations	Description
View external data sources	You can view all associated external data sources under the current log topic in the External data sources tab on the project details page. Click the data source name to view its details.
Edit external data source settings	To modify the settings for the selected external data source, click Edit in the Actions column of the External data sources list. Modifying the type or database type of a data source is not supported.
View data source connection status	You can check the connection status in the Connection status column of the External data sources list. If the connection fails, you can update the settings and click Retry.
Delete external data source	If you no longer need a data source, you can click Delete in the action column of External data sources list.
External data source joint analysis	During query and analysis, use the JOIN clause to combine log topics with MySQL databases for joint queries and analysis. You can also save the query results to a MySQL database. For details, see MySQL join query and analysis.

Last updated: 2026.07.30 17:29:25