Docs

Torch Log Service

expression function

resource function

This document describes the syntax format and typical examples of data processing resource functions.

res_rds_mysql function

Attention

The maximum number of data pull is 200,000, more than 200,000 data will be truncated.

res_rds_mysql function is used to pull table data from Volcano Engine cloud database MySQL edition and self-built public MySQL database. The following three pull methods are supported:

pull mode	explain
normal pull mode	When the machining task is executed for the first time, the log service will pull the full amount of table data and will not pull it later. If the data in your database is not continuously updated, you can choose the normal pull mode.
full update mode	During the execution of a machining task, the log service will periodically pull the full amount of table data once. Each time the full amount is pulled, it is very time-consuming. If the amount of data in your database is small (data volume < = 2GB) and updates are infrequent (configuration refresh_interval > = 300s), you can choose the full update mode.
incremental update mode	During the execution of a machining task, the logging service periodically pulls new data based on the timestamp field in the database & table. In the incremental update mode, the log service only pulls new data each time, which is efficient. You can also configure the refresh_interval to 10 seconds to synchronize data in the database & table at the second level with high immediacy. If you have a large amount of data in your database & tables, updates are frequent, and you have high data real-time requirements, you can choose the incremental update mode.

Function syntax/formatting

res_rds_mysql(host="data base address",port="data base port", username="data base username", password="data base password", database="data base name", table="table name", vpc=None, fields=None, include_fields=None, exclude_fields=None,primary_keys=None, refresh_interval=0, update_time_key=None,base_retry_back_off=1, max_retry_back_off=60)​

The parameters are described as follows:

parameter	Parameter type	Is it required?	default value	explain
host	String	Yes	/	MySQL instance domain name, currently supports Volcano Engine cloud database MySQL version and self-built public network database. When using Volcano Engine Cloud Database MySQL Edition, set the host to the IP address of the VPC to which the MySQL instance belongs. You can view the IP address of the target VPC on its overview page. For details, please refer to View Private Network Details .
port	String	Yes	/	Database access port.
username	String	Yes	/	Database access user name.
password	String	Yes	/	Database access password.
database	String	Yes	/	Database name.
table	String	Yes	/	Data database & table name.
vpc	String	No	/	When using Volcano Engine Cloud Database MySQL Edition, you need to configure the VPC ID used by the MySQL instance, such as vpc-2byx **** .
fields	String	No	All fields	List of strings or string aliases to be pulled. If left blank, all columns returned by the table parameter are used by default. Use half-width commas (,) when specifying multiple fields, such as field1, field2 .
include_fields	String	No	/	Configure field whitelist. Table data matching the include_fields is pulled and the rest is discarded. Single field names and values are joined with a half-width colon (:), and multiple fields are separated by a half-width comma (,), such as field1: value1, field2: value2 .
exclude_fields	String	No	/	Configure field blacklist, matching exclude_fields table data will be discarded, the rest of the data is pulled. Single field names and values are concatenated with colons (:), and multiple fields are separated by commas (,), such as field1: value1, field2: value2 . explain If you set both include_fields and exclude_fields parameters, the system preferentially matches include_fields parameters and then exclude_fields parameters.
primary_keys	String	No	/	Primary Key. After configuring this parameter, the data in the table will be pulled as key: Value in dictionary form and saved to memory. The key is the value of the primary_keys parameter, and the value is the entire row of data pulled into the table. When the amount of data in the database & table is large, it is recommended to use this parameter to improve efficiency. primary_keys The value of the parameter must exist in the field pulled from the data database & table.
update_time_key	String	No	/	For incremental data acquisition. For example update_time_key = "update_time" , where update_time is the updated time field of the data in the MySQL database, which supports the Datetime, Timestamp data types. Please ensure that the value of this time field increases over time. If this parameter is not configured, the logging service will perform a full update. explain Data processing does data filtering based on the time field to achieve incremental updates. Please ensure that the field is indexed in the data table, otherwise a full table scan will be performed, which will greatly affect performance.
refresh_interval	Int	No	0	The time interval between pulling data from the MySQL database, in seconds. The default value is 0, indicating that the full amount is pulled only once.
base_retry_back_off	Int	No	1	The time interval for re-pulling after failure to pull data, the default value is 1, in seconds.
max_retry_back_off	Int	No	60	The maximum time interval to retry a request after a failed data pull. The default value is 60, in seconds. It is recommended to use the default value.

Function example

Example 1: All data is pulled every 300 seconds

Scenario: Pull summary table data from table_tls table in Volcano Engine MySQL database test_sql every 300 seconds.

Processing rules:

res_rds_mysql(
host="192.168.**.**",
port="3306",
username="test_username",
password="****",
database="test_sql",
vpc="vpc-xxxxx",
table="table_tls",
refresh_interval=300
)

Example 2: Specifying a black and white list for data pulling

Scenario: Pull data from the table_tls table in the Volcano Engine MySQL database test_sql, and specify that only the size field value is 400, and the method field value is not put data.

Processing rules:

res_rds_mysql(
host="192.168.**.**",
port="3306",
username="test_username",
password="****",
database="test_sql",
vpc="vpc-xxxxx",
table="table_tls",
refresh_interval=300,
include_fields='size:400',
exclude_fields='method:put'
)

Example 3: Pull data using primary key mode

Scenario: Use the primary key schema to pull data from table_tls tables in the Volcano Engine MySQL database test_sql.

Processing rules:

f_table_map(
res_rds_mysql(
address="192.168.**.**",
username="test_username",
password="****",
database="test_sql",
vpc="vpc-xxxxx",
table="table_tls",
primary_keys="id",
),
"id",
["name", "desc"],
)

database data

id
name
desc
1
zhangsan
Man out of law

Sample log:

[
{
"id":1,
"age":10
},
{
"id":2,
"age":20
}
]

Processing result:

[
{
"id":1,
"age":10,
"name":"zhangsan",
"desc":"Man out of law"
},
{
"id":2,
"age":20
}
]

Example 4: Pull data using incremental update mode

Scenario: Use incremental update mode to pull data from table_tls tables in the Volcano Engine MySQL database test_sql.

Attention

When using incremental update mode to pull data, the following conditions must be met:

The data database & table must have a unique primary key and a time field, such as id and update_time in the example.

When you use incremental update mode to pull data, you must set the primary_keys, refresh_interval, and update_time_key parameters in the function.

Processing rules:

f_table_map(
res_rds_mysql(
address="192.168.**.**",
username="test_username",
password="****",
database="test_sql",
vpc="vpc-xxxxx",
table="table_tls",
primary_keys="id",
update_time_key="update_time",
refresh_interval=10
),
"id",
["name", "desc"],
)

database data

id
name
desc
update_time
1
zhangsan
Man out of law
2024-07-08 14:23:14
2
lisi
Hey man, what can i say!
2024-07-08 14:23:15

Sample log:

[
{
"id":1,
"age":10
},
{
"id":2,
"age":20
}
]

Processing result:

[
{
"id":1,
"age":10,
"name":"zhangsan",
"desc":"Man out of law"
},
{
"id":2,
"age":20，
"name":"lisi",
"desc":"Hey man, what can i say!"
}
]

Last updated: 2026.05.12 10:39:54

id	name	desc	update_time
1	zhangsan	Man out of law	2024-07-08 14:23:14
2	lisi	Hey man, what can i say!	2024-07-08 14:23:15