Call the CreateImportTask API to create a data import task.

Request instructions

Request method: POST

Request URL: https://{tls-endpoint}/CreateImportTask

Request parameters

The table below lists interface-specific and some common request parameters only. See Common Parameters for more information.

Body

Parameters	Type	Required	Example value	Description
Description	String	NOT	this is a task	Data import task description.
ImportSourceInfo	Object of ImportSourceInfo	Yes	/	Source information for data import tasks
ProjectID	String	NOT	fc69****	Project ID used for storing data.
SourceType	String	Yes	tos	Source type for import. tos: Import TOS data. kafka: Import Kafka data.
TargetInfo	Object of TargetInfo	Yes	/	Output information for TOS data import tasks.
TaskName	String	Yes	task-name-1	Data import task name.
TopicID	String	Yes	fa************	Log topic ID for storing data.

ImportSourceInfo

Parameters	Type	Required	Example value	Description
TosSourceInfo	Object of TosSourceInfo	NOT	/	TOS source import information. When sourceType is tos, the TosSourceInfo field is required.
KafkaSourceInfo	Object of KafkaSourceInfo	NOT	/	Kafka source import information. When sourceType is kafka, the KafkaSourceInfo field is required.

TargetInfo

Parameters	Type	Required	Example value	Description
Region	String	Yes	cn-shanghai	Region ID.
LogType	String	Yes	json_log	Specify the log parsing type when importing. delimiter_log: CSV type. multiline_log: Multiline full text type. minimalist_log: Single-line full text type. json_log: JSON type.
LogSample	String	NOT	/	Log sample. When you set LogType to multiline_log, you need to configure log samples. We recommend providing at least two log samples to ensure the regular expression matches the first line of each log. We suggest that you use real samples from the production environment.
ExtractRule	Object of ExtractRule	NOT	/	Log extraction rule.
UserDefineRule	Object of UserDefineRule	NOT	/	Custom extraction rules.

TosSourceInfo

Parameters	Type	Required	Example value	Description
bucket	String	NOT	import01	TOS bucket that contains log files.
prefix	String	NOT	accesslog/	The path of the file to be imported in the TOS bucket.
region	String	NOT	cn-shanghai	Region where the TOS bucket is located. Supports cross-region data import.
compress_type	String	NOT	snappy	Compression mode for data in the TOS bucket. none: No compression. snappy: Compress using snappy. gzip: Compress using gzip. lz4: Compress using lz4. tar.gz: Use tar.gz for compression. This feature is in closed beta. If you have relevant business needs, you can contact your account manager to enable the allowlist.
import_archived	Boolean	NOT	false	Check if the file you import from the TOS bucket is an Archive or Cold Archive file. true: This is an Archive or Cold Archive file. false: This is not an Archive or Cold Archive file.

KafkaSourceInfo

Parameters	Type	Required	Example value	Description
host	String	NOT	kafka-cnngc7an0qpv****.kafka.ivolces.com:9092	Different types of Kafka clusters have different service addresses. Details are provided below: Volcengine Message Queue for Kafka: the access point of the Kafka instance. For more information, see View access point. If the Kafka instance and the TLS Project are in the same region, you can use the private network to access. Otherwise, you need to use public network to access. Self-hosted Kafka cluster: IP address and port or domain name and port of the Kafka broker. Only supports public network. Multiple service addresses need to be separated with commas (,).
group	String	NOT	mygroup	Kafka consumer group. If left blank, the system will automatically create a Kafka consumer group.
topic	String	NOT	test	Kafka Topic name. Multiple Kafka Topic need to be separated with commas (,).
encode	String	NOT	UTF-8	Encoding format of the data. Optional values include UTF-8 and GBK.
password	String	NOT	/	Kafka SASL user password for authentication
protocol	String	NOT	plaintext	Secure transmission protocol. Options include plaintext, sasl_ssl, ssl, and sasl_plaintext.
username	String	NOT	/	Kafka SASL username used for authentication.
mechanism	String	NOT	PLAIN	Password authentication mechanism; available options include PLAIN, SCRAM-SHA-256, and SCRAM-SHA-512.
instance_id	String	NOT	kafka-cnngbnntswg1****	If you are using Volcengine Message Queue for Kafka, set this value to the Kafka instance ID.
is_need_auth	Boolean	NOT	true	Enable authentication. If you are using a public service address, we recommend enabling authentication.
initial_offset	Integer	NOT	/	Starting position for data import. Optional values: 0: Earliest time, which means import starts from the first data in the specified Kafka Topic. 1: Latest data imports start from the most recently generated data in the specified Kafka Topic.
time_source_default	Integer	NOT	/	Specify the log time. Optional values: 0: Use Kafka message timestamp. 1: Use the current system time.

ExtractRule

Parameters	Type	Required	Example value	Description
Keys	Array of String	NOT	[ "time", "", "level", "msg" ]	List of log field names (Key). Valid only when LogType is delimiter_log. You can configure up to 100 field names. Duplicate field names are not allowed, and all field names cannot be left empty.
Quote	String	NOT	"	Quotation marks. Content wrapped by quotation marks will not be separated, but will be parsed as a complete field. Valid only when LogType is delimiter_log.
TimeKey	String	NOT	request_time	The field name for the log timestamp. If you use a specific time field in the log as the log timestamp, you need to provide TimeKey and TimeFormat. TimeKey and TimeFormat must be used together.
TimeZone	String	NOT	GMT+08:00	Time zone: Supports Host time zone (default) and Custom time zone. Custom time zone includes GMT and UTC. GMT format: GMT+08:00. UTC format: Asia/Shanghai.
Delimiter	String	NOT	#	Log separator. Valid only when LogType is delimiter_log.
BeginRegex	String	NOT	[(\d+-\d+-\w+:\d+:\d+,\d+)]\s[(\w+)]\s(.*)	This regular expression is used to identify the first line in each log entry, and the matched part is identified as the start of the log. When LogType is set to multiline_log, you need to provide log samples.
TimeFormat	String	NOT	%Y-%m-%dT%H:%M:%S,%f	Parsing format of the time field. If you use a specified time field in the log as the log timestamp, you must specify both TimeKey and TimeFormat. TimeKey and TimeFormat must be used together. For configuring time formats, see Time Format.
SkipLineCount	Integer	NOT	2	Number of skipped log lines. For example, if you set it to 1, logs will be imported starting from the second line in the CSV file. This is valid only when the collected log type is delimiter_log and the import type is tos.
UnMatchLogKey	String	NOT	LogParseFailed	When uploading logs that failed to parse, the key name for those logs is used. UnMatchUpLoadSwitch=true and UnMatchLogKey must be used together.
TimeExtractRegex	String	NOT	[0-9]{0,2}\/[0-9a-zA-Z]+\/[0-9:,]+	Regular expression for extracting time, used to extract the time value from the TimeKey field and parse it as the corresponding collection time
UnMatchUpLoadSwitch	Boolean	NOT	true	Upload logs that failed to be parsed. UnMatchUpLoadSwitch=true and UnMatchLogKey must appear together. true: Upload logs that failed to be parsed. false: Do not upload logs that failed to be parsed.
TimeSample	String	NOT	2006-01-02T15:04:05Z07:00	Time example. Used to check whether the entered time parsing format is correct.

UserDefineRule

Parameters	Type	Required	Example value	Description
Plugin	Object of Plugin	NOT	/	Plugin configuration for LogCollector. After enabling plugin configuration, you can add one or more LogCollector processor plugins to parse logs with complex or inconsistent structures.

Plugin

Parameters	Type	Required	Example value	Description
processors	Array of JSON Map	NOT	{"processors":[{"json":{"field":"_source","allow_overwrite_keys":true,"allow_empty_values":true,"trim_keys":{"mode":"all","chars":"\""},"trim_values":{"mode":"all","chars":"\""}}}]}	LogCollector plugin. For the list of supported plugins and parameter details, see LogCollector Plugin Overview.

Response parameters

The table below lists only the unique response fields for this API. For more details, see Response structure.

Parameters	Type	Example value	Description
TaskId	String	fa************	Import task ID.

Sample request

Return example

Error Code

You can visit public error codes for more information on error codes.

Torch Log Service