Intelligent Data Insight
Intelligent Data Insight
- Documentation
Intelligent Data InsightDataWind User GuideData PreparationVisual ModelingData OutputOutput to Kafka
Copy page
Download PDFCopy pageDownload PDFThe Output to Kafka operator supports sending data processed by real-time tasks to downstream systems. This operator is suitable for scenarios where cleansed data needs to be sent in real time to a Kafka cluster for downstream real-time computing (such as Flink/Spark Streaming), monitoring and alerting, or data archiving.
- This operator currently supports outputting data only to external Kafka and can only be used in real-time tasks. You must first apply to enable the external output operator feature before you can use this operator.
- The data output format only supports JSON.
- Before starting configuration, ensure that your target Kafka cluster has been created, the data connection has been established, and the connectivity test for the data connection has been completed. For details, see Kafka.
Add the Kafka operator
- Click "Data Management", select "Visual Modeling", click "Create", and select "Create Real-Time Task".
- Follow the steps in (实时创建功能已取消,不发布)Create Real-Time Tasks to complete the creation of the real-time task. When configuring processing nodes, select "Output > Output to Kafka" and complete the relevant configuration.
Configure output to Kafka
Click "Create Output Table" and configure the Kafka output information. The specific parameter configuration instructions are as follows,
Parameters | Configuration instructions |
|---|---|
Data connection | You can select Kafka from the data connections. |
Topic | You can select the Topic to which subsequent data needs to be written from the dropdown list, with support for fuzzy search. If you do not find the corresponding Topic in the dropdown list, you can also directly enter the Topic name (within 64 characters) in the dropdown. The corresponding Topic will be automatically created for you when subsequent data is written. tip When directly creating a topic by entering the topic name, you must ensure that the corresponding Kafka instance has enabled the setting to allow automatic topic creation ( |
Advanced configuration
In most cases, you do not need to perform advanced configuration. If you encounter scenarios such as data write latency or require scaling, you can modify the advanced configuration here. This allows you to set runtime parameters for dataset synchronization as needed to ensure successful synchronization or improve synchronization performance. The supported parameters and their descriptions are shown in the following figure.
Parameters | Note |
|---|---|
parallelism.default | Default parallelism |
taskmanager.memory.process.size | Total process memory size of taskmanager |
yarn.containers.vcores | Number of virtual cores for each yarn container |
taskmanager.numberOfTaskSlots | Number of slots provided by taskmanager |
jobmanager.memory.process.size | Total process memory size of jobmanager |
yarn.appmaster.vcores | Number of virtual cores for appmaster in yarn |
Dirty data settings
In general, you do not need to configure dirty data settings. If you encounter data write errors, upstream data formats that do not match expectations, or need to monitor data in real time, you can enable the dirty data collection switch and set the dirty data collection frequency.
Parameters | Configuration instructions |
|---|---|
Enable dirty data collection | After enabling dirty data collection, the system will automatically collect dirty data. You can view dirty data details in task monitoring, which facilitates dirty data monitoring and analysis. For details, refer to the Task Monitoring section below. tip If you no longer need to collect dirty data, you can disable this switch again. |
Dirty data collection rate per second | Set the upper limit for the number of dirty data collected per second. |
After completing visual modeling, you can monitor dirty data and task running status on the task management page.
Configure monitoring rules
Click Monitoring Configuration. In the monitoring configuration panel that appears on the right, click Add Rule to add the corresponding monitoring rules.
Consumption monitoring: Supports configuring monitoring rules for input data consumption latency, write interruption, and write surge. For example, as shown in the figure, the consumption latency monitoring rule can be set to send a notification if there is no data for 20 consecutive batches between 8:00 and 20:00 each day.
Task monitoring: Supports configuring monitoring for this visual modeling task, including monitoring the quantity of dirty data and monitoring the task running status. For example, as shown in the figure, the monitoring rule for the quantity of dirty data can be set to send a notification when the number of dirty data entries exceeds 10,000.
Monitoring notifications: Supports configuring notification methods to send emails or mailing lists to specified users.
View task run charts
In the "Operation & Consumption Records" section of the task management page, you can view production and consumption statistics, dirty data charts, and detailed information about dirty data.
