Intelligent Data Insight
Intelligent Data Insight
- Documentation
Intelligent Data InsightDataWind User GuideData PreparationData ConnectionSystem MetadataMetadata Connection and Application
Copy page
Download PDFCopy pageDownload PDFAs Intelligent Data Insight is used more extensively and promoted within the company, it becomes increasingly important to monitor and manage the resources created by each user in the product. To address this, Intelligent Data Insight provides metadata connection and template application features, which allow all data assets in your Intelligent Data Insight environment to form a data center. This enables you to quickly view relevant information, monitor and manage resources, prevent misuse and abuse of resources, and also build core monitoring for certain key dashboards and datasets to better control data output and dashboard applications.
- Members of the IT or operations department responsible for on-premises deployment within the enterprise can use metadata to understand and analyze how analysts or business users use Intelligent Data Insight, and then further carry out operations-related tasks.
- Members of the IT or operations department, or those responsible for security monitoring or information security audit within the enterprise, can utilize metadata to build a security monitoring and information security audit mechanism, effectively protecting the security of enterprise data assets.
3.1 Workflow
3.2 Feature description
3.2.1 Metadata connection
In the data connection section of "Data Preparation," project administrators can select "DataWind Metadata" and click to create DataWind metadata. After creation, project administrators can query the system metadata they have permission for, based on their project permissions.
Metadata has two statuses:
- Available status: Click to create DataWind metadata. After creation, project administrators can query the system metadata they have permission for, based on their project permissions.
- Disabled status: The system contains all metadata, and only project administrators or group administrators can operate.
After clicking Test Connection, the interface indicates that the connection is successful. Then, click Save to enter the details page for this data connection.
3.2.2 Metadata data connection details
Currently, the metadata provided by Intelligent Data Insight includes the following four categories:
- Behavior event subject data: Records all user operation logs within different projects on the platform, including user behavior auditing, user activity analysis, resource popularity, and more. Fields such as resource ID, name, and creation time mainly pertain to resources like visualization modeling, datasets, charts, and dashboards. If there is no event information, it means the resource has not been accessed or operated.
- Asset detail subject data: Records detailed asset data for datasets, visualization modeling, charts, and dashboards. Datasets and visualization modeling provide information such as scheduling tasks, instance tasks, and underlying table TTL resources, enabling asset statistics, task management, resource management, and related analyses.
- Asset relationship subject data: Records the interdependencies among datasets, visualization modeling, charts, and dashboards, which can be used for asset analysis, asset queries, and lineage viewing.
- User subject data: Records user data under different projects, used for analyzing project changes, user changes, and other analysis data.
On the metadata data connection details page, you can view basic information and perform the following operations:
- View lineage: View downstream resources such as visualization modeling and datasets. Detailed information includes the names, IDs, types, owners, creation times, and update times of these resources.
- Permission assignment: Grant permissions to users, user groups, roles, departments, and more by searching.
- Edit: Modify the data connection name.
- Apply: Use metadata to create datasets or visualization modeling tasks.
- Delete.
3.2.3 Metadata application
You can use metadata to create datasets or visualization modeling tasks.
- Application entry: On the data connection details page, click "Application" in the upper right corner.
- Notes:
a. When creating a dataset, it is recommended to build one dataset for each subject metadata. The dataset can use direct connection or extraction mode. For extraction mode, check the data range for backfill; otherwise, the query may return no data. Direct connection mode allows querying all data in the project up to the present.
b. The scope of query permissions for the dataset follows the role permissions of the creator, not those of dashboard users.
c. If the owner is transferred and no further edits are made, the permissions remain unchanged.
d. Users must have edit permissions for the corresponding project to create a metadata dataset.
3.2.4 Additional capabilities
Based on system metadata, you can build an information dashboard from an operational perspective. For this purpose, we provide enterprise customers with a dashboard template for the information overview. This template is a value-added module. To use it, contact your company's sales representative or customer success manager for purchase information.
4.1 Subject data of behavioral events
Overview: Records all user operation logs within different projects on the platform, used for behavioral auditing, user activity analysis, resource popularity, and more. Fields such as resource ID, name, and creation time mainly include information about visualization modeling, datasets, charts, and dashboard resources. If there is no event information, it means the resource has not been accessed or operated.
Content:
Field name | Expression | Display name | Remarks |
|---|---|---|---|
p_date | p_date | p_date | Date partition field |
Event timestamp | event_timestamp | event_timestamp | Event timestamp = p_date, that is, the data table is stored according to incremental logic. |
Project ID | app_id | app_id | |
Event user | event_user | Event user | |
Event occurrence time (date format) | toDate(event_timestamp/1000) | Event occurrence time (date format) | |
Resource ID | resource_id | Resource ID | |
Resource name | resource_name | Resource name | |
Resource owner | resource_owner | Resource owner | |
Resource creation time | resource_create_time | Resource creation time | |
Project ID to which the resource belongs | resource_app_id | Project ID to which the resource belongs | |
Resource type | resource_type | Resource type | |
Event occurrence time | to_date(event_timestamp/1000) | Event occurrence time | |
Event type | event_type | Event type | |
Event resource ID | event_resource_id | Event resource ID | |
Event module | event_module | Event module | |
Event action | event_action | Event action | |
Event details | event_payload | Event details | |
Event resource name | event_resource_name | Event resource name | |
Event resource type name | event_resource_type_name | Event resource type name | |
Page views (PV) | count(event_user) | Page views (PV) | Calculated field |
Unique visitors (UV) | count(distinct event_user) | Unique visitors (UV) | Calculated field |
Resource folder | resource_folder_name | Resource folder |
4.2 Subject data of asset details
Overview: Records detailed asset data for datasets, visualization modeling, charts, and dashboards. Datasets and visualization modeling provide information on scheduling tasks, instance tasks, and underlying table TTL resources, enabling asset statistics, task management, resource management, and more.
Content:
Note: Assets include all asset content, which may contain system-generated tasks.
Field name | Expression | Display name | Remarks |
|---|---|---|---|
p_date | p_date | p_date | |
Base table creation time | data_table_create_time | Base table creation time | |
Base table storage size | data_table_table_size | Base table storage size | |
Base table update time | data_table_update_time | Base table update time | |
Base table lifecycle | data_table_ttl | Base table lifecycle | |
Base table row count | data_table_line_num | Base table row count | |
Visual modeling update type | prep_schedule_type | Visual modeling update type |
|
Visual modeling task type | prep_task_type | Visual modeling task type |
|
Visual modeling task schedule information | prep_schedule_conf | Visual modeling task schedule information | |
Visual modeling task execution status | prep_execute_status | Visual modeling task execution status |
|
Total visual modeling count | uniq(case when resource_type= visual modeling resource_id end) | Total visual modeling count | Calculated field |
Latest visual modeling sync end time | prep_last_sync_end_time | Latest visual modeling sync end time | |
Latest visual modeling sync start time | prep_last_sync_time | Latest visual modeling sync start time | |
Task count (visual modeling/dataset) | uniq(case when instance_schedule_time is not null then resource_id end ) | Task count (visual modeling/dataset) | Calculated field |
Instance ID | instance_id | Instance ID | |
Instance creation time | instance_create_time | Instance creation time | |
Instance queue | instance_queue | Instance queue | |
Instance end time | instance_end_time | Instance end time | |
Instance start time | instance_start_time | Instance start time | |
Instance name | instance_name | Instance name | |
Instance configuration information | instance_conf | Instance configuration information | |
Instance task execution time | instance_task_time | Instance task execution time | |
Number of instances | uniq(instance_id) | Number of instances | |
Instance scheduling time | instance_schedule_time | Instance scheduling time | |
Instance execution waiting time (s) | dateDiff('second', toDateTime(instance_schedule_time), toDateTime(instance_start_time)) | Instance execution waiting time (s) | Calculated field |
Instance execution duration (s) | dateDiff('second', toDateTime(instance_start_time), toDateTime(instance_end_time)) | Instance execution duration (s) | Calculated field |
Instance status | instance_status | Instance status | |
Dataset lifecycle | dataset_ttl | Dataset lifecycle | |
Dataset synchronization type | dataset_sync_type | Dataset synchronization type | |
Total number of datasets | uniq(case when resource_type= '数据集' then resource_id end) | Total number of datasets | Calculated field |
Total number of charts | uniq(case when resource_type= '图表' then resource_id end) | Total number of charts | Calculated field |
Project ID | app_id | Project ID | |
Project creation time | app_create_time | Project creation time | |
Project update time | app_update_time | Project update time | |
Project name | app_name | Project name | |
Project status | app_status | Project status | |
Total number of projects | uniq(app_id) | Total number of projects | |
Total number of dashboards | uniq(case when resource_type= '仪表盘' then resource_id end) | Total number of dashboards | Calculated field |
Resource ID | resource_id | Resource ID | |
Resource creation time | resource_create_time | Resource creation time | |
Resource update time | resource_update_time | Resource update time | |
Resource type | resource_type | Resource type | |
Resource name | resource_name | Resource name | |
Resource deletion time | resource_delete_time | Resource deletion time | |
Resource owner | resource_owner | Resource owner | |
Dataset type | data_set_type | data_set_type |
4.3 Subject data of asset relationships
Overview: Records the interdependencies among datasets, visual modeling, charts, and dashboards, which can be used for asset analysis, asset queries, lineage viewing, and other analyses.
Content:
*Note: Since the association relationships store many-to-many mappings, please perform deduplication when conducting statistics.
Field name | Expression | Display name | Remarks |
|---|---|---|---|
p_date | p_date | p_date | |
Number of visual modeling resources | uniq([Visual modeling resource ID]) | Number of visual modeling resources | Calculated field |
Visual modeling resource ID | prep_resource_id | Visual modeling resource ID | |
Visual modeling resource creation time | prep_resource_create_time | Visual modeling resource creation time | |
Visual modeling resource name | prep_resource_name | Visual modeling resource name | |
Visual modeling resource owner | prep_resource_owner | Visual modeling resource owner | |
Number of datasets | uniq([Dataset resource ID]) | Number of datasets | Calculated field |
Dataset resource ID | dataset_resource_id | Dataset resource ID | |
Dataset resource creation time | dataset_resource_create_time | Dataset resource creation time | |
Dataset resource name | dataset_resource_name | Dataset resource name | |
Dataset resource project ID | dataset_resource_app_id | Project ID of the dataset resource | |
Owner of the dataset resource | dataset_resource_owner | Owner of the dataset resource | |
Chart creation time | report_create_time | report_create_time | |
Chart update time | report_update_time | report_update_time | |
Number of charts | uniq([Chart resource ID]) | Number of charts | Calculated field |
Email prefix of the chart owner | report_owner_email_prefix | report_owner_email_prefix | |
Chart status | report_status | report_status | |
Chart resource ID | report_resource_id | Chart resource ID | |
Chart resource creation time | report_resource_create_time | Chart resource creation time | |
Chart resource name | report_resource_name | Chart resource name | |
Owner of the chart resource | report_resource_owner | Owner of the chart resource | |
Project ID | coalesce(app_id,dashboard_resource_app_id,report_resource_app_id,dataset_resource_app_id,prep_resource_app_id) | Project ID | Calculated field |
Dashboard ID | dashboard_id | dashboard_id | |
Dashboard update time | dashboard_resource_update_time | dashboard_resource_update_time | |
Number of dashboards | uniq([Dashboard resource ID]) | Number of dashboards | Calculated field |
Dashboard resource ID | dashboard_resource_id | Dashboard resource ID | |
Dashboard resource creation time | dashboard_resource_create_time | Dashboard resource creation time | |
Dashboard resource name | dashboard_resource_name | Dashboard resource name | |
Owner of the dashboard resource | dashboard_resource_owner | Owner of the dashboard resource |
4.4 User-related data
Overview: Records user data under different projects, used for analyzing data such as project changes and user changes.
Content:
*Note: Different projects have different configurations, so user information content may vary.
Field name | Expression | Display name | Remarks |
|---|---|---|---|
p_date | p_date | p_date | Date partition field |
Project ID | app_id | app_id | |
Project creation time | app_create_time | app_create_time | |
Project update time | app_update_time | app_update_time | |
Project name | app_name | Project name | |
Project status | app_status | app_status | |
User ID | user_id | user_id | |
User email prefix | user_email_prefix | user_email_prefix | |
User creation time | user_create_time | user_create_time | |
User update time | user_update_time | user_update_time | |
User name | user_name | user_name | |
User status | user_status | user_status |
|
Total number of users | uniq(user_id) | Total number of users | Calculated field |
User department | department_name | department_name |
(1) After importing the template, if you do not select to replace the dataset, queries on the dashboard may not work properly after import. How should this be resolved?
- Cause: After importing the template, if the dataset is not replaced, the template dataset still uses the historical permission configuration, which may not be effective in the new environment. You need to re-edit and update the permission information.
Import configuration | Issue description | Solution |
|---|---|---|
|
(2) After importing the template and selecting to replace the dataset, many field names in both Chinese and English appear. How should this be resolved?
- Cause: The imported template is a standardized dataset with optimized and modified dataset naming, while the original dataset uses English field names. At this point, both will exist simultaneously. To avoid confusion, you can delete all English field names to improve readability.
(3) What is the data lifecycle of system metadata?
The TTL (data lifecycle) of data connected to metadata is 7 days by default. When deploying the product, you can set the TTL. For specific operations, please consult customer service.
