For some business operations, looking at long-term data and comparing current business conditions is a necessary analytical tool. Therefore, it is often necessary to view long-term data at this time, and it is necessary to do dataset backtracking at a relatively high frequency, which affects the scheduled dataset scheduling execution. For example, managers within a company often look at long-term data, or planning/operations personnel often look at comparative data at different stages, possibly with a backtracking frequency of at least 20 datasets per month. In response to this scenario, Datawind has launched the ability to "support separate queues for backtracking tasks", which helps users configure different queues for scheduled tasks and data re-run tasks according to business conditions, reducing the impact of data re-run on routine tasks.

In the current version, the queue configuration only supports 1 queue. The multi-queue configuration will be supported in the future.
The users of the queue configuration function use the link as shown in the figure:

Queue management #

In the system management, click the system configuration button and select the queue configuration module in the left sidebar to perform queue management.

Default queue #

Currently only one queue is supported, which is the default queue and is named offline. According to the following figure, you can see that the default queue information includes: queue name, queue resource, queue description, authorization project scope, group default queue, and applicable task type.

Click the Edit button to modify the queue description, as shown in the figure below.

Applicable task type #

The default queue offline applies to scheduled tasks and backtracking tasks, and each task type currently supports one backtracking queue.

Default policy #

As shown in the figure below, the default policy is to prioritize the default queue. Hover the mouse over the Default Policy Description icon to see the specific description of the default policy.

Scope of authorization #

Authorization is supported according to project scope or user scope. The available scope of the queue is the sum of the scope for authorized items and authorized users.

Task configuration #

Dataset management #

Enter the dataset in the data preparation module, select the corresponding dataset, and enter the synchronization status bar. Click "Data Sync" in the upper right corner to configure the synchronization.

• Basic Settings: Set the selection range, only data within the synchronization lifecycle range is supported. You can choose whether the synchronization status list displays this time range after synchronization is enabled.
• Back Filling Settings: You can set up dependency checks and queues. Dependency checking can be set to execute as a dependency setting or immediately. Queues support selection of backtracking queues.
  

Dataset editing #

Click Data Preparation - Dataset, select the corresponding dataset, and click Dataset Editing - Synchronization Settings - Advanced Settings to select the queue. The default option is the default queue. For other synchronization settings configuration operations, please refer to the synchronization settings documentation for more information.

Visual Modeling Management #

Click Data Preparation - Visual Modeling, select the corresponding task, click Run, and select the queue in the pop-up window - Traceback settings. The default option is the default queue. Other running configuration operations can be skipped to task operations - run the documentation to learn more.

Visual modeling editing #

Click Data Preparation - Visual Modeling, select the corresponding task, click Edit, and select Advanced Settings in the output to select the queue. The default option is the default queue. For other output configuration operations, redirect to the output documentation to learn more.

Cluster configuration #

The number of clustered tables limit function is used to set the number of tables and the amount of available resources by project in a given cluster. Users can enter the configuration panel of the corresponding cluster on the cluster configuration page, view the project name, the current number of tables, and set the upper limit of the dataset table, the upper limit of the resource, and the upper limit of the visual modeling table.
This feature is suitable for scenarios where the number of item tables needs to be viewed and set within the cluster.

Prerequisite #

• The user needs to have administrator-level permission and can click on the cluster configuration page to enter the cluster table limit configuration panel;
• The target cluster needs to have completed the basic configuration in the system.

Operation steps #

1. Click the System Administration button in the upper right corner of the product page, and select Data Operations Configuration > Queue Clustering in the side navigation.
2. On the page, click the cluster configuration tab, in the operation column of the row where the target cluster is located, click the control button, the system opens the cluster number limit configuration panel on the right, and the panel title is the current cluster name.
   
   
3. To limit the number of Dataset Tables (Unit: tables)/Visual Modeling Tables (Unit: tables)/Resource Control (Unit: GB), click the corresponding switch of Number of Dataset Tables/Number of Visual Modeling Tables/Resource Control, turn it on.
4. After turning on the corresponding switch, fill in the upper limit value in the text box. Complete the setting of dataset table upper limit, resource upper limit and visual modeling table upper limit respectively. After opening, exceeding the limit policy prohibits the creation of new extraction tasks.

   tip

   Control project management policies when resources are exceeded through constraint logic. If new creation is restricted, it will affect the visual modeling output dataset and independently created extraction datasets, and users need to switch to another cluster before saving.
   After exceeding the limit, only new additions are intercepted, and edits to existing datasets will not be intercepted; existing visual models are intercepted if new output datasets are added.

   warning

   • Limiting the upper limit is not a direct operation on storage, but is logically limited by functions, so there may be cases where the actual number of tables in the project is greater than the upper limit;
   • [Only new extraction tasks are prohibited] Only new entries are controlled, but there may be running tasks that continue to generate new resource occupation, so there may be situations where the actual used resources of the project are greater than the upper limit resources. At this time, it is recommended to go to the task overview to view all running tasks. Guide to stop tasks;
   • After the current resource limit is exceeded, the processing method is "only prohibit new extraction tasks"; if you need to release resources, it is recommended to go to the task overview to view and process the continuously running tasks.
   • When the resource usage of the project under the current cluster reaches the limit value, when the user creates a new extraction dataset or saves a visual modeling output dataset, the cluster will no longer be automatically recommended and will be displayed as unselectable.
   

Query parameters #

In large data volume and long-cycle analysis scenarios, individual queries may consume too many resources, which may affect the normal execution of other tasks. The query parameter function allows administrators to configure query fuse thresholds for a single dataset, and will be terminated in time when the query exceeds the threshold to avoid affecting the overall cluster stability.

• Configuration granularity: In units of datasets, different parameter values can be configured for different datasets.
• Effective method: After the configuration is saved, the downstream modules such as visual query and dashboard will automatically bring the parameters specified here when querying the dataset.
• Support range: Only extraction datasets (including historical stock datasets and Fabric datasets) are supported, and direct connection datasets do not support configuration.

Function entrance #

1. Log in to Datawind and click the Settings icon in the upper right corner of the page.
2. Select Data Operations Configuration > Queue Clusters in the left navigation bar.
3. Select the Query Parameters tab on the Queue Clusters page to enter the Query Parameters Configuration page.

Precautions #

• Query parameters only support extraction datasets, including historical stock datasets and Fabric datasets; direct-connected datasets cannot be configured and are not affected by this configuration;
• The parameter configuration takes effect globally: after saving, all downstream modules such as visual query and dashboard will carry corresponding parameters when querying the dataset. Please set it carefully according to the data volume and query characteristics of the dataset;
• If the fuse threshold is set too small, the normal query may be terminated prematurely. It is recommended to gradually adjust it according to the actual query time, memory and CPU usage;
• After deleting the query parameters, the configuration cannot be directly restored. If you need to restart it, please add a new configuration for this dataset.

Page description #

After entering the query parameters tab, you can quickly locate the dataset by searching and filtering, and add, edit, and delete query parameters.

• Search: A search box is provided at the top right of the page. Enter the dataset ID in the search box (placeholder prompt: please enter the dataset ID) to quickly locate the target dataset.
• Filtering: The three-column header of item ID, dataset ID, owner of table supports filtering, and the list can be filtered by item, dataset, or dataset owner.
• Configuration List: The table shows the dataset of configured query parameters, with the following columns.

Add query parameters #

1. Click the Add button at the top right of the query parameters page to pop up the Add Query Parameters window.

2. Fill in the dataset ID (the dataset id is obtained in the url of the dataset details page, and the number in the url is the dataset ID), and configure the query parameters (both form and JSON are supported).

   • Use the form: When configuring using the form method, the following query fuse parameters are supported, and the parameter values are integers:

   Parameter name	Meaning	Unit	Performance beyond a threshold
   max_memory_usage	The amount of memory a single Query can use.	bytes	Returns an out-of-memory exception.
   max_execution_time	Query timeout for a single query.	second	Returns a query timeout exception.
   max_query_cpu_seconds	CPU time used cumulatively by a single query.	second	The query will be terminated.

   • JSON format editing: In the pop-up window, click Fill in using JSON format to enter the JSON editing state.

3. Click OK to complete the addition; after the configuration is successful, the dataset appears in the configuration list, and you can directly enter the complete JSON parameter object in the text box to query.

Query parameter list operation #

Supports different operations on specific query parameters in the list, including:

• Edit: Find the corresponding dataset in the list, click the edit icon of the operation column, and the edit query parameter window will pop up. The relevant configuration items can be adjusted directly. Attention: Dataset IDs do not support modification.
• Delete: Find the target dataset in the configuration list and click the Delete icon in the Actions column to delete the current query parameter. Note: After deletion, the query dataset will not carry query parameters, and will not reply to relevant query parameters. Please operate with caution.