The evaluation(review) function is used to evaluate the performance of different agents on specific tasks or general capabilities. You need to build the evaluation set, evaluation method, and access the agent to be evaluated in turn, and then create an evaluation task to evaluate and score the designated agent according to the evaluation set parameters and evaluation methods.

New review set #

1. Log into the Marketing Agent console and select Marketing Interactive Assistant in the left navigation bar.
2. In the left navigation bar of the Marketing Interactive Assistant, select Reviews > Reviews Set.
3. In the upper right corner of the review set page, click New review set.
   
4. Configure the basic information and parameters of review set and click Create.
   • Basic information
     • Name: Required, custom name, identifies the current review set.
     • Description: Optionally, a detailed description of the evaluation set.
   • Configuration column
     Set the column parameters contained in the evaluation set, add inputand reference_outputparameters by default, and support modifying or adding other column parameters.
5. Once created, you will be redirected to the review set details page, where you can view the column parameters.
   
6. Column parameters include system preset parameters and parameters that you add when creating a review set. You can click Edit Column to modify the parameter configuration.
   
7. Click Add Data to add parameter values for column parameters as a data reference when evaluating agents.
   
8. The addition methods include manual addition and local import.
   
   • Manually add: In the add data interface, set the parameter value for the column parameter, you can add multiple data items at the same time, the upper limit is 10.
     
   • Local Import:
     
     • Upload data: Support importing parameter values in the form of csv, zip, xlsx, xls files, the maximum file support is 200 MB, and only one file can be imported at a time.
     • Import method: append data or replace all data.
9. After adding data, you need to submit a new version in the upper right corner of the review set details page, and the review set can be used in association with the review task after submission.
   

Management Review Set #

In the review set list, you can view information such as the name of the review set, column name, data item, and version. In the Actions column, support for viewing or deleting review sets.

View review set details #

In the details page of the specified review set, support:

• Create a new evaluation task, continue to add data, and view the evaluation set version.
  
• View the evaluation task execution results associated with this evaluation set, including a summary of ratings and evaluation task details. On this page, you can directly perform operations such as viewing details, copying, exporting, and deleting on the evaluation task.
  

New evaluation method #

1. Log into the Marketing Agent console and select Marketing Interactive Assistant in the left navigation bar.
2. In the left navigation bar of the Marketing Interactive Assistant, select Evaluation > Evaluation Method.
3. In the upper right corner of the test method page, click New test method.
   Currently supports the LLM evaluation method.
   
4. In the Select Template pop-up, select Preset Evaluation Method and click Confirm.
   The system presets a wealth of evaluation methods. You can choose the specified method and check whether the prompt word (Prompt) meets the evaluation requirements.
   
5. On the New Evaluation Method page, complete the following configuration.
   
   • Basic information: custom name, description.
   • Configuration information:
     • Model selection, support for configuring model parameters, including maximum reply length, generation randomness, and top P.
     • Check whether the preset prompt word meets your requirements, support re-selection of evaluation methods, and support custom prompt words.
6. After completing the configuration, click Debug in the lower right corner of the page to construct input and output parameters to debug whether the current evaluation method meets expectations.
   
7. After debugging, return to the New Evaluation Method page and click Create in the lower right corner of the page.

Management evaluation method #

In the list of evaluation methods, you can view the name, type, latest version, and update time of the evaluation method. In the operations column, the following operations are supported:

• Details: Enter the evaluation method details page, support modifying the model, prompt words, and debugging. After the modification is completed, a new version needs to be submitted.
• Copy: Copy the current evaluation method and create a new one.
• Deletion: Deletion is supported for evaluation methods that are no longer in use.

  warning

  Once the evaluation method is deleted, it cannot be restored. Please proceed with caution.

The self-built agent in the Marketing Interactive Assistant is displayed in the Agent list by default, and supports editing or deleting operations.

1. Log into the Marketing Agent console and select Marketing Interactive Assistant in the left navigation bar.
2. In the left navigation bar of the Marketing Interactive Assistant, select Reviews > Agent Access.
3. In the Agent access list, view the Agent details.
   
4. On the details page, you can view the basic information and version records of the Agent, and in the upper right corner of the page, you can support debugging or editing the Agent.
   
   When editing, the parameters are described as follows:
   • Agent Name: A custom name that identifies the current Agent and cannot exceed 50 words.
   • Agent description: Custom description, such as agent introduction, precautions and other information, cannot exceed 200 words.
   • Destination interface address: Agent interface address with a character limit of 200.
   • Version number: The version number of the current agent, which cannot be edited.
   • Input Field JSON: Agent input field for receiving evaluation set input data.
   • Output Field JSON: Agent output field used by the evaluation system to extract Agent response data.
   • Body parameter: Agent interface requests body data structure. After configuration, click Validate JSON at the bottom of the code box to check whether the parameter configuration is wrong.
   • Header: Request header information for the Agent interface. For example,the x-api-key is the secret key of the Agent interface.
5. (Optional) Delete agents that are no longer needed in the Agent access list.

   warning

   Once the agent is deleted, it cannot be restored. Please proceed with caution.

   

After completing the access and construction of the evaluation set, evaluation method, and agent, the evaluation task can be systematically constructed to evaluate the performance of the agent on specific tasks or general capabilities.

New evaluation task #

1. Log into the Marketing Agent console and select Marketing Interactive Assistant in the left navigation bar.
2. In the left navigation bar of the Marketing Interactive Assistant, select Review > Review Tasks.
3. In the upper right corner of the review task page, click New review task.
   
4. On the Basic Information screen, complete the following configuration and click Next: Evaluation Set.
   • Name: Required, custom name, identifies the current task. No more than 50 words.
   • Description: Optional, custom description, can fill in the introduction, requirements, rules and other information of the current task, and can exceed 200 words.
   • Maximum number of concurrent executions: Optional, default is 5. The evaluation task supports concurrent execution of the items in the evaluation set, but is limited by the concurrency of the evaluation object and the model TPM limitations for invoking the evaluation method. Here you set the ideal maximum number of executes, up to a maximum of 20.
5. In the review set interface, select the specified review set and version, and click Next: Review Objects.
   
   • The selection displays the description, column names, and total amount of data for the review set.
   • Click View Details in the upper right corner of the page to jump to review set details.
6. (Optional) In the test object interface, complete the following configuration and click Next: Test Method.

   tip

   If your review set column parameters, such as actual_output, already contain the reviews that the Agent actually outputs, you can skip this step by clicking Skip at the bottom of the page.

   • Agent application: Select the agent to be evaluated.
   • Version number: Select the specified version of Agent.
   • Data field mapping: Specifies the input parameter mapping relationship between the evaluation object and the evaluation set, which is used for the evaluation object to accurately obtain the input data.
7. (Optional) On the Measurement Methods page, complete the following configuration and click Confirm Measurement Task Configuration.

   tip

   You can click Skip at the bottom of the page to skip this step directly. The evaluation task only obtains the actual output of the Agent, which is suitable for manually annotated evaluation scenarios.

   
   1. Click Add measurement method.
      The evaluation method supports adding multiple, with an upper limit of 10.
   2. Select the evaluation method by name and specify the version.
   3. In the data field mapping area, configure the mapping relationship between the evaluation method field and the evaluation set field, and the actual output field of the evaluation object, which is used for the evaluation method to accurately obtain the input for evaluation.
8. After confirming that the configuration of the evaluation task is correct, click to initiate the evaluation task.
9. On the evaluation task data details tab, wait for the evaluation task execution to complete.
   You can click Refresh to update the evaluation task execution status.
   

View evaluation task execution results #

On the test task details page, go to the Data Details tab, in the list, you can view the input, output, and scoring results of each piece of data.

• In the upper right corner of the list, you can adjust the column fields displayed in the list, as well as the view type (including compact view and loose view).
  
• In the operation column of the specified data, click Details to jump to the detailed information of the evaluation results, including: evaluation set data, actual output, evaluation score and score reason.
  

On the evaluation task details page, enter the indicator statistics tab to view the evaluation method aggregation score and score details.

Comparison and evaluation task #

For the evaluation tasks that use the same evaluation set and have been completed, data comparison of the evaluation results can be carried out.

1. Go to the specified evaluation task details page and click evaluation task comparison in the upper right corner of the page.
   
   You can also batch select tasks for comparison in the evaluation task list.
   
2. Check the tasks that need to be compared in the evaluation task list, and click to initiate evaluation task comparison.
   
3. View data breakdown or comparison details of indicator statistics.
   In the upper right corner of the page, it is supported to switch the benchmark task and continue to add the comparison task.
   

Manage evaluation tasks #

In the evaluation task list, the following administrative operations are supported:

• View information such as the name, associated test set, status, and score of each test task.
• At the top of the list, there is support for searching tasks by name, setting filters to filter tasks, refreshing the list, and adjusting the displayed column fields.
  
• Click Bulk Selection at the top right of the list.
  
  Support batch comparison tasks, or batch deletion tasks.

  warning

  Once the evaluation task is deleted, it cannot be restored. Please proceed with caution.

  
• Click Details in the Actions column to go to the evaluation task details page.
• For failed status evaluation tasks, you can click Retry in the Action column to re-evaluate the failed part.
  
• Click Copy in the Actions column to initiate a new evaluation based on the current evaluation directly or by modifying the configuration.
• In the action column, export evaluation task detail data is supported.
  
  After exporting, you can view the status of the exported task in the export record and download the file.
  
• In the Actions column, it is supported to delete evaluation tasks that are no longer in use.

  warning

  Once the evaluation task is deleted, it cannot be restored. Please proceed with caution.