API is one of the most common solutions for data transmission and interaction between systems. This article describes how to create a new REST API 2.0 data connection.

Create via data connection #

1. Log in to BytePlus, navigate to a specific project, click Data Preparation, find Data Connection in the drop-down list, and click Data Connection.

   

2. Select REST API 2.0 on the page.

3. Next, fill in the required basic information, configure request parsing and path settings, and test the connection. After the connection is successful, click Save to complete all configurations for the current data connection.

Create via dataset #

1. Log in to BytePlus, click Data Preparation --> Dataset, select the "Create" button in the upper left corner to create a new dataset.
   

2. When selecting a data connection, click Create Configuration.

   

3. The following steps are the same as steps 2 and 3 in Create via data connection. After completing the configuration, you will remain in the pop-up window for selecting a data connection for the dataset, and you can proceed directly to the next step of dataset creation.

Data connection name #

Custom names are supported, or you can use the system default name (for example: REST_API_2.0_timestamp).

URL address #

Enter the URL address of the API to connect to. The user can enter it manually or use cURL import to complete the URL address, request method, and parameter settings. For completed configurations, cURL format export is also supported.

curl -X POST "https://api.example.com/import" \
  -H "Content-Type: application/json" \
  -H "Authorization: Bearer YOUR_API_KEY" \
  -d '{
    "name": "John Doe",
    "email": "john@example.com"
  }'

• cURL import: Click the blue "cURL" button, paste the cURL content to be imported into the pop-up "Import via cURL" window, and click "Parse and Import". The system will automatically recognize the URL address, request method, and parameter settings, and fill them in the corresponding fields.
  
• cURL export: After manually configuring the URL address, request method, and parameter settings, to export in cURL format, click "Export". In the pop-up "Export cURL" window, click "Copy". The system will automatically generate the cURL-formatted content and copy it to the clipboard.
  

Request method #

Supports two request methods: GET and POST.

• GET: Used to retrieve resources, with parameters appended to the URL.
• POST: Used to submit data, with parameters passed in the request body.

OAuth2 authentication #

Supports the OAuth2 authorization code mode.

1. Configure basic information. Click Authorize to open the sidebar, and enter the basic authorization information (AK, SK, and authorization redirect address). For specific acquisition methods, refer to the relevant third-party website documentation. After configuration, click "Next".

2. If the user is configuring authorization information with this combination of AK and authorization redirect address for the first time, it is necessary to further obtain the AccessToken configuration and RefreshToken configuration. Configure the API address, request method, request parameters, response parameters, and other related information. After clicking Confirm, the browser will automatically initiate the authorization request link and redirect to the third-party authorization page.
3. After the user completes authorization, the ABI unified authorization callback address is automatically invoked. The ABI callback will, based on the user's previous authorization configuration, synchronously initiate [Get Access Token] and [Get Refresh Token] requests in the backend. After verification is successful, the configuration takes effect, and when you return to the data connection page, you will see that authorization is complete.

   warning

   • Third-party website authorization is usually based on a single AK dimension. If multiple data connections use the same AK, they must share the same authorization.
   • The system automatically matches a valid authorization based on the host and AK of the authorization link. If none exists, initial configuration is required.
   • The system cannot validate third-party output parameters (no unified format). Please carefully check the parameters to avoid interface failures caused by incorrect parameters. If the authorization address and AK are the same, the system will automatically match the previous configuration. Incorrectly matched data can only be corrected and reconfigured.
   • In the [Obtain Access Token] callback request, the parameter key corresponding to code is fixed as "code" and cannot be changed.

Parameter settings #

The parameter settings section guides users through a standardized process to configure the basic information for API requests, including request headers, request parameters, request body, as well as pre-processing and post-processing operations. The system provides built-in variables and built-in functions. For details, refer to System built-in variables and System built-in functions.

Request headers #

Supports configuring key and value, as well as add and delete operations, with up to 10 groups configurable.

Request parameters #

Enter all query parameters required by the API.

Request body #

Select the request body type (raw, form-data, or x-www-form-urlencoded); all support user input of a JSON-formatted request body.

General rule: When operating on JSON, $ represents the root node, and . represents accessing a child node.
If the JSON data is as follows:

{
  "showapi_res_error": "",
  "showapi_res_id": "6597a2f5fb638cf45b4d460c",
  "showapi_res_code": 0,
  "showapi_fee_num": 1,
  "showapi_res_body": {
    "list": [
      {
        "time": "14:26:57",
        "zhesuan": "193.69",
        "code": "AED",
        "hui_out": "195.91",
        "chao_in": "192.95",
        "chao_out": "199.47",
        "name": "阿联酋迪拉姆",
        "hui_in": "194.55",
        "day": "2024-01-05"
      },
      {
        "time": "14:26:57",
        "zhesuan": "477.56",
        "code": "AUD",
        "hui_out": "481.77",
        "chao_in": "478.08",
        "chao_out": "483.16",
        "name": "澳大利亚元",
        "hui_in": "478.57",
        "day": "2024-01-05"
      }
    ],
    "listSize": 28,
    "ret_code": 0
  }
}

Then the values corresponding to some paths are as follows:

$.showapi_fee_num = 1
$.showapi_res_id = "6597a2f5fb638cf45b4d460c"
$.showapi_res_body.list.* = [{"time":"14:26:57","zhesuan":"193.69" ... // Omitted below

Pre-processing operations #

To perform additional steps to prepare data or obtain necessary information before the main request, use this configuration.

1. Click the "Add pre-request action" button. On the configuration page, enter the details of the pre-request action, including the URL and request method.
   
2. After clicking "Send request", the system will parse this pre-request and display the parsing result in the "Request result" area. The user needs to configure the key for dynamic parameters and the result field path to facilitate applying the pre-request result to the current request. For guidance on writing dynamic parameters, refer to Dynamic parameter writing rules.

Post-request actions #

Provides a more flexible and powerful way to configure custom parameters, supports Groovy syntax, and allows users to write code for multi-level request parsing. For rules on writing post-request actions, refer to Post-processing operation writing rules.

Advanced settings #

Allows users to set the request timeout, number of retries, pagination, and other options.

Pagination types include numeric pagination and cursor pagination:

The following explains the relevant parameter configurations:

Limits for the number of fields, items per page, and pagination configuration:

Example:
For example, to end pagination when the data list in the request result is empty, and to set the maximum number of paginations as the upper limit of request attempts (default is 10, maximum is 10,000), configure as follows:

When in use, reference ${pageNum}. This variable starts from the initial page and continues until the termination condition is met.

Configure request parsing #

After completing the above basic configuration steps, click "Send request" to have the system parse the request. The parsing order is "Pre-request action" > "Current request" > "Post-request action", and the "Parsing result" will be displayed.

Configure path #

After the system completes request parsing, the user also needs to configure the path, including setting the field path, configuring fields, and previewing data.

Field path #

Enter the field path, using $ to represent the root node and a period (.) to indicate a child node. Click the "Get Fields" button. The system will parse and display the "Field Configuration" section.
This section configures the root path of the parsing result, with the following rules:

• The root path must point to a Map or Array; otherwise, the field key cannot be parsed.
• If it points to an Array
• After configuration is complete, clicking Get Fields will only parse the first-level field structure under this path, and all these fields will be of basic data types.

Example:

// Retrieve the entire result. The result fields include: showapi_res_error, showapi_res_id, ..., showapi_res_body
$
// Retrieve the contents of the array
$.showapi_res_body.list.* = [{"time":"14:26:57","zhesuan":"193.69" ... // The following is omitted

Field configuration #

Use the "Add" button to create new fields. The parsing logic is based on the rootPath you entered to generate the result. You can also delete configured fields.
This section configures the path of the final fields obtained after parsing the root path, which is also the final field result of this data connection.

Data preview #

Click the "Data Preview" button to view the first 100 rows of detail data for the currently selected field.

After all content is configured, click Test Connection to test the connectivity of the data source. After the test is successful, click Save to complete all configurations for the current data connection.

1. Create a dataset using the previously created data connection: Click Data Preparation --> Dataset, then select the "New" button in the upper left corner to create a new dataset.
   
2. Search for or select the previously created data connection from the dropdown.

Dynamic parameter writing rules #

Both dynamic parameters and post-processing scripts use Groovy as the script engine.
For ${} appearing in params, body, header, or url, it is executed as a Groovy string interpolation expression to obtain the result. Some custom functions are also built in, such as the md5() method, date variables, and more. Below are some examples of execution results:

假如目前有两个参数，是在前置请求中配置好的，如下：
num1 = 1
num2 = 2

则${}里面的内容会被作为表达式执行，其余部分为普通字符串，下面表达式结果分别为：
abc               // abc
${num1}           // 1
${num1+num2}      // 3
${num1}+num2      // 1+num2
${num3}           // Error: parameter num3 is missing
${"${num1}"}      // ${num1}

Post-processing operation writing rules #

The post-processing operation example is as follows:

def convertJson(response) {
    /*Enter post-processing code hereresponse represents the result returned by the API. Suppose the API returns the following result    {"city": "上海市",        "forcast": [            {"temperature": 5, "day": "2024-01-01"},            {"temperature": 6, "day": "2024-01-02"},        ],        "test": "test"    }Some code examples:Scenario 1: Add, edit, and loop to add city to each element in the list    ls = response.forcast    for (i=0; i< ls.size(); i++) {        ls[i].city = response.city    }Scenario 2: Remove a specific key    response.remove("test")    */
    return response;
}

Termination condition writing rules #

Groovy syntax is supported. When the termination condition is met, the next page will not be accessed. response represents the result of this request. Example:

response.data.size() == 0   // Stop pagination when the length of the data list under response is 0 (empty list)
response.hasMore == false // Stop pagination when hasMore under response is false

System built-in variables #

The following dynamic parameters are built into the system. When naming variables, avoid using the same names as system variables; otherwise, they will be overwritten.

Scope: url, header, params, body. Not valid in post-processing operations.

${date} // Date, format: yyyyMMdd
${DATE} // Date, format: yyyy-MM-dd
${hour} // Hour, format: HH
${HOUR} // Hour, format: H
${year} // Year, format: yyyy
${month} // Month, format: MM
${day} // Day, format: dd
${timestamp} // Timestamp, format: 13-digit timestamp
${last_date} // Last day of the previous month, format: yyyyMMdd
${last_DATE} // Last day of the previous month, format: yyyy-MM-dd
${last_day} // Last day of the previous month, format: dd

以上变量支持日期加减，例如 ${date-1}，代表前一天

System built-in functions #

Scope: url, header, params, body, post-processing operations.

Instructions for use #

• Within the applicable scope of system built-in functions, you can use the "${}" syntax to access supported built-in functions and predefined variables (the content inside the curly braces is a Groovy expression).
• Multiline statements are supported. Separate multiline statements with a semicolon. When using multiline expressions, the final value is the return value of the last expression.
• Example:

  • Add md5 value and current system timestamp in the URL

    

  • Add a value calculated by a complex expression in the request header

Digest functions #

String md2Hex(String data);
String md5Hex(String data);
String shaHex(String data);
String sha1Hex(String data);
String sha256Hex(String data);
String sha384Hex(String data);
String sha512Hex(String data);

byte[] md2(String data)
byte[] md5(String data)
byte[] sha(String data)
byte[] sha1(String data)
byte[] sha256(String data)
byte[] sha384(String data)
byte[] sha512(String data)

Mathematical functions #

java.lang.Math all related functions

Get date and time, format, add or subtract dates #

// format: Format date as a string.
currentDate = new Date()
formattedDate = currentDate.format('yyyy-MM-dd HH:mm:ss')

// plus and minus: Add or subtract dates.
tomorrow = currentDate + 1 
yesterday = currentDate - 1 

String extension methods #

// capitalize(): Capitalize the first character of a string.
str = "groovy"
str.capitalize() // Output: "Groovy"

// reverse(): Reverse a string.
str.reverse()     // Output: "yvoorg"

// contains(): Check if a string contains a specified substring.
str.contains("oo") // Output: true

// toInteger(), toLong(), toFloat(), toDouble(): Convert a string to the corresponding numeric type.
str = "123"
str.toInteger() // Output: 123

// split(): Split a string.
str = "a,b,c"
str.split(",") // Output: ["a", "b", "c"]

// isNumber(): Check if a string is a number.
str.isNumber() // Output: true

Collection extension methods #

// each: Execute a closure operation on each element of a collection.
list = [1, 2, 3]
list.each { println it } // Output: 1 2 3

// findAll: Filter elements in the collection that meet the criteria.
list = [1, 2, 3, 4, 5]
evens = list.findAll { it % 2 == 0 }
println evens // Output: [2, 4]

// collect: Map each element in the collection to another value.
list = [1, 2, 3]
squares = list.collect { it * it }
println squares // Output: [1, 4, 9]

// sum: Calculate the sum of all elements in the collection.
list = [1, 2, 3]
println list.sum() // Output: 6

// join: Join the elements in the collection into a string using the specified separator.
list = ["a", "b", "c"]
println list.join("-") // Output: "a-b-c"

Map extension methods #

// each: Execute a closure operation on each entry in the Map.
map = [a: 1, b: 2, c: 3]
map.each { key, value -> println "$key: $value" } // Output: a: 1 b: 2 c: 3

// findAll: Filter entries in the Map that meet the criteria.
map = [a: 1, b: 2, c: 3]
filtered = map.findAll { key, value -> value > 1 }
println filtered // Output: [b: 2, c: 3]

// collect: Map each entry in the Map to another value.
map = [a: 1, b: 2, c: 3]
collected = map.collect { key, value -> "$key=$value" }
println collected // Output: ["a=1", "b=2", "c=3"]

// getOrDefault(Object key, Object defaultValue): Get the value for the specified key; if the key does not exist, return the default value.
map = [a: 1, b: 2]
println map.getOrDefault('c', 0) // Output: 0

// collectEntries(Closure closure): Convert the Map to another Map.
map = [a: 1, b: 2, c: 3]
newMap = map.collectEntries { key, value -> [(key): value * 2] }
println newMap // Output: [a: 2, b: 4, c: 6]

// find(Closure closure): Find the first entry in the Map that meets the criteria.
map = [a: 1, b: 2, c: 3]
entry = map.find { key, value -> value > 1 }
println entry // Output: [b:2] or [c:3]

// keySet(): Get all keys in the Map.
map = [a: 1, b: 2, c: 3]
println map.keySet() // Output: [a, b, c]

// values(): Get all values in the Map.
map = [a: 1, b: 2, c: 3]
println map.values() // Output: [1, 2, 3]

// flatten(): Flatten a nested Map.
map = [a: [b: 2, c: 3], d: 4]
println map.flatten() // Output: [a.b:2, a.c:3, d:4]

List extension methods #

// unique(): Remove duplicate elements from the list.
list = [1, 2, 2, 3, 4, 4]
println list.unique() // Output: [1, 2, 3, 4]

// sort(): Sort the list.
list = [3, 1, 4, 1, 5, 9]
println list.sort() // Output: [1, 1, 3, 4, 5, 9]

// flatten(): Flatten a nested list into a single list.
list = [1, [2, 3], [4, [5, 6]]]
println list.flatten() // Output: [1, 2, 3, 4, 5, 6]

// count(Closure closure): Count the number of elements that meet the criteria.
list = [1, 2, 3, 4, 5]
println list.count { it % 2 == 0 } // Output: 2

Range extension methods #

// step(int step): Iterate over the elements in the range with the specified step size.
range = 1..10
range.step(2).each { println it } // Output: 1 3 5 7 9

// containsWithinBounds(Object value): Check whether the value is within the range (including boundaries).
range = 1..10
println range.containsWithinBounds(10) // Output: true

// subList(int fromIndex, int toIndex): Get the sublist within the specified index range.
range = 1..10
println range.subList(2, 5) // Output: [3, 4, 5]

// toList(): Convert the range to a list.
range = 1..5
println range.toList() // Output: [1, 2, 3, 4, 5]

Encryption methods #

// Base64 encoding and decoding
// Encoding
str = "hello world"
encoded = str.bytes.encodeBase64().toString()
println "Base64 Encoded: $encoded" // Output: Base64 Encoded: aGVsbG8gd29ybGQ=
// Decoding
encoded = "aGVsbG8gd29ybGQ="
decoded = new String(encoded.decodeBase64())
println "Base64 Decoded: $decoded" // Output: Base64 Decoded: hello world

Digest methods #

// MD5
str = "hello world"
md5Hex = str.md5()
println "MD5: $md5Hex" // Output: MD5: 5eb63bbbe01eeed093cb22bb8f5acdc3

// SHA-1
str = "hello world"
sha1Hex = str.digest("SHA-1")
println "SHA-1: $sha1Hex" // Output: SHA-1: 2aae6c35c94fcfb415dbe95f408b9ce91ee846ed

// SHA-256
str = "hello world"
sha256Hex = str.digest("SHA-256")
println "SHA-256: $sha256Hex" // Output: SHA-256: b94d27b9934d3e08a52e52d7da7dabfac484eefc56bf5ef4b102e3d8fdedfb68