Application scenario #

If you need to push the buried data collected by the Finder-SDK to the target address in real time (such as your business data warehouse), the data output module can do it for you.

Usage restrictions #

Data output method #

The current privatization environment supports real-time and offline data transmission. The main processes for real-time and offline data transmission are as follows.

• Real-time transmission (privatization)
  The current real-time transmission supports real-time access to data through "webhook" channels.
• Offline transmission (privatization)
  • After privatization 4.6.1 (inclusive), it supports one-time or regular offline export of behavioral data to FTP/SFTP on the basis of real-time data transmission, and realizes offline data output.
  • After privatization 5.3.1 (inclusive), it supports synchronization of user subgroup data to the media platform.

If you have other data transfer scenarios, please contact your Customer Success Manager.

Prerequisite #

Create data output task #

1. After logging into the DataFinder console, click Data Management > Data Output > Data Output in the top navigation bar to enter the Data Output page.
   

   tip

   If you cannot find the "Data Export" entry in the application management list, please contact your administrator to grant permission for you. (Permission configuration entrance: Group settings - role management - growth analysis - configuration function permissions)

2. Click New Task in the upper right corner of the page, configure the data output task details, and click OK when done.

Real-time task #

Real-time tasks are used to forward behavioral data in real time. The configuration details of the task are as follows.

1. Configuration basic information.

   Parameter	Parameter description
   Task name	You can customize the name, such as XX product event tracking data push.
   Application	Select which application to transfer the collected data to the destination address.
   Task type	The current privatization environment supports real-time , offline - single , offline - daily routine three types of tasks. If you want the data to be collected and reported by the SDK to be transmitted to the destination address in real time, you can select the task type as real-time .
   Output content	To configure the data content to be distributed to the target address, the supported configuration is: Original report data : that is, the original data source collected and reported by the SDK, and the data is delivered to the target address within 30 seconds after being collected and reported by the SDK. Transaction table data : The SDK collects and reports the transaction table data stored in the inventory after being processed by the DataFinder. The data will be delivered to the target address within 10 minutes after being collected and reported by the SDK. Because there is a data processing process when the original data source is dropped, the distribution time will be delayed compared with the original reported data. If the original data source collected does not meet the requirements of the DataFinder's database data specification, this part of the original data source will not be distributed to the target address (for example, the attribute data type is inconsistent). warning When real-time data output starts data transmission, some data has not yet been stored, such as data that needs to be parsed twice, such as IP resolution provinces & cities; and real-time output transmission does not include data reported by the profile (user data) and item (business object) APIs. Therefore, if you need to receive such data, it is recommended to use offline transfer tasks.

2. Data connection configuration.

   Parameter	Parameter description
   Connection protocol	HTTP is selected by default.
   Access method	Webhooks are selected by default
   Webhook URL	Configure the Webhook URL address for distribution. Format example: http://localhost:6666/payload . Http/https is supported. warning A data distribution task only supports filling in one URL address. If there are multiple addresses, multiple tasks can be created. In a privatized environment, if the receiving service is on the external network, you need to configure the exit IP of the privatized machine.
   Headers	You need to configure webhook headers according to the webhook requirements of the receiving end for data sending, and support setting multiple headers. For example, if the receiver requires authentication information to be configured, you can configure the authentication information in headers, and subsequent DataFinder transfers will carry the header information configured here.
   Test access connection	You can click Test Access Connection to test whether it is connected to the webhook. Only tasks that have been connection-verified can be created.

3. Configuration data management.

   tip

   After privatization 4.4.1 (inclusive), new filter conditions can be added to select the range of data distributed according to the combination of event and attribute rules.

   

   Parameter	Parameter description
   Add event	Configure the list of events to be distributed to the destination address. The event list contains logic that only selected events will be distributed.
   Event property addition rule	The added events are filtered by the event attribute, that is, only data within the scope of the event list and meeting the rules of the event attribute will be distributed.

Offline tasks: synchronizing behavioral data, user data #

Offline tasks : used to export historical behavior data in batches. The configuration details of the task are as follows.

1. Configuration basic information.

   Parameter	Parameter description
   Task name	You can customize the name, such as XX product event tracking data push.
   Task type	The current privatization environment supports real-time , offline - single , offline - daily routine three types of tasks. If you want the data to be collected and reported by the SDK, offline batch transmission to the destination address, you can choose the task type as offline - single or offline - daily routine .
   Output content	Offline export supports the selection of behavioral data and user data . Data that is not dropped due to disabling or non-compliance with data specifications will not be transferred to the destination address. warning Privatization Since the 5.3.1 version, it supports exporting user sub-group data to the media platform. This function is a value-added function. You need to buy and turn on the function switch before using it. You need to configure the account authorization operation of the corresponding media platform according to the export needs. Please refer to the function details Media Authorization. This chapter focuses on the operation of synchronizing user behavior data and user data through offline tasks. For details on the task configuration of synchronizing user groups, please refer to the following Offline task: synchronizing user grouping datachapters.
   Offline time range / Offline end time	Select the configuration according to the task type: Offline - Single Task: Configure a single transfer time range, and the DataFinder will start transferring data within this time range T + 1 days after the task is created. Offline - Daily Routine Tasks: Configure the end time of a routine offline transfer. The DataFinder will start transferring data T + 1 days after the task is created, and stop data transfer after the end time. warning Offline tasks can export up to 30 days' worth of historical data at a time.
   Application	Select which application to transfer the collected data to the destination address.

2. Data connection configuration.

   Parameter	Parameter description
   Connection protocol	Select FTP or SFTP.
   Access method	Select the standard version by default.
   Connection address	Configure the FTP/SFTP server address in IP: port format. Example: XXX. 168.0.1: 8080
   Authentication method	Default does not authenticate. If the server sets a user name and password for authentication connection, select PASSWORD mode and enter the server account password.
   Remote storage path	That is, the specific storage directory.
   Test access connection	You can click Test Access Connection to test whether the FTP/SFTP server is connected. Only tasks that have been connection-verified can be created.

3. Configuration data management.

   tip

   After privatization 4.4.1 (inclusive), new filter conditions can be added to select the range of data distributed according to the combination of event and attribute rules.

   

   Parameter	Parameter description
   Add event	Configure the list of events to be distributed to the destination address. The event list contains logic that only selected events will be distributed.
   Event property addition rule	The added events are filtered by the event attribute, that is, only data within the scope of the event list and meeting the rules of the event attribute will be distributed.

Offline task: synchronizing user grouping data #

The following describes the task configuration details for synchronizing user group data to the media platform through offline tasks.

1. Configuration basic information.
   
   The configuration parameters of the basic information are similar to the above-mentioned synchronization tasks for user behavior data and user data. Please refer to the configuration parameter description above.

2. Configure clustering rules.

   Parameter	Parameter description
   Push channel	In the drop-down box, select which media platform you want to push user clustering data to.
   Selective grouping	Select the created user cluster to be synchronized.
   Group name after push	After setting the user group name on the DataFinder to sync to the media platform, the user group name on the media platform side.
   Push account	Choose which account to use to sync data to the media platform, you can select the media account that has been authorized in the drop-down box, see the media authorization operation Media Authorization.
   Update method	Sets how data synchronization tasks are triggered. Current support: manual or automatic updates. If you choose the automatic update method, you also need to configure the automatic update time period. Currently, it is supported to set the automatic update time period at the granularity of day, week, and month. For example, setting the automatic update to be triggered every Wednesday morning at 09:00:00.
   Life Cycle	Set the retention time of user grouping data after synchronization to the media platform, time unit: days.

3. Click OK when you are finished configuring.

(Optional) Manually trigger tasks #

If you create a data output task that is manually triggered, you can trigger the task manually after the task is created.
Click the "Update" button after the task list to manually trigger the data push.

Data output result confirmation #

The data output task is expected to take effect within 10 minutes after it is triggered. You can check whether the receiver has received the data normally after 10 minutes.

View/stop/edit tasks #

You can view all data output tasks on the data output task list page:

• After the task is created, the default state is "in progress", and the data is distributed normally at this time. If you need to stop data output, you can click "Stop" to change the status to "Deactivated". The task switch status is expected to take effect within 10 minutes.
  
• An in-progress task cannot be edited or deleted. If you need to modify the data transfer address, please deactivate the in-progress task first, and then re-enable it after editing.
• For offline tasks, you can view the latest execution status of subtasks.
  
  • The execution time of the task is the time range of the behavior data corresponding to the current transmission task, and is segmented by day granularity by default.
  • The last push time represents the time when the task is executed. The latest status of the task is whether the sub-task has completed the push. It is recommended to focus on the task in the failed state and try to push it again.
  • We will additionally count the number of failures. If you find that there are too many failures, you can also easily re-push it directly in the operation bar.

Real-time transmission: Distributing data examples #

The body of a request is a JSON array that contains 1 to 50 events.

[{
  "user": {
    "user_unique_id": "mock_user_unique_id",
    "user_id": 6934486383370142000,
    "user_type": 13,
    "user_is_auth": false,
    "user_is_login": false,
    "is_upgrade_user": false,
    "web_id": 6934486383370142000,
    "ip_addr_id": 0,
    "ssid": "acf3dd8f-7a18-42b1-996b-56a20156249c"
  },
  "header": {
    "app_id": 10000010,
    "app_name": "mock_web",
    "access": "",
    "client_ip": "221.122.122.122",
    "carrier": "",
    "os_name": "mac",
    "os_version": "10_15_5",
    "product_id": 107,
    "product_name": "default_to_b",
    "custom": "{\"app_id\":10000010,\"screen_width\":1440,\"screen_height\":900}",
    "trace_id": "16015901108710420012361360828522",
    "language": "zh-CN",
    "device_model": "Macintosh",
    "resolution": "1440x900",
    "width": 1440,
    "height": 900,
    "timezone": 8,
    "tz_offset": -28800,
    "platform": "web",
    "browser": "Chrome",
    "browser_version": "78.0.3904.108",
    "referrer": "",
    "referrer_host": ""
  },
  "params": "{\"app_name\":\"mock_web\",\"referrer\":\"\",\"user_unique_id\":\"\",\"time\":1601590110322,\"is_bav\":1,\"title\":\"测试页面\",\"event_index\":1616590857270,\"url\":\"http://demo.com.cn/product/list\",\"url_path\":\"/product/list\"}",
  "event_name": "predefine_pageview",
  "session_id": "aa7b79a1-4e27-44fe-bed8-56adfffddc07",
  "datetime": 1601590110,
  "server_time": 1601590110,
  "rnd": "ne0000",
  "log_type": "mario_event",
  "local_time_ms": 1601590110322
},{其他事件},{其他事件}]

Real-time transmission: individual event specific field descriptions #

//An event
Event {
          User user //The format requirements within user are described in the following text
          Header header //The format requirements within header are described in the following text
          string params  //Event parameters, single-layer json map, need to be dumped into a string
          string event_name  // Event name
          string session_id
          uint64 server_time  //The time log reached server
          string log_type 
          uint64 local_time_ms    //Event happen timestamp(millisecond)
}

//user field inside event

User {
     string user_unique_id  // unique User id
     uint64 device_id  // equipment id
     string open_udid  // openudid
     string client_udid  // device id generated by the client itself
     string mc    // equipment mac address
     string build_serial    // hardware serial number
     string serial_number    // serial number
     string idfa    // idfa
     uint64 web_id 
     string ssid 
     string idfv 
     string oaid 
     string user_unique_id_type    // The caliber of the user_unique_id type field, used in tob scenarios, the default caliber is empty or default_user_unique_id_type
}

//header field inside event

Header {
     uint32 app_id  // app_id
     string app_name  // app name
     string app_version  // app version
     string app_package  // app package name
     string display_name  // app name
     string app_channel  // app distribution channel
     string access  // network access type
     string client_ip  // client ip
     string carrier    // carrier
     string os_name    // operating system name
     string os_version    // operating system version
     string custom // custom event public attribute, which is a json string
     uint32 client_port    // client port
     string app_key    // app key
     uint32 version_code    // version code
     string vendor_id    // app vendor id
     string app_language    // app language 
     string language    // operating system language
     string app_region    // app setting region
     string region    // operating system region
     string device_model    // device model
     string device_brand    // device brand
     string device_manufacturer    // device manufacturer
     string resolution    // screen resolution
     string display_density    // screen pixel display level
     uint32 density_dpi    // screen pixel display density
     sint32 width    // screen width
     sint32 height    // screen height
     string cpu_abi    // cpu type
     float timezone    // timezone
     string tz_name    // timezone name
     sint32 tz_offset    // timezone offset
     string mcc_mnc    // nation+running trademark record
     uint32 os_api    // client operating system API version number
     string rom    // rom
     string rom_version    // rom version
     string sim_region    // sim region
     string sim_serial_number    // sim series number
     bool is_jailbroken    // is the device jailbroken
     string platform    // platform type
     string ab_sdk_version    // ab_test field
     string utm_source    // advertisement source
     string user_agent    // user_agent
     string utm_medium    // advertisement medium
     string utm_campaign    // advertisement name
     string browser 
     string browser_version 
     string referrer 
     string referrer_host 
     string app_version_minor    //4-digit version number
     string utm_term    //advertisement keyword term
     string utm_content    //advertisement content
     string sdk_version_v2    //string type sdk version
     string tracer_data    // SDK tracer custom field，json string
     uint64 harmony_os_api    // 524~530 are unique identifiers of the HarmonyOS system
     string harmony_os_version 
     string harmony_release_type 
     uint64 harmony_build_version 
     uint64 harmony_major_version 
     uint64 harmony_senior_version 
     uint64 harmony_feature_version 
 
}

Offline transfer: FTP/SFTP file format description #

After the offline task is started, it will create a directory according to the configured remote path, and divide the data into one or more compressed files according to certain rules. Each file is a multi-line JSON format reporting event, as follows:

	Explain	Example
Directory structure	Daily routine export: Based on the customer configuration path, generate subdirectories according to the data date for storing the data of the corresponding date. Single export: Create corresponding date subdirectories according to the exported date range The directory structure is: {Client configuration upload path}/{% Y-% m-% d date}/	Routine export: The July 11, 2024 run exports the data for July 10, generating: /{client defined root path}/2024-07-10/ Single export: The July 11, 2024 run exports the data from July 8-9, generating: /{client defined root path}/2024-07-08/ /{client defined root path}/2024-07-09/
File name	In the folder corresponding to the date, divide it into 1 to multiple gz files according to the data size of the corresponding date, and name it as follows: {Project id} _ {application id} _ {export task name} _part {prat_num} _ {16-bit random_id} .txt.gz Where prat_num starts at 0, if there is only 1 document on that day, this rule is still followed	1_10000000_test1_2023-10-18_part0_rvgycnvjxpunuevg.txt.gz
Document content description	A single file is a gz compressed file, and after decompression, it is a txt file with multiple lines of data, each of which contains 1 event in json format. See the data example for the JSON format.	For example: 1_10000000_test1_2023-10-18_part0_rvgycnvjxpunuevg.txt Unknown size

Offline Transmission: Example of Event Attribute Data #

Example of a single piece of data:

{
   "user": {
      "user_unique_id": "4010980525",
      "web_id": "",
      "ssid": "1729384455933548568",
      "anonymous_id": null,
      "device_id": "7279348669067838213",
      "hash_uid": "1729384455933548568"
   },
   "header": {
      "app_id": 10000000,
      "app_name": "toutiao",
      "ab_version": [],
      "custom": {
         "continent": "AS",
         "device_model": "iphone 7 plus",
         "platform": "ios",
         "getui_client_id": "A2800992-931C-4836-A884-AF6D6B35D064",
         "platform_type": "web/h5",
         "browser": "safari",
         "sdk_version": "6180690",
         "browser_version": "9.1.3",
         "sdk_lib": "ios",
         "__trace_id": "1694880771043042001034172826147014",
         "session_id": "f5de541d7433774ade7931a766c6087088ba",
         "network_carrier": "at&t",
         "ad_id": "0",
         "device_brand": "apple",
         "is_login": 1,
         "width": 0,
         "age": 56,
         "height": 0
      },
      "app_language": "zh_CN",
      "app_version": "0.15",
      "timezone": "8",
      "language": "zh_CN",
      "resolution": "320*570",
      "loc_country_id": "1814991",
      "app_region": "China",
      "loc_province_id": "1796231",
      "client_ip": "180.156.197.138",
      "loc_city_id": "1796236",
      "package": "ios.package",
      "os_version": "11.4.0",
      "app_channel": "\u6296\u97f3\u89c6\u9891",
      "os_name": "ios",
      "network_type": "4g",
      "region": "China"
   },
   "sdk": null,
   "preset_user_profiles": "",
   "params": {
      "is_background": "false",
      "duration": 0,
      "session_duration": 0
   },
   "event_name": "app_launch",
   "time": {
      "event_time": 1694879843000,
      "server_time": 1694879843000,
      "event_date": "2023-09-16"
   }
}

Offline Transfer: User Data Distribution Data Example #

Examples of data are as follows:

{
    "app_id": "10000***",
    "project_id": "1",
    "subject_id": "1",
    "ssid": "172938885398003****",
    "stat_standard_id": "172938885398003****",
    "device_id": "",
    "web_id": "745710618994439****",
    "user_id": "879303****",
    "user_unique_id": "879303****",
    "hash_uid": "172938885398003****",
    "update_time": "1736242840766",
    "last_active_date": "2025-01-06",
    "user_profiles": {
        "gender": "female",
        "campaign_name": "",
        "cheat_type": ""
    }
}

{
        "gender": "female",
        "campaign_name": "",
        "cheat_type": ""
    }

Offline Transmission: Mapping of Quadratic Resolved Fields #

The data transmitted offline may additionally contain pre-set attributes that have undergone secondary processing, such as the region ID (which needs to be obtained in Chinese using the mapping table), whether to log in, and so on. During the transmission process, such data is stored in the form of an ID, and the mapping relationship between the ID and its corresponding Chinese and English names, attribute values, etc. is shown in the following table. You can refer to the following table to map the received data for subsequent query analysis.

geo_info.csv

16.44MB

Offline transmission: Offline data field description #

Q1 Can I obtain historical data? #

Real-time tasks are not supported. Please self-push historical data through offline tasks, and support one-time export of historical data for up to T-30 days.
In addition, if a deactivated real-time task is re-enabled, the data during the deactivation period cannot be traced back.

Q2 Data sending time? #

Real-time data is normally sent within 60 seconds after the data is reported.

Q3 What should I do if the data transmission fails? #

For the sending failure caused by the service reception of the business itself, the current system has the following retry mechanism:
In a privatized environment, there is no limit to retry within the same day; if a piece of data fails three times in a row after version 4.6 (inclusive), the system will no longer push the message; within 7 days, you can contact the customer success manager or Volcano's technical support to manually trigger the push, and it will be automatically deleted after expiration. Recovery is no longer supported

Q4 Why is the number of data pieces accepted different from the indicators of interface statistics? #

It cannot be ruled out that some data may be repeatedly pushed due to various reasons such as network instability. For special circumstances, please contact your Customer Success Manager or Volcano's technical support.