This document introduces the usage of feature engineering module operators, including binarization, principal component analysis, Cartesian product, discrete cosine transform, row normalization, column normalization, singular value decomposition, feature hashing, one-hot encoding, Array processing, Map processing, feature importance, feature selection, and many others.

Feature engineering generally refers to the process of transforming raw data into features that better represent the essence of the problem, so that using these features in predictive models can improve the model's prediction accuracy on unseen data. After completing data input, feature engineering operators can be used to further process the input data. The following sections provide a detailed introduction to these operators.

Binarization #

Binarization converts numerical features into binary features, 0 or 1. In the field of data mining, the purpose of binarization is to divide quantitative features into "yes or no" categories to eliminate redundant information.
Usage example:
Example 1: Customer value segmentation

• Operation field: "Annual spending amount"
• Parameter settings:
  • Selection type: single
  • Binary classification threshold: 10 000
  • Replace original column: Yes
  • Output column name: High-value customer flag
• Output result:
  • Original value ≥ 10000 → Converted to 1
  • Original value < 10000 → Converted to 0

Example 2: Financial risk assessment

• Operation field: "Debt-to-asset ratio"
• Parameter settings:
  • Selection type: single
  • Binary classification threshold: 0.7
  • Replace original column: No
  • Output column suffix: _risk flag
• Output result:
  • Add new column "Debt-to-asset ratio_risk flag"
  • Original value ≥ 0.7 → New column value is 1
  • Original value < 0.7 → New column value is 0

Parameter description:

Principal component analysis #

Principal component analysis (PCA) is a dimensionality reduction algorithm that maps high-dimensional features to a low-dimensional orthogonal space (principal components), reducing the number of features while retaining most of the original information. This function only supports operations on attributes that are numeric and have the role "regular". The output results are usually used as input for other analytical models.

Usage example:
Example 1: Dimensionality reduction for customer credit scoring

• Business scenario: A bank needs to perform dimensionality reduction on 15 customer financial indicators
• Parameter settings:
  • Feature columns: Select all 15 financial indicators (such as "annual income", "debt ratio", "credit card limit", and more)
  • Dimensionality reduction method: Select "keepVariance"
  • Percentage: Set to 0.90 (retain 90% of the original information) (after execution, the system automatically determines to retain 5 principal components)
• Result application: Use the 5 output principal components as inputs for the credit scoring model

Example 2: Visualization of product sales data

• Business scenario: An e-commerce platform wants to visualize 20 product features after dimensionality reduction
• Parameter settings:
  • Feature columns: Select 20 sales-related features (such as "monthly sales volume", "number of favorites", "click-through rate", and more)
  • Dimensionality reduction method: Select "fixedNumber"
  • k value: Set to 3 (for three-dimensional visualization)
• Result application: Use the 3 principal components to perform three-dimensional scatter plot analysis of product clustering

Parameter description:

Cartesian product #

The Cartesian product refers to the Cartesian product of two sets X and Y, also known as the direct product, denoted as X × Y. Each ordered pair consists of a member from X and a member from Y.
Usage example:
Example 1: Generating product specification combinations

• Original data table (products)

  product_id	length	width	height
  1001	30	20	10
  1002	40	25	15

• Parameter settings:

  • Left operand column: length
  • Right operand column: width

• Output result:

  product_id	length	width	height	interaction result
  1001	30	20	10	600
  1002	40	25	15	1000

Parameter description:

Discrete cosine transform #

The discrete cosine transform (DCT) is a digital signal processing technique that converts time-domain signals into frequency-domain representations. This function uses a mathematical transformation to convert a real-valued sequence of length N into a frequency-domain coefficient sequence of the same length, mainly used for data compression and feature extraction.

Usage example:
Example 1: Audio signal feature extraction

• Business scenario: Extracting frequency-domain features from audio waveform signals for speech recognition or compression.

• Parameter settings:

  • Feature column: audio_waveform (audio sample point sequence)
  • Inverse: false (forward DCT, time domain → frequency domain)
  • Output column: feature_transformed (frequency-domain coefficients are automatically generated)

• Output effect:

  audio_waveform (original signal)	feature_transformed (DCT coefficients)
  0.1	12.8
  0.5	5.3
  -0.2	-1.2
  ...	...

Parameter description:

Row normalization #

Row normalization is a data standardization method that normalizes each row of data, ensuring that all rows are comparable on the same scale, which facilitates subsequent comprehensive analysis and modeling.
Usage example:
Example 1: Normalizing user behavior data

• Original data:

  User ID	Click count	Browsing duration (seconds)	Purchase amount (CNY)
  1001	20	300	500
  1002	50	600	200

• Parameter settings:

  • Feature columns: ["Number of clicks", "Browsing duration", "Purchase amount"]
  • p-value: 2 (Euclidean normalization)

• Result after processing:

  User ID	feature_transformed (array format)
  1001	[0.036, 0.539, 0.842]
  1002	[0.078, 0.937, 0.312]

Parameter description:

Column normalization #

Column normalization is a data preprocessing method that scales specified columns to make data with different units comparable. It supports three scaling methods: standardization (standard), min-max scaling (min-max), and max-abs scaling (max-abs). In-place modification is not supported for now. The generated result is a single array column. To split it into separate columns, add a vector decomposer operator after this operator.

Usage example:
Example 1: Standardization (standard) scaling

• Calculation formula: (x - mean of 5000) / standard deviation of 2449.49

  User ID	Original sales amount	Standardization result (std_scale)
  1001	2000	-1.2247
  1002	5000	0
  1003	8000	1.2247

Example 2: Min-max scaling

• Calculation formula: (x - min of 18) / (max of 52 - min of 18)

  User ID	Original age	Min-max result (std_scale)
  1001	18	0.0
  1002	35	0.5
  1003	52	1.0

Example 3: Max-abs scaling example

• Calculation formula: x / max(abs(800), abs(-400))

  User ID	Original profit	Max-abs result (std_scale)
  1001	-400	-0.5
  1002	200	0.25
  1003	800	1.0

Parameter description:

Singular value decomposition #

Singular value decomposition (SVD) is a dimensionality reduction method that simplifies numerical data by projecting it into a lower-dimensional space using higher singular values. This feature only supports operations on numerical attributes with the role "regular". The output result usually needs to be used as input for other analytical models.

Usage example:
Example 1: Dimensionality reduction for financial risk control features

• Business scenario: A bank needs to reduce the dimensionality of 30 highly correlated financial indicators for use in a credit risk assessment model
• Parameter settings:
  • Feature columns: Select 30 financial indicators (including "debt-to-asset ratio", "current ratio", "profit margin", and more)
  • Dimensionality reduction method: Select "keepVariance"
  • Percentage: Set to 0.85 (retain 85% of the original information) (After execution, the system automatically determines to retain 8 principal components)
• Execution result: The system automatically retains 8 principal components
• Result application: Use the 8 SVD features as input for the risk control model

Example 2: Retail user profile construction

• Business scenario: The e-commerce platform needs to reduce the dimensionality of 100-dimensional user behavior data
• Parameter settings:
  • Feature columns: Select 100 behavioral features ("click-through rate", "dwell time", "add-to-cart frequency", and more)
  • Dimensionality reduction method: Select "fixedNumber"
  • k value: Set to 10
• Execution result: Generate 10 core features
• Result application: Used for the user segmentation system to improve clustering speed

Parameter description:

Feature hashing #

Feature hashing (Hashing Trick) is a fast dimensionality reduction technique that maps high-dimensional features (such as text word frequencies and categorical variables) into a fixed-dimensional vector space using a hash function. It is suitable for feature compression of large-scale sparse data and avoids the dimensionality explosion problem of traditional one-hot encoding.
Usage examples:
Example 1: Text word frequency dimensionality reduction

• Business scenario: Compress 10,000-dimensional text word frequency features to 512 dimensions
• Parameter settings:
  • Selection type: single
  • Select column: word_counts (a dictionary column storing word frequencies, such as {"apple":3, "banana":1})
  • Number of features: 512
  • Reverse selection: false
• Execution result:
  • Original data: {"apple":3, "banana":1, ...} (more than 10,000 words)
  • Output result: [0.0, 1.2, 0.0, ..., -0.8] (512-dimensional vector)

Example 2: User behavior feature compression

• Business scenario: Compress 5,000-dimensional user behavior ID features to 256 dimensions
• Parameter settings:
  • Selection type: subset
  • Select columns: ["click_ids", "view_ids"]
  • Number of features: 256
  • Reverse selection: false
• Execution result:
  • Original data: click_ids=[123,456,...], view_ids=[789,...]
  • Output result: [0.5, 0.0, -1.3, ..., 0.7] (256-dimensional vector)

Parameter description:

One-hot encoding #

One-hot encoding is a method for converting discrete categorical variables into a form that is easier for machine learning algorithms to understand. It expands a feature with N different values into N binary features (0/1), with each new feature corresponding to one possible value of the original feature.
Usage example:
Example 1: Color feature encoding

• Original data:

  product_id	color
  1001	red
  1002	blue
  1003	green

• Parameter settings:

  • Column: ["color"]

• Processed result:

  product_id	color
  1001	[1, 0, 0]
  1002	[0, 1, 0]
  1003	[0, 0, 1]

Example 2: Simultaneous encoding of multiple columns

• Original data:

  user_id	gender	age_group
  2001	male	young
  2002	female	middle
  2003	male	old

• Parameter settings:

  • Columns: ["gender", "age_group"]

• Processed result:

  user_id	gender	age_group
  2001	[1, 0]	[1, 0, 0]
  2002	[0, 1]	[0, 1, 0]
  2003	[1, 0]	[0, 0, 1]

Parameter description:

Array processing #

Array processing is an operator for batch operations on array-type fields (such as [1.2, 0.5, -0.3]), with the core function of converting array types into one-hot encoding types.
Usage example:
Example 1: Convert array to one-hot encoding

• Original data:

  ID	raw_vector
  1	[1, 2, 3]
  2	[2, 3]
  3	[3]

• Parameter settings:

  • Column: ["raw_vector"]
  • Number of sampled data rows: 200000 (used to analyze all possible values in the array)

• After processing:

  ID	features
  1	[1, 1, 1]
  2	[0, 1, 1]
  3	[0, 0, 1]

Parameter description:

Map processing #

Map processing is an operator used with machine learning algorithms, operating on feature fields of type map<string, double>. The core function is to convert map<string, double> type to an array type with one-hot encoding. The system first samples the specified number of rows to analyze all possible keys, then processes the entire dataset uniformly.
Usage example:
Example 1: Convert map data to one-hot encoding

• Original data table

  order_id	order_detail (map<string, double> type)
  10001	{"price":299.0, "items":2.0}
  10002	{"price":599.0, "discount":0.8}

• Parameter settings:

  • Column: ["order_detail"]
  • Number of sampled data rows: 200000 (used to analyze all possible keys)

• Processed result table:

  order_id	features
  10001	[1, 1, 0]
  10002	[1, 0, 1]

Parameter description:

Feature importance #

Feature importance is used to calculate the weight of each feature in the dataset, helping to identify the features that have the greatest impact on the target variable. Multiple calculation methods are supported and are applicable to vector or single-column features. For vector columns, the output format is original_column_name__index (index starts from 0).
Usage example:
Example 1: House price prediction

• Original data table (house_data)

  house_id	area	rooms	location_score	price_label
  1001	85	3	[0.7,0.2]	1
  1002	60	2	[0.5,0.3]	0
  1003	120	4	[0.9,0.1]	1

• Parameter settings

  • Feature columns: ["area", "rooms", "location_score"]
  • Label column: price_label
  • Calculation method: gini (Gini coefficient)

• Processed result table

  column	weight
  area	0.78
  rooms	0.65
  location_score__0	0.42
  location_score__1	0.15

Example 2: Credit card default analysis

• Original data table (credit_data)

  user_id	income	debt_ratio	payment_history	default_flag
  2001	50000	0.35	[1,1,0,1]	0
  2002	32000	0.72	[0,1,0,0]	1

• Parameter settings

  • Feature columns: ["income", "debt_ratio", "payment_history"]
  • Label column: default_flag
  • Calculation method: info_gain_ratio (information gain ratio)

• Processed result table

  column	weight
  debt_ratio	0.91
  payment_history__0	0.47
  payment_history__1	0.32
  income	0.28

Parameter description:

Feature selection #

Feature selection calculates the importance weight of each feature with respect to the target variable, helping users identify key features and remove redundant ones. Supports numerical and vector features, and automatically expands vector columns (format: original_column_name__index). Applicable to preliminary feature selection for classification and regression tasks.
Usage example:
Example 1: Financial risk control feature selection

• Original data table (loan_risk)

  user_id	income	debt_ratio	behavior_scores	is_default
  1001	50000	0.3	[0.8, 0.2, 0.5]	0
  1002	32000	0.6	[0.6, 0.3, 0.1]	1

• Parameter settings

  • Feature columns: ["income", "debt_ratio", "behavior_scores"]
  • Label column: is_default
  • Calculation method: info_gain_ratio (information gain ratio)
  • Selection threshold: 0.3 (retain features with importance ≥ 0.3)

• Processed result

  Feature name	Importance score	Keep
  debt_ratio	0.85	✓
  behavior_scores__0	0.42	✓
  income	0.25	✗
  behavior_scores__1	0.18	✗
  behavior_scores__2	0.10	✗

  Final retained features: debt_ratio, behavior_scores__0

Example 2: E-commerce recommendation feature selection

• Original data table (user_behavior)

  user_id	click_count	purchase_history	user_group
  2001	20	[1, 0, 1, 1, 0]	A
  2002	5	[0, 1, 0, 0, 1]	B

• Parameter settings

  • Feature columns: ["click_count", "purchase_history"]
  • Label column: user_group
  • Computation method: gini (Gini coefficient)
  • Selection threshold: 0.4

• Result after processing

  Feature name	Importance score	Keep
  purchase_history__0	0.62	✓
  purchase_history__3	0.55	✓
  click_count	0.38	✗
  purchase_history__1	0.22	✗

  Final retained features: purchase_history__0, purchase_history__3

Parameter description: