Description #

Chinese simplified-traditional conversion operator based on OpenCC

Key features #

• Multi-directional conversion: Supports multiple conversion directions including simplified, traditional, Taiwan traditional, and Hong Kong traditional
• Mixed text processing: Correctly handles content mixed with Chinese and English, converting only the Chinese portion
• Efficient batch processing: Supports fast conversion for large volumes of text

Technical implementation #

• Conversion engine: Uses OpenCC for conversion

Operator parameters #

Input #

Output #

The converted text column, with element type string.

Parameters #

If a parameter does not have a default value, it is required.

Examples #

The following code demonstrates how to use daft to run the operator for Chinese simplified-traditional conversion.

from __future__ import annotations

import os

import pandas as pd

import daft
from daft import col
from daft.las.functions.text.chinese_text_converter import ChineseTextConverter
from daft.las.functions.udf import las_udf

if __name__ == "__main__":

    if os.getenv("DAFT_RUNNER", "native") == "ray":
        import logging

        import ray

        def configure_logging():
            logging.basicConfig(
                level=logging.INFO,
                format="%(asctime)s - %(name)s - %(levelname)s - %(message)s",
                datefmt="%Y-%m-%d %H:%M:%S.%s".format(),
            )
            logging.getLogger("tracing.span").setLevel(logging.WARNING)
            logging.getLogger("daft_io.stats").setLevel(logging.WARNING)
            logging.getLogger("DaftStatisticsManager").setLevel(logging.WARNING)
            logging.getLogger("DaftFlotillaScheduler").setLevel(logging.WARNING)
            logging.getLogger("DaftFlotillaDispatcher").setLevel(logging.WARNING)

        ray.init(dashboard_host="0.0.0.0", runtime_env={"worker_process_setup_hook": configure_logging})
        daft.set_runner_ray()
    daft.set_execution_config(actor_udf_ready_timeout=600)
    daft.set_execution_config(min_cpu_per_task=0)

    samples = {
        "text": [
            "這是一個繁體中文的測試文本，包含了一些專業術語和技術名詞。",
            "Hello 世界！這裡有中英文混雜的內容，測試OpenCC是否能正確處理。",
            "Mixed content: 繁體字轉換測試 with English words and numbers 123。",
            None,
        ]
    }

    df = pd.DataFrame(samples)
    ds = daft.from_pandas(df)

    ds = ds.with_column(
        "converted_text",
        las_udf(
            ChineseTextConverter,
            construct_args={
                "direction": "t2s",
            },
            num_gpus=0,
            batch_size=1,
            concurrency=1,
        )(col("text")),
    )

    ds.show()
    # ╭─────────────────────────────────────────────────────────────────────────────────┬─────────────────────────────────────────────────────────────────────────────────╮
    # │ text                                                                            ┆ converted_text                                                                  │
    # │ ---                                                                             ┆ ---                                                                             │
    # │ Utf8                                                                            ┆ Utf8                                                                            │
    # ╞═════════════════════════════════════════════════════════════════════════════════╪═════════════════════════════════════════════════════════════════════════════════╡
    # │ 這是一個繁體中文的測試文本，包含了一些專業術語和技術名詞。 ┆ This is a test text in traditional Chinese, containing some technical terms and technical vocabulary.                                 │
    # ├╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌┼╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌┤
    # │ Hello 世界！ 這裡有中英文混雜的內容，測試OpenCC是否能正確處理。 ┆ Hello 世界！ This content contains a mix of Chinese and English, testing whether OpenCC can handle it correctly.                           │
    # ├╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌┼╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌┤
    # │ Mixed content: 繁體字轉換測試 with English words and numbers 123。 ┆ Mixed content: Traditional character conversion test with English words and numbers 123.                 │
    # ├╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌┼╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌┤
    # │ None                                                                            ┆ None                                                                            │
    # ╰─────────────────────────────────────────────────────────────────────────────────┴─────────────────────────────────────────────────────────────────────────────────╯