Description #

Word repetition ratio calculator - Text feature extraction based on N-gram phrase repetition ratio

Key features #

• Word repetition ratio calculation: Accurately calculates the proportion of repeated phrases in the text
• Bilingual support: Supports tokenization for both Chinese and English
• Flexible configuration: Supports calculation of N-gram phrases of different lengths

Application scenarios #

• Text quality assessment
• Data cleaning and preprocessing
• Content duplication detection

Technical features #

• Supports two tokenization modes:
  • Whitespace-based tokenization: Suitable for languages separated by spaces, such as English
  • Model-based tokenization: Uses the sentencepiece model; this mode must be used for Chinese
• Configurable N-gram length: Supports combinations of 1 to N words
• Intelligent filtering: Automatically filters special characters to improve calculation accuracy

Operator parameters #

Input #

Output #

Word repetition ratio column, elements are floating-point numbers representing the proportion of repeated phrases

Parameters #

If a parameter does not have a default value, it is required

Examples #

The following code demonstrates how to use daft to run the operator to calculate the N-gram-based phrase repetition ratio in text.

from __future__ import annotations

import os

import daft
from daft import col
from daft.las.functions.text.word_repetition_calculator import WordRepetitionCalculator
from daft.las.functions.udf import las_udf

if __name__ == "__main__":

    if os.getenv("DAFT_RUNNER", "native") == "ray":
        import logging

        import ray

        def configure_logging():
            logging.basicConfig(
                level=logging.INFO,
                format="%(asctime)s - %(name)s - %(levelname)s - %(message)s",
                datefmt="%Y-%m-%d %H:%M:%S.%s".format(),
            )
            logging.getLogger("tracing.span").setLevel(logging.WARNING)
            logging.getLogger("daft_io.stats").setLevel(logging.WARNING)
            logging.getLogger("DaftStatisticsManager").setLevel(logging.WARNING)
            logging.getLogger("DaftFlotillaScheduler").setLevel(logging.WARNING)
            logging.getLogger("DaftFlotillaDispatcher").setLevel(logging.WARNING)

        ray.init(dashboard_host="0.0.0.0", runtime_env={"worker_process_setup_hook": configure_logging})
        daft.set_runner_ray()
    daft.set_execution_config(actor_udf_ready_timeout=600)
    daft.set_execution_config(min_cpu_per_task=0)

    samples = {
        "text": [
            "",
            "",
            None,
        ]
    }
    repetition = 2
    lang = "zh"
    tokenization = True

    ds = daft.from_pydict(samples)
    ds = ds.with_column(
        "repetition_ratio",
        las_udf(
            WordRepetitionCalculator,
            construct_args={
                "repetition": repetition,
                "lang": lang,
                "tokenization": tokenization,
                "model_path": os.getenv("MODEL_PATH", "/opt/las/models"),
                "model_name": "kenlm/wikipedia",
            },
        )(col("text")),
    )

    ds.show()
    # ╭─────────────────────────────────────────────────────────────┬─────────────────────╮
    # │ text                                                        ┆ repetition_ratio    │
    # │ ---                                                         ┆ ---                 │
    # │ String                                                      ┆ Float64             │
    # ╞═════════════════════════════════════════════════════════════╪═════════════════════╡
    # │ Artificial intelligence technology is developing rapidly, and the application of artificial intelligence technology in various fields is becoming more and more... ┆ 0.14285714285714285 │
    # ├╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌┼╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌┤
    # │ This is a sample text without repeated words; each word appears only once.       ┆ 0                   │
    # ├╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌┼╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌┤
    # │ None                                                        ┆ None                │
    # ╰─────────────────────────────────────────────────────────────┴─────────────────────╯