Description #

English text quality scoring operator - Text quality evaluation based on FastText

Key features #

• Quality scoring: Uses the FastText model to score the quality of English text, with a preference for scientific knowledge, and supports only CPU environments.
• Batch processing: Supports batch processing of text to improve processing efficiency

Scoring criteria #

• 0: Low quality (Low)
• 1: Moderate quality (Mid)
• 2: High quality (High)
• The final score is a floating-point number between 0 and 2; the higher the score, the better the quality.
• Generally, a score above 0.5 indicates relatively good text quality.

Operator parameters #

Input #

Output #

A column containing text quality scores, with element type float64.

Parameters #

If a parameter does not have a default value, it is required.

Examples #

The following code demonstrates how to use daft to run the operator and score the quality of English text based on the FastText model.

from __future__ import annotations

import os

import daft
from daft import col
from daft.las.functions.text.en_text_quality_scorer import EnTextQualityScorer
from daft.las.functions.udf import las_udf

if __name__ == "__main__":

    if os.getenv("DAFT_RUNNER", "native") == "ray":
        import logging

        import ray

        def configure_logging():
            logging.basicConfig(
                level=logging.INFO,
                format="%(asctime)s - %(name)s - %(levelname)s - %(message)s",
                datefmt="%Y-%m-%d %H:%M:%S.%s".format(),
            )
            logging.getLogger("tracing.span").setLevel(logging.WARNING)
            logging.getLogger("daft_io.stats").setLevel(logging.WARNING)
            logging.getLogger("DaftStatisticsManager").setLevel(logging.WARNING)
            logging.getLogger("DaftFlotillaScheduler").setLevel(logging.WARNING)
            logging.getLogger("DaftFlotillaDispatcher").setLevel(logging.WARNING)

        ray.init(dashboard_host="0.0.0.0", runtime_env={"worker_process_setup_hook": configure_logging})
        daft.set_runner_ray()
    daft.set_execution_config(actor_udf_ready_timeout=600)
    daft.set_execution_config(min_cpu_per_task=0)

    samples = {"text": ["This is a well-written scientific article about quantum physics.", None]}
    batch_size = 4
    model_path = os.getenv("MODEL_PATH", "/opt/las/models")
    model_name = "llm-data-textbook-quality-fasttext-classifier-v2/model_quantized.bin"

    ds = daft.from_pydict(samples)
    ds = ds.with_column(
        "quality_score",
        las_udf(
            EnTextQualityScorer,
            construct_args={
                "batch_size": batch_size,
                "model_path": model_path,
                "model_name": model_name,
            },
            num_gpus=0,
            batch_size=1,
            concurrency=1,
        )(col("text")),
    )

    ds.show()
    # ╭──────────────────────────────────────────────────────────────────────────┬─────────────────────╮
    # │ text                                                                     ┆ quality_score       │
    # │ ---                                                                      ┆ ---                 │
    # │ Utf8                                                                     ┆ Float64             │
    # ╞══════════════════════════════════════════════════════════════════════════╪═════════════════════╡
    # │ This is a well-written scientific article about quantum physics. ┆ 0.6918241381645203  │
    # ├╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌┼╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌┤
    # │ None                                                                     ┆ None                │
    # ╰──────────────────────────────────────────────────────────────────────────┴─────────────────────╯