Description #

Multilingual text quality scoring operator - Multilingual text quality assessment based on the E5 model

Key features #

• Multilingual support: Uses the multilingual-e5 model to support quality scoring for texts in multiple languages
• Deep learning evaluation: Performs quality assessment using the E5 model based on the Transformer architecture
• GPU acceleration: Supports GPU inference acceleration to improve processing efficiency
• Batch processing: Supports batch processing of texts to optimize inference performance

Scoring criteria #

• Output range: Floating-point numbers between 0 and 1
• The higher the score, the better the text quality
• Generally, a score above 0.5 indicates good text quality

Supported languages #

• English, Chinese, Japanese, Korean, French, German, Spanish, and many other languages
• Based on the multilingual-e5-small-aligned-quality model

Operator parameters #

Input #

Output #

A column containing text quality scores, element type is float32

Parameters #

If a parameter does not have a default value, it is required

Examples #

The following code demonstrates how to use daft to run the operator and score multilingual text quality based on the E5 model.

from __future__ import annotations

import os

import daft
from daft import col
from daft.las.functions.text.multilingual_text_quality_scorer import MultilingualTextQualityScorer
from daft.las.functions.udf import las_udf

if __name__ == "__main__":

    if os.getenv("DAFT_RUNNER", "native") == "ray":
        import logging

        import ray

        def configure_logging():
            logging.basicConfig(
                level=logging.INFO,
                format="%(asctime)s - %(name)s - %(levelname)s - %(message)s",
                datefmt="%Y-%m-%d %H:%M:%S.%s".format(),
            )
            logging.getLogger("tracing.span").setLevel(logging.WARNING)
            logging.getLogger("daft_io.stats").setLevel(logging.WARNING)
            logging.getLogger("DaftStatisticsManager").setLevel(logging.WARNING)
            logging.getLogger("DaftFlotillaScheduler").setLevel(logging.WARNING)
            logging.getLogger("DaftFlotillaDispatcher").setLevel(logging.WARNING)

        ray.init(dashboard_host="0.0.0.0", runtime_env={"worker_process_setup_hook": configure_logging})
        daft.set_runner_ray()
    daft.set_execution_config(actor_udf_ready_timeout=600)
    daft.set_execution_config(min_cpu_per_task=0)

    samples = {
        "text": [
            "这是一篇关于人工智能技术发展的高质量学术论文，内容详实且具有很强的参考价值。",
            "これは量子物理学とその現代技術への応用に関するよく書かれた科学論文です。",
            "이것은 양자물리학과 현대 기술에의 응용에 관한 잘 쓰여진 과학 논문입니다.",
            None,
        ]
    }

    model_path = os.getenv("MODEL_PATH", "/opt/las/models")
    model_name = "multilingual-e5-small-aligned-quality"
    dtype = "float32"
    batch_size = 100

    ds = daft.from_pydict(samples)
    ds = ds.with_column(
        "quality_score",
        las_udf(
            MultilingualTextQualityScorer,
            construct_args={
                "model_path": model_path,
                "model_name": model_name,
                "dtype": dtype,
                "batch_size": batch_size,
                "rank": 0,
            },
            num_gpus=1,
            batch_size=1,
            concurrency=1,
        )(col("text")),
    )

    ds.show()
    # ╭──────────────────────────────────────────────────────────────────────────┬─────────────────────╮
    # │ text                                                                     ┆ quality_score       │
    # │ ---                                                                      ┆ ---                 │
    # │ Utf8                                                                     ┆ Float32             │
    # ╞══════════════════════════════════════════════════════════════════════════╪═════════════════════╡
    # │ This is a high-quality academic paper on the development of artificial intelligence technology, with detailed content and strong reference value.           ┆ 0.6294931           │
    # ├╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌┼╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌┤
    # │ これは量子物理学とその現代技術への応用に関するよく書かれた科学論文です。              ┆ 0.73835784          │
    # ├╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌┼╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌┤
    # │ 이것은 양자물리학과 현대 기술에의 응용에 관한 잘 쓰여진 과학 논문입니다。                   ┆ 0.7729334           │
    # ├╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌┼╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌┤
    # │ None                                                                     ┆ None                │
    # ╰──────────────────────────────────────────────────────────────────────────┴─────────────────────╯