Operator ID: daft.las.functions.audio.audio_ctc_aligner.AudioCTCAligner

Description #

The multilingual audio CTC alignment operator (AudioCTCAligner) is a multilingual audio CTC alignment operator based on the MMS Forced Aligner model (CTC, Connectionist Temporal Classification), designed to precisely align audio content with the corresponding transcript along the time dimension.

Key features #

• Support for multilingual input: Natively supports text alignment in Chinese and English, meeting the requirements for mainstream languages.
• Support for multi-channel input: Seamlessly processes audio data from local file paths, HTTP URLs, TOS/S3 object storage, or raw byte streams.
• Automated preprocessing: Built-in text preprocessing workflows for Chinese and English, including case conversion, number expansion, pinyin conversion, and romanization to meet model input requirements. In addition, built-in audio decoding and preprocessing capabilities automatically resample input audio to 16 kHz mono format, simplifying the usage process.

Supported models #

Audio CTC alignment is implemented based on the MMS Forced Aligner model. You can click here to download the corresponding model file.

Operator parameters #

Input #

Output #

The operator organizes classification results using a JSON array (as shown below, each input audio corresponds to a JSON array object). Each element in the array contains word, score (confidence), start (start timestamp), and end (end timestamp), with timestamps in milliseconds.

[
    {"word": "i", "score": 1.0, "start": 644, "end": 664},
    {"word": "had", "score": 0.98, "start": 704, "end": 845},
    {"word": "that", "score": 1.0, "start": 885, "end": 1026},
    {"word": "curiosity", "score": 1.0, "start": 1086, "end": 1790},
    {"word": "beside", "score": 0.97, "start": 1871, "end": 2314},
    {"word": "me", "score": 1.0, "start": 2334, "end": 2414},
    {"word": "at", "score": 1.0, "start": 2495, "end": 2575},
    {"word": "this", "score": 1.0, "start": 2595, "end": 2756},
    {"word": "moment", "score": 1.0, "start": 2837, "end": 3138},
]

Parameters #

If a parameter does not have a default value, it is required.

Examples #

The following code demonstrates how to use Daft to run the operator for CTC alignment of speech and text.

from __future__ import annotations

import logging
import os

import daft
from daft import col
from daft.las.functions.audio.audio_ctc_aligner import AudioCTCAligner
from daft.las.functions.udf import las_udf

if __name__ == "__main__":
    if os.getenv("DAFT_RUNNER", "ray") == "ray":

        def configure_logging():
            logging.basicConfig(
                level=logging.INFO,
                format="%(asctime)s - %(name)s - %(levelname)s - %(message)s",
                datefmt="%Y-%m-%d %H:%M:%S.%s".format(),
            )
            logging.getLogger("tracing.span").setLevel(logging.WARNING)
            logging.getLogger("daft_io.stats").setLevel(logging.WARNING)
            logging.getLogger("DaftStatisticsManager").setLevel(logging.WARNING)
            logging.getLogger("DaftFlotillaScheduler").setLevel(logging.WARNING)
            logging.getLogger("DaftFlotillaDispatcher").setLevel(logging.WARNING)

        import ray

        ray.init(dashboard_host="0.0.0.0", runtime_env={"worker_process_setup_hook": configure_logging})
        daft.set_runner_ray()

    daft.set_execution_config(actor_udf_ready_timeout=600)
    daft.set_execution_config(min_cpu_per_task=0)

    tos_dir_url = os.getenv("TOS_DIR_URL", "las-cn-beijing-public-online.tos-cn-beijing.volces.com")
    samples = {
        "audio_path": [f"https://{tos_dir_url}/public/shared_audio_dataset/参观八达岭长城。.wav"],
        "text": ["参观八达岭长城"],
        "lang": ["zh"],
    }

    df = daft.from_pydict(samples)
    df = df.with_column(
        "ctc_result",
        las_udf(
            AudioCTCAligner,
            construct_args={
                "model_path": "/opt/las/models",
                "model_name": "MMS/ctc_alignment_mling_uroman_model.pt",
            },
            num_gpus=0.1,
            batch_size=16,
            concurrency=8,
        )(col("audio_path"), col("text"), col("lang")),
    )
    df.show()