Description #

Speech recognition module – a multilingual speech-to-text solution based on the FireRed ASR model

Key features #

• Multilingual recognition: Supports Chinese, English, and Chinese dialects
• Multiple model selection: Supports various models, including AED models and LLM models
• Audio type: Supports mono, 16 kHz sampling rate wav audio files

Recommended practices #

• To ensure recognition quality, for AED models, it is recommended that the audio length does not exceed 60 seconds; for LLM models, it is recommended that the audio length does not exceed 30 seconds
• When performing batch inference with LLM models, it is recommended to ensure that the audio lengths are similar. If the difference is large, set batch_size=1
• It is recommended to standardize the audio before use and convert it to a supported audio file format

Supported models #

• FireRedASR-AED-L
• FireRedASR-LLM-L

Operator parameters #

Input #

Output #

Processed array, with each element being the transcription for each audio file. For audio files that fail to process, an empty string is returned.

Parameters #

If a parameter does not have a default value, it is required

Examples #

The following code demonstrates how to use Daft to run the operator and convert speech to text.

from __future__ import annotations

import logging
import os

import daft
from daft import col
from daft.las.functions.audio.audio_asr_firered import AudioAsrFireRed
from daft.las.functions.udf import las_udf

if __name__ == "__main__":
    if os.getenv("DAFT_RUNNER", "ray") == "ray":

        def configure_logging():
            logging.basicConfig(
                level=logging.INFO,
                format="%(asctime)s - %(name)s - %(levelname)s - %(message)s",
                datefmt="%Y-%m-%d %H:%M:%S.%s".format(),
            )
            logging.getLogger("tracing.span").setLevel(logging.WARNING)
            logging.getLogger("daft_io.stats").setLevel(logging.WARNING)
            logging.getLogger("DaftStatisticsManager").setLevel(logging.WARNING)
            logging.getLogger("DaftFlotillaScheduler").setLevel(logging.WARNING)
            logging.getLogger("DaftFlotillaDispatcher").setLevel(logging.WARNING)

        import ray

        ray.init(dashboard_host="0.0.0.0", runtime_env={"worker_process_setup_hook": configure_logging})
        daft.set_runner_ray()

    daft.set_execution_config(actor_udf_ready_timeout=600)
    daft.set_execution_config(min_cpu_per_task=0)

    tos_dir_url = os.getenv("TOS_DIR_URL", "las-cn-beijing-public-online.tos-cn-beijing.volces.com")
    samples = {
        "audio_path": [
            f"https://{tos_dir_url}/public/shared_audio_dataset/sample_normal.wav"
        ]
    }

    model_path = os.getenv("MODEL_PATH", "/opt/las/models")
    audio_src_type = "audio_url"
    beam_size = 3
    decode_min_len = 0
    decode_max_len = 100
    nbest = 1
    softmax_smoothing = 1.25
    aed_length_penalty = 0.6
    eos_penalty = 1.0
    repetition_penalty = 3.0
    llm_length_penalty = 1.0
    temperature = 1.0
    use_fp16 = True
    batch_size = 1
    num_gpus = 1

    df = daft.from_pydict(samples)

    # Each UDF requires one GPU. If there is only one GPU, please comment out the code for one UDF and run only the other.
    df = df.with_column(
        "asr_result_aed",
        las_udf(
            AudioAsrFireRed,
            construct_args={
                "audio_src_type": audio_src_type,
                "model_path": model_path,
                "model_name": "FireRedAsr/FireRedASR-AED-L",
                "batch_size": batch_size,
                "beam_size": beam_size,
                "decode_min_len": decode_min_len,
                "decode_max_len": decode_max_len,
                "nbest": nbest,
                "softmax_smoothing": softmax_smoothing,
                "aed_length_penalty": aed_length_penalty,
                "eos_penalty": eos_penalty,
            },
            num_gpus=num_gpus,
            batch_size=1,
            concurrency=1,
        )(col("audio_path")),
    )

    df = df.with_column(
        "asr_result_llm",
        las_udf(
            AudioAsrFireRed,
            construct_args={
                "audio_src_type": audio_src_type,
                "model_path": model_path,
                "model_name": "FireRedAsr/FireRedASR-LLM-L",
                "batch_size": batch_size,
                "beam_size": beam_size,
                "decode_min_len": decode_min_len,
                "decode_max_len": decode_max_len,
                "repetition_penalty": repetition_penalty,
                "llm_length_penalty": llm_length_penalty,
                "use_fp16": use_fp16,
                "temperature": temperature,
            },
            num_gpus=num_gpus,
            batch_size=1,
            concurrency=1,
        )(col("audio_path")),
    )
    df.show()

    # ╭────────────────────────────────┬────────────────────────────────────────────┬───────────────────────────────────────────╮
    # │ audio_path                     ┆ asr_result_aed                             ┆ asr_result_llm                            │
    # │ ---                            ┆ ---                                        ┆ ---                                       │
    # │ Utf8                           ┆ Utf8                                       ┆ Utf8                                      │
    # ╞════════════════════════════════╪════════════════════════════════════════════╪═══════════════════════════════════════════╡
    # │ https://las-cn-beijing-publ…   ┆ I kept myself safe, got the gold, I, old Sun, do not care about fame, only wish to return ┆ I kept myself safe, got the gold, I, old Sun, do not care about fame, only wish to         │
    # │                                ┆ to Flower Fruit Mountain and spend the rest of my life there…                             ┆ return to Flower Fruit Mountain and live out my days there…                          │
    # ╰────────────────────────────────┴────────────────────────────────────────────┴───────────────────────────────────────────╯