Description #

Audio classification module – Multilingual audio classification solution based on the BEATs model
AudioBeatsClassifier is an audio classification operator based on the BEATs model. It is used to identify the main sound events in audio and returns the top K classification labels with the highest probability.

Key features #

• Support for multiple audio types: Automatically processes various types of audio and identifies 527 sound categories defined by Google AudioSet, such as "music", "speech", "alarm sound", or "animal sound", and more.
• Support for multiple input channels: Seamlessly processes audio data from local file paths, HTTP URLs, TOS/S3 object storage, or raw byte streams.
• Automated preprocessing: Built-in audio decoding and preprocessing capabilities automatically resample input audio to 16 kHz mono format, simplifying the invocation process.

Supported models #

Audio classification is implemented based on the BEATs model. You can download the corresponding model files from the microsoft/unilm repository.

Operator parameters #

Input #

Output #

The operator organizes classification results using a JSON array (as shown below, each input audio corresponds to a JSON array object). Each element in the array represents a recognized sound classification and contains two fields:

• label: The unique identifier for the classification label, following the Google AudioSet labeling system (for example, "/m/04rlf" represents music).
• probability: The confidence score for the classification, ranging from 0.0 to 1.0.

[
    {"label": "/m/04rlf", "probability": 0.85},
    {"label": "/m/09x0r", "probability": 0.39},
    {"label": "/m/03qc9zr", "probability": 0.33},
    {"label": "/m/07sr1lc", "probability": 0.27},
    {"label": "/m/07s2xch", "probability": 0.15}
]

Parameters #

If a parameter does not have a default value, it is required.

Examples #

The following code demonstrates how to use Daft to run the operator for audio classification:

from __future__ import annotations

import logging
import os

import daft
from daft import col
from daft.las.functions.audio import AudioBeatsClassifier
from daft.las.functions.udf import las_udf

if __name__ == "__main__":
    if os.getenv("DAFT_RUNNER", "ray") == "ray":

        def configure_logging():
            logging.basicConfig(
                level=logging.INFO,
                format="%(asctime)s - %(name)s - %(levelname)s - %(message)s",
                datefmt="%Y-%m-%d %H:%M:%S.%s".format(),
            )
            logging.getLogger("tracing.span").setLevel(logging.WARNING)
            logging.getLogger("daft_io.stats").setLevel(logging.WARNING)
            logging.getLogger("DaftStatisticsManager").setLevel(logging.WARNING)
            logging.getLogger("DaftFlotillaScheduler").setLevel(logging.WARNING)
            logging.getLogger("DaftFlotillaDispatcher").setLevel(logging.WARNING)

        import ray

        ray.init(dashboard_host="0.0.0.0", runtime_env={"worker_process_setup_hook": configure_logging})
        daft.set_runner_ray()

    daft.set_execution_config(actor_udf_ready_timeout=600)
    daft.set_execution_config(min_cpu_per_task=0)

    tos_dir_url = os.getenv("TOS_DIR_URL", "las-cn-beijing-public-online.tos-cn-beijing.volces.com")
    samples = {"audio_path": [f"https://{tos_dir_url}/public/shared_audio_dataset/参观八达岭长城。.wav"]}

    df = daft.from_pydict(samples)
    df = df.with_column(
        "classify_result",
        las_udf(
            AudioBeatsClassifier,
            construct_args={
                "model_path": "/opt/las/models",
                "model_name": "BEATs/BEATs_iter3_plus_AS2M_finetuned_on_AS2M_cpt2.pt",
                "top_k": 5,
                "precision": 2,
            },
            num_gpus=0.25,
            batch_size=8,
            concurrency=4,
        )(col("audio_path")),
    )
    df.show()