Description #

Video motion score calculation

Key features #

• Multi-algorithm support: Supports multiple optical flow calculation algorithms to suit different scenarios and performance requirements:
• Farneback: Classic dense optical flow algorithm, balancing accuracy and speed.
• TV-L1: Optical flow algorithm based on total variation, robust to illumination changes.
• DIS (Dense Inverse Search): Fast optical flow algorithm, supports multiple presets (ULTRAFAST, FAST, MEDIUM, accurate).
• MEMFOF: Deep learning-based optical flow estimation algorithm, highest accuracy.
• Multi-dimensional metrics: Provides a rich variety of motion scoring metrics:
• Basic metrics: Mean, median, 95th percentile.
• Adaptive metrics: Dynamic scores normalized by frame size to eliminate resolution effects.
• Density metrics: Density score reflecting the proportion of motion regions.
• Intelligent sampling: Supports proportional sampling and intelligent downsampling to balance computational overhead.
• High-performance computing: Supports CUDA GPU acceleration and multi-process parallel processing.

Applicable scenarios #

• Video quality assessment
• Video classification and labeling
• Dynamic cover selection
• Video content analysis

Format support: #

• MP4 (.mp4)
• AVI (.avi)
• MOV (.mov)
• MKV (.mkv)
• Other common video formats

Operator parameters #

Input #

Output #

Structured array (Struct) containing the results of motion score calculation. Each element includes the following fields:

• standardized_score (float): Standardized score
• motion_pattern (str): Motion pattern classification
• mean_score (float): Mean value of original optical flow magnitude
• median_score (float): Median value of optical flow magnitude
• dynamic_mean_score (float): Size-adaptive mean (eliminates resolution effects)
• high_percentile_score (float): Original 95th percentile magnitude
• dynamic_high_percentile_score (float): Size-adaptive 95th percentile magnitude
• density_score (float): Motion density score
• video_resolution (list[int]): Video resolution [width, height]
• total_frames (int): Total number of frames in the video
• sample_frames_count (int): Actual number of sampled frames used for calculation
• used_algorithm (str): Optical flow algorithm used
• used_metric (str): Motion score metric used
• status (str): Processing status (success/error/empty)
• total_process_time (float): Total processing time (seconds)

Parameters #

If a parameter does not have a default value, it is required.

Examples #

The following code demonstrates how to use Daft (for distributed environments) to run the operator for video motion score calculation.

from __future__ import annotations

import os

import daft
from daft import col
from daft.las.functions.udf import las_udf
from daft.las.functions.video import VideoMotionScore

if __name__ == "__main__":
        if os.getenv("DAFT_RUNNER", "native") == "ray":
        import logging

        import ray

        def configure_logging():
            logging.basicConfig(
                level=logging.INFO,
                format="%(asctime)s - %(name)s - %(levelname)s - %(message)s",
                datefmt="%Y-%m-%d %H:%M:%S.%s".format(),
            )
            logging.getLogger("tracing.span").setLevel(logging.WARNING)
            logging.getLogger("daft_io.stats").setLevel(logging.WARNING)
            logging.getLogger("DaftStatisticsManager").setLevel(logging.WARNING)
            logging.getLogger("DaftFlotillaScheduler").setLevel(logging.WARNING)
            logging.getLogger("DaftFlotillaDispatcher").setLevel(logging.WARNING)

        ray.init(dashboard_host="0.0.0.0", runtime_env={"worker_process_setup_hook": configure_logging})
        daft.set_runner_ray()

    daft.set_execution_config(actor_udf_ready_timeout=600)
    daft.set_execution_config(min_cpu_per_task=0)

    tos_dir_url = os.getenv("TOS_DIR_URL", "las-cn-beijing-public-online.tos-cn-beijing.volces.com")
    samples = {
        "video_path": [f"https://{tos_dir_url}/public/shared_video_dataset/singer.mp4"],
    }

    ds = daft.from_pydict(samples)

    constructor_kwargs = {
        "optical_flow_algorithm": "dis-fast",
        "sample_ratio": 0.03,
        "use_cuda": True,
        "num_workers": 1,
        "batch_size": 10,
        "downsample_ratio": 1.0,
        "model_path": "/opt/las/models",
        "mag_threshold": 1.5,
        "flow_threshold": 6,
    }

    ds = ds.with_column(
        "motion_results",
        las_udf(VideoMotionScore, construct_args=constructor_kwargs)(col("video_path")),
    )

    ds = ds.select(
        col("video_path").alias("视频路径"),
        col("motion_results")["used_algorithm"].alias("使用算法"),
        col("motion_results")["standardized_score"].alias("标准化运动分"),
        col("motion_results")["mean_score"].alias("平均运动幅值"),
        col("motion_results")["median_score"].alias("运动幅值中位数"),
        col("motion_results")["dynamic_mean_score"].alias("尺寸自适应均值"),
        col("motion_results")["high_percentile_score"].alias("原始95分位"),
        col("motion_results")["dynamic_high_percentile_score"].alias("尺寸自适应95分位"),
        col("motion_results")["motion_pattern"].alias("运动等级"),
        col("motion_results")["video_resolution"].alias("视频分辨率"),
        col("motion_results")["total_frames"].alias("总帧数"),
        col("motion_results")["status"].alias("处理状态"),
        col("motion_results")["avg_frame_process_time"].alias("单帧平均处理时间(s)"),
        col("motion_results")["sample_frames_count"].alias("采样帧数"),
    )

    ds.show()
# ╭────────────────────────────────┬──────────┬──────────────┬──────────────┬────────────────┬────────────┬─────────────┬────────┬──────────┬─────────────────────┬──────────╮
# │ Video path                    ┆ Algorithm used ┆ Normalized motion score ┆ Average motion magnitude ┆ Median motion magnitude ┆      …     ┆ Video resolution ┆ Total frames ┆ Processing status ┆ Average processing time per frame (s) ┆ Sampled frames │
# │ ---                            ┆ ---      ┆ ---          ┆ ---          ┆ ---            ┆            ┆ ---         ┆ ---    ┆ ---      ┆ ---                 ┆ ---      │
# │ String                         ┆ String   ┆ Float32      ┆ Float32      ┆ Float32        ┆ (4 hidden) ┆ List[Int32] ┆ Int64  ┆ String   ┆ Float64             ┆ Int64    │
# ╞════════════════════════════════╪══════════╪══════════════╪══════════════╪════════════════╪════════════╪═════════════╪════════╪══════════╪═════════════════════╪══════════╡
# │ https://las-ai-qa-online.tos-… ┆ dis-fast ┆ 0.57431227   ┆ 36.396885    ┆ 35.626892      ┆ …          ┆ [640, 360]  ┆ 665    ┆ success  ┆ 0.015               ┆ 21       │
# ╰────────────────────────────────┴──────────┴──────────────┴──────────────┴────────────────┴────────────┴─────────────┴────────┴──────────┴─────────────────────┴──────────╯