Description #

CommonCrawl web content extractor supporting multiple parsing strategies

Key features #

• Multi-extractor support: trafilatura/justext/goose3
• Supports local and remote WARC files
• Supports binary data and base64 encoding
• Batch processing of WARC files to extract web page main content
• Intelligent content extraction, filtering advertisements and navigation elements

Extractor comparison #

• justext: Fastest speed, minimal resource usage, strict filtering
• trafilatura: Moderate speed, built-in rules plus lightweight NLP model, high extraction quality
• goose3: Slower speed, multi-layer logic extraction enabled, mainly for English news

Operator parameters #

Input #

Output #

List of extraction results, each element contains the following fields:

• url: Web page URL
• content: Extracted main content
• warc_file: Source WARC file identifier
• extractor: Name of the extractor used

Parameters #

If a parameter does not have a default value, it is required

Examples #

The following code demonstrates how to use daft to run the operator and extract web page main content from CommonCrawl WARC files, supporting multiple input formats such as file path, binary data, and base64 encoding.

from __future__ import annotations

import os

import daft
from daft import col
from daft.las.functions.text.commoncrawl_content_extractor import CommonCrawlContentExtractor
from daft.las.functions.udf import las_udf

if __name__ == "__main__":

    if os.getenv("DAFT_RUNNER", "native") == "ray":
        import logging

        import ray

        def configure_logging():
            logging.basicConfig(
                level=logging.INFO,
                format="%(asctime)s - %(name)s - %(levelname)s - %(message)s",
                datefmt="%Y-%m-%d %H:%M:%S.%s".format(),
            )
            logging.getLogger("tracing.span").setLevel(logging.WARNING)
            logging.getLogger("daft_io.stats").setLevel(logging.WARNING)
            logging.getLogger("DaftStatisticsManager").setLevel(logging.WARNING)
            logging.getLogger("DaftFlotillaScheduler").setLevel(logging.WARNING)
            logging.getLogger("DaftFlotillaDispatcher").setLevel(logging.WARNING)

        ray.init(dashboard_host="0.0.0.0", runtime_env={"worker_process_setup_hook": configure_logging})
        daft.set_runner_ray()
    daft.set_execution_config(actor_udf_ready_timeout=600)
    daft.set_execution_config(min_cpu_per_task=0)

    tos_dir_url = os.getenv("TOS_DIR_URL", "las-cn-beijing-public-online.tos-cn-beijing.volces.com")
    samples = {
        "warc_data": [
            f"https://{tos_dir_url}/public/shared_file_dataset/sample.warc.gz"
        ]
    }

    extractor_type = "trafilatura"
    max_records = 5
    df = daft.from_pydict(samples)
    df = df.with_column(
        "extracted_content",
        las_udf(
            CommonCrawlContentExtractor,
            construct_args={
                "warc_src_type": "warc_url",
                "extractor_type": extractor_type,
                "max_records": max_records,
            },
            num_gpus=0,
            batch_size=1,
            concurrency=1,
        )(col("warc_data")),
    )
    df.show()

    # ╭──────────────────────────────────────────────┬─────────────────────────────────────────────────────────────╮
    # │ warc_files                                   ┆ extracted_content                                           │
    # │ ---                                          ┆ ---                                                         │
    # │ Utf8                                         ┆ List[Struct[url: Utf8, content: Utf8, warc_file: Utf8,      │
    # │                                              ┆ extractor: Utf8]]                                           │
    # ╞══════════════════════════════════════════════╪═════════════════════════════════════════════════════════════╡
    # │ https://las-public-data-qa.tos…              ┆ [{url: http://00852imports.com/detail/5389084.html,         │
    # │                                              ┆ content: With the development of the Internet, people's requirements for network speed are increasing…,         │
    # │                                              ┆ warc_file: sample.warc.gz, extractor: trafilatura},         │
    # │                                              ┆ {url: http://02y3tcpv.gd9.cc/?penglaibexdkcl224396.html,    │
    # │                                              ┆ content: View more related content

Unfollow in today's digital era…,            │
    # │                                              ┆ warc_file: sample.warc.gz, extractor: trafilatura}]         │
    # ╰──────────────────────────────────────────────┴─────────────────────────────────────────────────────────────╯