Data-juicer #

Introduction #

Data-Juicer is an all-in-one data processing system designed specifically for foundation models (typically LLMs), capable of preprocessing and managing both text and multimodal data (such as images, audio, video, and more). Data-Juicer supports rapid generation of “data recipes” (Data Recipes) from multi-source heterogeneous data for model training, and comes with built-in visualization and automatic evaluation features to enable data and model collaboration.

Core features #

1. Library of operators:
   • Data-Juicer provides more than 50 built-in operators covering common operations such as data cleaning, filtering, transformation, and augmentation. Users can quickly build complex pipelines through simple combinations, offering flexible support for different data recipes.
   • In version 2.0, the number of operators has been expanded to over one hundred, with new dedicated operators for processing multimodal data (images, audio, video). It is also compatible with mainstream dataset interfaces such as Hugging Face and ModelScope, making experimentation and engineering deployment more convenient.
2. Pipeline and data recipe management:
   • Through the "Probe-Analyze-Refine" workflow, Data-Juicer can quickly analyze the quality of raw data, automatically filter out abnormal and noisy samples, and adjust data recipes in an interactive interface to create high-quality datasets suitable for pre-training or fine-tuning.
   • The built-in visualization module provides multi-dimensional statistics such as data distribution, missing rate, and annotation consistency, helping researchers promptly identify issues and optimize recipes.
3. Distributed scheduling and high performance:
   • Data-Juicer is deeply integrated with the Ray distributed computing framework, enabling efficient task scheduling in cloud clusters. In experiments, in an environment with 6,400-core CPUs (50 Ray nodes), Data-Juicer 2.0 can process 70 billion data samples in 2.1 hours; in an environment with 1,280-core CPUs (8 nodes), it can complete deduplication of 5TB of data in 2.8 hours.
   • In addition, there are dedicated optimizations for large-scale datasets, including a MinHash-LSH-based deduplication operator and streaming I/O based on Apache® Arrow, giving Data-Juicer excellent scalability and throughput in cloud computing environments.

     Apache Arrow, the names of other Apache projects, and the ASF logo are either registered trademarks or trademarks of the Apache Software Foundation in the United States and/or other countries.

System architecture and workflow #

1. Flexible and modular execution engine:
   • Data-Juicer abstracts operators and datasets as composable modules, and the underlying execution layer can seamlessly switch between standalone mode and Ray distributed mode. The core functions of operators are engine-agnostic and are scheduled and parallelized by dedicated Executors (such as RayExecutor), enabling code reuse and easy extensibility.
   • For large-scale operators such as deduplication, Data-Juicer automatically pre-splits data subsets to balance the number of files and compute nodes, ensuring optimal job scheduling and I/O efficiency.
2. Multi-interface support:
   • In addition to the traditional Python SDK, Data-Juicer 2.0 adds RESTful API and conversational command support, allowing users to quickly submit and monitor tasks via the web console or command line. With the built-in JupyterLab Playground, users can experiment and tune directly in the browser without local environment configuration.

Librosa #

Introduction #

Librosa is an open-source Python library designed for music and audio analysis, providing essential modules and tools required to build music information retrieval (MIR) systems. It was originally developed by the LabROSA research team at Columbia University, led by Brian McFee and others, aiming to provide users with a unified and efficient solution for audio processing and feature extraction in the Python environment.

Core features #

1. Audio loading and preprocessing
   • Provides the librosa.load interface, allowing audio files on disk to be read into NumPy arrays with a single line of code, and supports resampling to a specified sampling rate (default 22050 Hz).
   • Internally depends on soundfile and audioread, is compatible with multiple platforms (Windows, macOS, Linux) and various formats, and simplifies the cumbersome process of reading audio files.
2. Time–frequency transformation and analysis
   • Supports multiple time–frequency representations, including short-time Fourier transform, Mel spectrogram, constant-Q transform (CQT), and more.
   • Users can flexibly generate different spectrograms using functions such as librosa.stft, librosa.feature.melspectrogram, and librosa.cqt, providing support for subsequent feature extraction and model input.
3. Feature extraction
   • Provides a series of audio feature calculation methods, such as MFCC, spectral contrast, spectral centroid, Tonnetz, and more.
   • These features are widely used in tasks such as music classification, music recommendation, and emotion recognition, and can be quickly obtained through interfaces like librosa.feature.mfcc, librosa.feature.spectral_contrast, and librosa.feature.tonnetz.
4. Audio beat and rhythm analysis
   • Functions such as librosa.beat.beat_track can estimate the tempo of music and detect beat events, which are critical for music synchronization, rhythm analysis, automatic mixing, and similar scenarios.
   • For percussion signals or tracks with prominent beats, Librosa provides mature and robust algorithms that can accurately extract rhythm information even from noisy audio (librosa.org, dawenl.github.io).
5. Time-domain audio effects processing
   • The librosa.effects submodule provides common time-domain effects such as pitch shifting, time stretching, and time reversal, making it suitable for data augmentation, sound effect production, and music generation.
   • For example, librosa.effects.pitch_shift can apply pitch shifting to speech or music, supporting seamless transitions between different pitches and facilitating music creation and data augmentation.
6. Visualization and display
   • The librosa.display submodule, based on Matplotlib, provides a series of plotting functions that can be used to display waveforms, spectrograms, beat annotations, and more.
   • For example, librosa.display.specshow can directly render Mel spectrograms or CQT plots onto the canvas, visually reflecting the time-frequency distribution of audio signals through color maps to assist users in visual analysis.

System architecture and module division #

Librosa's internal structure is highly modular, with each submodule encapsulating specific functionality and maintaining low coupling, allowing users to flexibly combine different modules to complete customized analysis and processing workflows. Librosa is highly compatible with underlying libraries such as NumPy, SciPy, Matplotlib, SoundFile, and Audioread, ensuring stable performance in most mainstream environments. The main submodules include:

• librosa.core: Key features, covering audio loading, basic time-frequency transformations, common utility functions, and more. Most features can be accessed directly through the top-level namespace librosa.*.
• librosa.feature: Feature extraction, including MFCC, spectral contrast, pitch centroid, harmonic-percussive source separation (HPSS), and more.
• librosa.beat: Beat and rhythm analysis, responsible for estimating tempo and detecting beat frames.
• librosa.effects: Time-domain audio effects processing, such as pitch shifting, time stretching, and time compression.
• librosa.decompose: Spectral decomposition based on matrix factorization (such as NMF and PCA), supporting separation and reconstruction of audio signals.
• librosa.display: Visualization tools for plotting waveforms, spectrograms, spectral coefficients, and more.
• librosa.util: General utility functions and low-level implementation details, including framing, overlapping windows, common mathematical operations, and more.

HuggingFace #

Introduction #

Hugging Face is an open platform and community for machine learning and data science, dedicated to helping users build, deploy, and train machine learning models.

Core features #

1. Model and dataset hosting (Model & Dataset Hub)
   • The Hugging Face Hub allows users to upload, version, and share various pretrained models (covering NLP, computer vision, audio, and other multimodal tasks) as well as datasets.
2. Transformers library
   • Transformers, maintained by the Hugging Face team, is an open-source Python package that implements pretraining and fine-tuning logic for many mainstream Transformer architectures (such as BERT, GPT, T5, RoBERTa, and more), compatible with PyTorch, TensorFlow, and JAX backends. Developers can load pretrained models, perform inference, or fine-tune on their own data with just a few lines of code, and supports a variety of downstream tasks (such as text classification, question answering, summarization, translation, and more).
3. Accelerate, Datasets, and Tokenizers
   • Accelerate: Provides a simple interface to enable models to support multi-GPU, distributed training, and mixed precision (fp16) acceleration features with a single command, without writing complex distributed code manually.
   • Datasets: A high-performance dataset loading and processing library that supports interfaces for hundreds of common datasets, and can automatically perform memory mapping and parallel preprocessing, making large-scale dataset training more efficient.
   • Tokenizers: A fast tokenization library written in Rust, capable of executing BPE, WordPiece, SentencePiece, and other tokenization algorithms at millisecond speed, with Python bindings for seamless integration with the Transformers library.
4. Spaces and Gradio
   • Spaces: Allows users to easily deploy interactive web applications based on Gradio or Streamlit on the Hugging Face Hub, without additional server configuration. Developers can demonstrate model results directly in the browser, and community members can experience model features online without installing dependencies.
   • Gradio: An open-source Python library tightly integrated with Hugging Face, designed for rapidly building interactive demos of machine learning models. Users only need to write a small number of callback functions to generate shareable links, making it easy to debug and demonstrate to clients, thoroughly streamlining the model deployment and user feedback process.

System architecture and module breakdown #

The Hugging Face architecture follows a highly modular design, with each subcomponent operating independently yet collaborating closely, forming a complete ecosystem from low-level data processing to high-level model hosting and inference:

• Hub service layer: Based on Git storage and version control, supporting the hosting of models, datasets, and Spaces applications. This layer provides a RESTful API and web interface, allowing users to upload, download, search, and manage versions via the command-line tool huggingface-cli, the Python SDK huggingface_hub, or an intuitive web interface.
• Transformers library: Located at the model usage and development layer, it encapsulates model definitions, pretrained weight downloads, and inference logic, and is compatible with other deep learning frameworks (PyTorch, TensorFlow, JAX), enabling users to call Transformer models just like regular Python functions.
• Accelerate, Tokenizers, Datasets libraries: Positioned at the performance optimization layer, where Accelerate handles training acceleration and distributed deployment, Tokenizers provides efficient tokenization and encoding, and Datasets serves as the data input layer, responsible for efficient dataset loading, preprocessing, and memory mapping to ensure throughput and stability during large-scale training.

OpenCV #

Introduction #

OpenCV (Open Source Computer Vision Library) is an open-source computer vision and machine learning software library, originally launched by Intel in 2000 and subsequently maintained by Willow Garage and Itseez (later acquired by Intel). It is cross-platform, supporting Windows, Linux, macOS, as well as mobile systems such as Android and iOS, and is freely available under the Apache 2.0 license. OpenCV includes more than 2,500 optimized algorithms, implementing functions such as image processing, video analysis, feature detection and description, object recognition, and machine learning. Since 2011, it has introduced GPU acceleration, making real-time vision applications possible.

Core features #

1. Extensive algorithms and modules:
   • The OpenCV core module provides fundamental data structures and matrix operations; the image (imgproc) module covers filtering, geometric transformations, color space conversions, and more; the video module supports real-time video capture and background modeling; the object detection (objdetect) module includes commonly used detectors such as Haar and HOG; the feature detection and description (features2d) module implements algorithms like SIFT, SURF, and ORB; the camera calibration and 3D reconstruction (calib3d) module provides essential matrix estimation, stereo matching, PnP solving, and more; the machine learning (ml) module comes with a variety of built-in classification and regression algorithms; the deep neural network (dnn) module can load models exported from frameworks such as TensorFlow, Caffe, and PyTorch, and run them efficiently on both CPU and GPU.
   • In addition to the core library, OpenCV also releases the Contrib add-on modules (opencv_contrib), which provide community-contributed extensions such as face detection, background separation, image segmentation, text recognition, and more. Users can compile and integrate these modules as needed.
2. Multi-language interfaces and cross-platform support:
   • The library itself is written in C++ and provides bindings for Python, Java, MATLAB/Octave, and multiple other languages. The Python interface (cv2) is especially popular, simplifying algorithm invocation and integrating seamlessly with the NumPy and SciPy ecosystems; the Java interface supports developing mobile vision applications on the Android platform; OpenCV.js provides JavaScript bindings for the web platform.
   • OpenCV can automatically detect and utilize hardware acceleration libraries such as Intel IPP, CUDA, and OpenCL to improve runtime efficiency across different hardware environments. It is also compatible with various processor architectures including x86, ARM, and MIPS, providing a unified solution for desktop, embedded, and cloud scenarios.

System architecture and module division #

OpenCV adopts a highly modular design, with each submodule encapsulating corresponding algorithms and data structures. The main modules include:

• Core: Fundamental data types (Mat), matrix operations, and basic algorithms (such as random number generation and file I/O).
• Imgproc: Includes image filtering (mean, Gaussian, median), edge detection (Canny, Sobel), morphological operations (erosion, dilation), geometric transformations (scaling, rotation), histogram computation, and equalization, among others.
• Video: Video capture, codec interfaces (with FFmpeg support), background modeling (MOG, KNN), and optical flow tracking (Lucas–Kanade, Farneback).
• Features2D & Objdetect: 2D feature detection (FAST, ORB, BRISK), feature matching (FLANN, BFMatcher), and object detection (Haar cascade, HOG+SVM).
• Calib3d: Camera geometric calibration (chessboard intrinsic parameter estimation), stereo matching (SGBM), PnP solving, fundamental matrix/essential matrix estimation.
• ML & DNN: Integrates OpenCV's built-in machine learning algorithms (decision trees, SVM, random forests, KNN, AdaBoost, EM, neural networks, and more), and the DNN module can load and infer models exported from frameworks such as Caffe, TensorFlow, Darknet, and PyTorch, supporting CPU, CUDA, and OpenCL acceleration.
• Highgui: Cross-platform window display, graphical interface (drawing lines, text, buttons), mouse/keyboard event handling, used for debugging and visualization.

The modules are loosely coupled, allowing users to flexibly select and compile only the required modules to reduce size or improve performance. Clear documentation, examples, and CMake build system instructions enable developers to quickly integrate and deploy across different platforms and configurations.

DataSketch #

Introduction #

Apache® DataSketches is an open-source, high-performance randomized streaming algorithm library used to perform approximate queries on single-pass streaming data. Sketches are small, stateful data structures that significantly accelerate compute-intensive tasks such as cardinality statistics, frequency estimation, and quantile calculation when processing massive data streams, while maintaining extremely low memory overhead and predictable error guarantees. DataSketches was originally developed by Yahoo!'s research team. After several years of iteration, it entered Apache incubation in 2019 and became an Apache top-level project (Top-Level Project) in 2020, aiming to provide mergeable, scalable, production-grade approximate computation capabilities for batch and real-time streaming systems. Apache DataSketches, the names of other Apache projects, and the ASF logo are either registered trademarks or trademarks of the Apache Software Foundation in the United States and/or other countries.

> Apache DataSketches, the names of other Apache projects, and the ASF logo are either registered trademarks or trademarks of the Apache Software Foundation in the United States and/or other countries.

Sketch algorithm library #

1. DataSketches provides a variety of randomized streaming algorithms, including cardinality estimation (Theta Sketch), frequency estimation (Frequent Items Sketch), quantile calculation (KLL Sketch), histogram and binning (other relevant sketches), all of which have rigorous mathematical proofs regarding approximate accuracy and resource usage.
   • All Sketches support merge operations, enabling distributed systems to independently compute at each node and obtain global approximate results through simple combination. All Sketches support the merge operation, allowing distributed systems to perform independent computation at each node and then obtain a global approximate result through simple combination.
   • These algorithms can achieve optimal spatiotemporal performance at different error levels (such as relative error of 1% or 5%) by fine-tuning relevant configurations (such as Sketch size and compression parameters), thereby meeting a wide range of requirements from small-scale testing to real-time big data production environments.
2. Multi-language and cross-platform support
   • The core of DataSketches is implemented in Java and C++, and the official team also provides a Python adapter (datasketches‐python), a C++ version (datasketches‐cpp), and JNI bindings, enabling users to seamlessly integrate the same Sketch data structures in Java, C++, Python, and more environments. All language implementations guarantee binary compatibility, allowing Sketch results to be exchanged and merged across heterogeneous systems.
   • At the underlying level, DataSketches can leverage hardware acceleration (such as parallel stream processors and SIMD instructions) to optimize critical paths, together with Java's G1 GC or C++ memory allocators, achieving low latency and high throughput suitable for both batch and stream processing scenarios.
3. Production-grade adapters and ecosystem integration
   • DataSketches offers a rich set of adapters that can be directly embedded into mainstream big data and OLAP systems such as Apache Hive, Apache Pig, PostgreSQL, Apache Druid, BigQuery, and more, helping users invoke approximate computation functions (UDAF/UDF) at the SQL query layer for rapid approximate analysis on large-scale datasets.
   • For example, Google BigQuery has built-in DataSketches UDFs, enabling users to directly compute metrics such as approximate cardinality (Approximate Count Distinct), quantiles (Approximate Quantiles), frequency (Approximate Heavy Hitters), and many other metrics in SQL statements without manually writing complex custom functions.

System architecture and module breakdown #

1. Core module (Core Module)
   • Java core (datasketches‐java): Contains the main Sketch algorithm implementations and utility functions, designed for high availability and low latency, suitable for any big data platform requiring approximate queries. This module exposes a unified Sketch interface externally (such as UpdateSketch, CompactSketch, Union, Intersection, and more), supporting streaming updates and merge operations.
   • C++ core (datasketches‐cpp): Functionally equivalent to the Java version, targeting scenarios sensitive to latency or with strict dependency management requirements, and can be compiled directly as a static or dynamic library for seamless project integration.
   • Python binding (datasketches‐python): Exposes the Java/C++ implementation to Python users via Cython or JNI, with interface design close to native Python style, making it easy to use in environments such as PySpark, Pandas, Jupyter Notebook, and more.
2. Adapters and extension modules
   • Hive and Pig adapters: Encapsulate DataSketches algorithms as Hive UDAF and Pig UDF, enabling developers to invoke approximate computations as functions within HiveQL or Pig Latin scripts.
   • PostgreSQL extension (PGXN): DataSketches provides a native PostgreSQL extension package, allowing the use of approximate statistical UDTF/UDFA in SQL queries, such as theta_count(distinct column) and quantiles_nd sketch.
   • Druid and Gaffer integration: Apache Druid natively integrates DataSketches in its real-time aggregation layer to achieve near real-time approximate aggregation; the Gaffer graph database also leverages DataSketches for large-scale graph attribute analysis.
3. Algorithm library and mathematical guarantees
   • Mergeable Summaries: All sketches can be serialized and merged, ensuring correctness and consistency of cumulative statistics on sharded data streams in distributed environments.
   • Error analysis: Each DataSketches algorithm is accompanied by mathematical proofs of measurement error and confidence intervals. For example, Theta Sketch provides error ≤ ε and probability ≥ 1 – δ for cardinality estimation; KLL Sketch achieves optimal balance between relative error and memory usage for quantile queries.

TextDedup #

Introduction #

Text-Dedup is an open-source Python script collection initiated and maintained by Chenghao Mou, focusing on deduplication of large-scale text data. This project does not aim to create a general-purpose, automated deduplication library. Instead, it adopts a "single algorithm per file" approach, aggregating various classic and modern text deduplication methods (including exact and approximate deduplication) in one repository, making it convenient for users to select, modify, or further develop as needed.
TextDedup supports a wide range of deduplication strategies, from simple hash comparison to advanced locality-sensitive hashing (LSH), SimHash, MinHash, and suffix arrays, meeting requirements from small-scale experiments to terabyte-scale data processing.

Core features #

1. Multiple deduplication algorithm scripts
   • MinHash + MinHashLSH: Supports MinHash-based approximate deduplication, including both standalone and Spark implementations (suitable for terabyte-scale datasets).
   • SimHash: Provides both 64-bit and 128-bit SimHash implementations for rapid detection of near-duplicates in text streams.
   • SuffixArray Substring: An exact substring deduplication method suitable for scenarios requiring "identical match" determination.
   • Bloom Filter: Uses Bloom filters to filter out obvious duplicates, saving memory and computational resources.
   • Exact Hash: Performs one-to-one exact deduplication of documents (or line-level text) based on hash values such as MD5 or SHA.
   • RETSim/UniSim (Embedding Based): Attempts to use sentence vectors or text embeddings (for example, Sentence-Transformer) to determine similarity, suitable for documents that are similar in content but not identicalgithub.com pypi.org.
2. Plug and play single-file design
   • Each algorithm in the repository is provided as an independent Python script, such as suffix_array.py, simhash.py, and minhash.py. Users can invoke them directly from the command line or use them as templates to easily trim or extend the code.
   • Each script encapsulates the necessary command-line arguments, such as input path, output path, hash function type, N-gram size, threshold, and more, making it easy to quickly test different strategies in production or experimental environments.

System architecture and module division #

1. Repository organization
   • Top-level script directory: Each algorithm corresponds to a folder or standalone script, such as minhash_lsh/, simhash.py, suffix_array/, bloom_filter.py, and exact_hash.py.
   • Configuration and utilities: Some common utilities (such as text preprocessing, N-gram generation, and cache management) are reused by different scripts, but the overall approach maintains the 'single-file runnable' philosophy, making it easy for users to copy into any project.
2. Algorithm implementation
   • MinHash + LSH: Follows a two-step process—first, generates a MinHash signature for each text, then uses LSH (Locality Sensitive Hashing) to bucket similar texts, and finally performs more refined comparisons within each bucket. The project includes both a standalone Python implementation and a large-scale parallel MinHashLSH based on Spark.
   • SimHash: Uses SimHash to vectorize text, enabling fast calculation of approximate Hamming distances across massive document collections. The script provides both 64-bit and 128-bit hash sizes, and supports specifying the number of blocks (num_blocks) and the Hamming distance threshold.
   • SuffixArray Substring: Concatenates all text and constructs a suffix array, using the suffix array to quickly find the longest common substring (Longest Common Substring) between documents, identifying and deduplicating identical or highly repetitive segments.
   • Bloom Filter: Each text is first hashed and inserted into a Bloom filter to filter out most duplicates, then the filtered candidates are compared exactly, balancing speed and accuracy.
   • Exact Hash: For each text (by line or by paragraph), calculate the hash value using SHA-1, MD5, and similar algorithms, and directly determine duplication by hash collision. This method is best suited for scenarios where deduplication is required only when the text content is exactly the same.
   • Embedding Based (RETSim/UniSim): By leveraging pre-trained sentence embedding models such as Sentence-Transformer, map the text into a low-dimensional vector space, and perform approximate deduplication based on vector similarity (such as cosine similarity). This approach is suitable for cases where the semantic content is similar but the surface form differs (currently still under development).
3. Multilingual adaptation and integration
   • Although the core implementation is based on pure Python, TextDedup can also be integrated with mainstream stream processing frameworks (Spark, Dask) or distributed storage systems (HDFS, S3). The MinHashLSH implementation in Spark can run directly in cluster environments such as YARN or Kubernetes; for small-scale environments, experiments can also be conducted on a single machine using Pandas or NumPy.
   • For scenarios requiring higher performance or memory control, users are advised to refer to the ideas in the script and rewrite the key parts using Cython, C++, or PySpark to further enhance scalability.