Solution background #

With the rapid development and widespread adoption of large language models, numerous challenges have emerged, especially as model sizes continue to grow and computational demands increase. Model distillation technology, as an effective solution, transfers knowledge from large, complex models (teacher models) to smaller, more efficient models (student models), significantly reducing computational resource requirements while maintaining model performance. As a result, it has attracted considerable attention.
The core concept of model distillation is to transfer the knowledge of large models to smaller models, preserving the reasoning capabilities and accuracy of the large models while substantially reducing storage and inference costs. Smaller models require fewer resources and are easier to deploy across various hardware platforms, greatly expanding the breadth and depth of practical applications. In addition, for deep reasoning tasks, the required chains of thought are difficult to achieve through manual annotation. Combining model distillation with fine-tuning maximizes the use of existing large model resources, enhances the reasoning capabilities of smaller models, compensates for their shortcomings in handling complex tasks, and provides efficient and reliable reasoning services. Distillation can specifically refer to both data distillation and model distillation.

Model distillation vs data distillation #

Core requirements analysis #

• Dataset requirements
  • Difficulty in collecting QA data records: The vast scale of reasoning data presents significant challenges for collecting QA data records. During the model distillation process, a large volume of QA data records must be collected to analyze and optimize model performance and behavior. QA data records are essentially reasoning logs.
  • Diverse datasets: During model distillation, there are considerable differences in input and output parameters across different models, resulting in a complex and varied distillation process. To ensure the smooth progress of data collection and model training, fine-grained management of data version lineage is particularly important.
  • Format conversion: During the model distillation process, data format conversion is inevitable. Different models and frameworks may use different data formats. To ensure compatibility of data across different models and frameworks, format conversion is required.
  • Exploration and filtering: During data storage, exploration and filtering are key steps to ensure data quality and validity. By exploring the data, it is possible to understand the distribution, missing values, and outliers, thereby providing a basis for subsequent data processing and model training.
  • Conversion between multiple storage media: During the model distillation process, data may need to be converted between different storage media, such as from disk storage to memory storage, or from local storage to cloud storage. Different storage media have different performance and cost characteristics, so it is necessary to select the appropriate storage media based on actual requirements.
• Data processing requirements
  • Distillation of high-quality question-answer (QA) pairs: High-quality QA pairs are an indispensable part of the model distillation process. By distilling high-quality QA pairs, the performance of the student model can be effectively improved.
  • Complex data cleaning process: Data cleaning is an important step in data processing, aiming to remove noise, errors, and inconsistencies from the data to improve its quality and usability. However, in current application scenarios, the data cleaning process faces significant challenges due to high complexity. The diversity of data sources and the complex structure of the data itself mean that cleaning requires a significant investment of time and workforce costs.
  • Lack of data quality assessment tools: During the model distillation process, assessing data quality is key to ensuring model performance. However, the current lack of effective data quality assessment tools makes it difficult to promptly identify and resolve data quality issues.

Volcano Engine large model data distillation solution #

Overall functional architecture #

Data distillation process #

Data distillation flow diagram

1. Data collection: During the data collection stage, user inference service logs are efficiently collected into the LAS system to ensure data integrity and accuracy, laying a solid foundation for subsequent processes.
2. Data exploration and selection: In the data exploration and selection phase, the collected log data is thoroughly explored and meticulously filtered according to specific requirements, enabling precise identification and selection of high-quality QA pairs suitable for model training, ensuring data relevance and effectiveness.
3. Data preprocessing: In the data processing phase, a series of professional operations are performed on the filtered raw QA pairs, including data cleaning, format standardization, content enhancement, and data augmentation. The goal is to improve data quality, expand data volume, and optimize data structure, providing higher-quality and richer data resources for model training to enhance and precisely optimize model performance.
4. Dataset management: Data is systematically managed using datasets as the basis for systematic management to ensure orderly storage and efficient utilization.
5. Data distillation: By leveraging high-quality models such as DeepSeek, higher-quality data that better aligns with business needs is obtained, further enhancing the value and applicability of the data.
6. Data post-processing: Data processing operations are performed on the QA pairs generated by model distillation to form high-quality datasets, laying a solid foundation for subsequent applications.
7. Model training: Fully utilize the capabilities of MLP, ModelArk, and third-party platforms to perform further pretraining, continued training, SFT, and other training operations on the model.

Resource consumption estimation

Data distillation is a batch task aimed at continuously improving model performance based on business needs. Typically, relevant data are merged according to the data recipe to form a high-quality dataset (distilled data + related data), which is used for fine-tuning the model.

QA data collection and exploration #

QA data collection
In both online inference and batch inference scenarios, to address the need for inference log collection—especially the challenges posed by large data volumes—an effective architectural solution has been designed. This architecture is intended to provide practical solutions for relevant applications, helping to address the challenges of log collection under massive data conditions, thereby ensuring smooth business operations and effective data utilization.
QA data exploration
In big data application scenarios, after completing log collection, business users typically perform filtering and exploration based on the log data itself, and use this to construct a preprocessing dataset. This solution is designed to effectively address the challenges of log exploration in big data scenarios. Through rigorous data filtering and analysis processes, it builds a high-quality preprocessing dataset, laying a solid foundation for subsequent work and supporting the overall process of model distillation.

Data distillation (LAS) #

Models such as DeepSeek-R1 are notable for their outstanding inference capabilities. This document uses DeepSeek as an example. For other commercial and open-source large models, please observe the usage restrictions.
Through this series of data distillation processes, users can leverage more powerful general-purpose large models or domain-specific models for distillation, thereby obtaining high-quality data resources and providing strong support for solving related problems in different application scenarios.

Data processing (LAS) #

The LAS data processing operator platform enables model developers to build and reuse data operations, configure corresponding data processing workflows, and evaluate the impact of LLMs' data pipelines on training under different data recipe conditions.

Workflow

LAS provides efficient, flexible, and powerful data processing workflow services. With a carefully designed architecture, LAS can organically combine and connect built-in operators, custom operators, and third-party operators to construct data processing workflows for different application scenarios, meeting diverse data processing needs.

In the field of data processing, operators play a key role in implementing fundamental data processing logic. By carefully combining various operators, optimizing the order and execution modes of operators, data processing efficiency and effectiveness can be significantly improved. For data distillation scenarios, the platform's built-in operators are mainly divided into the following two categories:

• Data cleaning: Supports cleaning the training set by removing noisy data, duplicate data, and more, thereby improving data quality;
• Data augmentation: Uses data augmentation techniques, such as synonym replacement and sentence reorganization for text, and rotation, flipping, and scaling for images, to increase data diversity and richness, thereby enhancing the model's generalization and robustness.

Distillation data preprocessing

Data preprocessing operators

Example of a data preprocessing pipeline

Distillation data postprocessing

Data postprocessing operators
LAS provides over 100 built-in data processing operators. For details, please refer to the attachment.

Example of a data postprocessing pipeline

Dataset management (LAS) #

LAS leverages Volcano Engine Data Lake Storage and deeply integrates lake format technology to build the dataset management module.

Basic features :

• Enables convenient creation of various structured and unstructured datasets.
  • Structured data: Tabular data in databases, with clearly defined fields, records, and relationships;
  • Unstructured data: For example, text files, images, audio, video, and various other formats, as well as data without a fixed structure.

Data labeling :

• Provides labeling and management for the entire dataset, enabling granular label operations on each data item within the dataset.
• Supports batch labeling to ensure that large volumes of data can be labeled quickly and accurately.
• Offers convenient label search capabilities, allowing users to quickly locate and filter the required datasets or data items by label, supporting data mining applications.

Version management :

• Each dataset is equipped with a dedicated meta file that records detailed metadata information related to the dataset.
  • Enables effective management of dataset versions, including version creation, updates, rollbacks, and other operations.
  • Tracks the lineage of datasets to understand the origin, evolution, and relationships with other data, providing a solid foundation for data reliability assessment, troubleshooting, and data governance.

Model distillation (LAS development machine, MLP, ModelArk) #

Mount the dataset on platforms with computing resources such as LAS development machines, ModelArk, or machine learning platforms. Depending on the specific task objectives and data characteristics, the dataset can be used for model pre-training, fine-tuning, reinforcement learning, and other training schemes.

Solution advantages #

To address the complex data storage requirements during deep learning model distillation, Volcano Engine provides comprehensive and detailed strategies.

• Efficiently solve data storage challenges

From supporting multiple storage media to refined management of data version lineage, the solution ensures efficient storage and management when handling diverse datasets and large-scale inference data, providing a solid data foundation for model training.

• Comprehensive data processing capabilities

The data processing operator platform encompasses a variety of data processing operator types, ranging from document information extraction and format conversion to various deduplication operations, as well as user-defined transformation and filtering features, greatly enhancing the flexibility and efficiency of data processing. The platform can meet strict data quality requirements in different scenarios, supporting the generation of high-quality distilled data.

• Flexible dataset management system

The dataset management module provides powerful features, including flexible dataset creation methods that do not add extra storage burden; comprehensive data label management for convenient data retrieval and classification; and robust version management and lineage tracing features to ensure data traceability and standardized management.

• Improve model training efficiency and quality

High-quality data can be obtained through inference data distillation. Datasets can be easily deployed on platforms with computing resources, supporting multiple training schemes. This effectively improves the efficiency and quality of model training and accelerates model optimization and deployment.

• Lower-cost data distillation solution

With built-in operators, it builds a more technically robust, more stable, and feature-rich service for invoking open and closed source large language models, providing high-quality service solutions for various application scenarios. While ensuring effectiveness is not compromised, it reduces related costs.

Application scenarios #

Intelligent customer service
In intelligent customer service systems, large language models need to process a large volume of user inquiry data. With this solution, complex and diverse conversational data can be efficiently stored and processed. Through model distillation technology, the knowledge of large models is distilled into small, efficient models, improving the response speed and accuracy of intelligent customer service while reducing deployment costs.
Online education
After adopting this large model data distillation solution, inference logs are used to comprehensively collect and understand students' learning behavior data. The data processing operator platform is used to clean and transform the data, the dataset management module classifies and manages versions of the data, and high-quality data is obtained through inference data distillation for model training. The knowledge of large and complex models is transferred to small and efficient models, which are deployed in intelligent education systems. This improves the system's response speed and the accuracy of teaching assistance, resulting in significant improvements in students' learning outcomes.
Financial risk control
Data in the financial sector is characterized by diversity and real-time characteristics. The data processing and management capabilities in this solution can clean, transform, and analyze massive amounts of financial data. Efficient risk prediction models are trained through model distillation, helping financial institutions promptly identify potential risks and make informed decisions.
Industrial inspection
A large amount of equipment operation data is generated in industrial production environments, and traditional methods struggle to effectively process and analyze this data to achieve accurate fault prediction. Key equipment operation data is collected through inference logs, the data processing operator platform is used to explore and process the data, the dataset management module effectively manages data at different stages, and models capable of more accurate fault prediction are trained through inference data distillation and model training.

Operation demonstration #

1. In ModelArk, enable inference data backflow with one click to automatically collect data into LAS.

2. In AI Data Lake Service (LAS), further analyze and process the backflow data into a training dataset.

3. Return to ModelArk to complete model fine-tuning, optimize model performance, and realize a data flywheel where models and data drive each other.

4. At the same time, model fine-tuning can also be performed using the LAS development machine feature.