The core resource for LAS data processing is the queue: it provides all the CPU, GPU, and memory required for running operators and scripts. The development machine, task management, and workflow are all built on the queue, each addressing the challenges of debugging, batch execution, and complex scheduling in their respective phases. This article introduces the dependencies and collaboration methods among the four core modules of LAS data processing—queue, development machine, task management, and workflow.

The core modules of LAS data processing (development machine / task management / workflow) rely on the underlying queue and read scripts, container images, data, and operator templates from external resources.

The division of responsibilities and dependencies among the four modules is as follows:

General workflow #

The complete workflow from creating a queue to viewing results, including debugging and scheduling branches (Daft / Python / Spark), typically consists of the following six steps:

Main workflow: Create a queue → Create a development machine → Upload code to TOS → Create a task → Submit to the queue for execution → View task instances. The upper branches represent three trigger methods, and the lower dashed line indicates the parameter iteration loop.

• Main workflow: Create a queue → Debug on the development machine → Upload code to TOS → Create a task → Submit to the queue for execution → View instance logs and results.
• Trigger methods: After a task is created, it can be triggered in three ways: manual execution / periodic scheduling / workflow orchestration. If the execution result does not meet expectations, return to the "Create task" step to adjust the script, environment variables, or resource parameters for iteration.

Operation example #

Taking the offline operator "Video Key Frame Extraction" provided by the LAS Operator Plaza as an example, this section brings together all the modules mentioned above and demonstrates how to run an operator on the queue.

Prerequisites #

• A compute queue has already been created in the LAS console (it is recommended to use a dedicated queue), and permission to submit tasks has been granted. For details on the queue creation process, see Queue management.
• A TOS bucket in the same region as LAS has been enabled for storing operator scripts and input/output data.
• The "Video Key Frame Extraction" operator has been located in the LAS Operator Marketplace, and its environment variable list and entry command template have been recorded.

Step 1: Prepare the operator code and upload it to TOS #

1. Refer to the sample code for "Video Key Frame Extraction" in the Operator Marketplace, save the script as test.py, with a structure as shown below (refer to the Operator Marketplace for the actual structure):

   # test.py —— Example of an offline operator for video keyframe extraction
   import os
   import daft
   
   # 1. Read parameters specified by the operator marketplace from environment variables
   input_uri  = os.getenv("INPUT_URI")        # Input video directory, for example tos://your-bucket/videos/
   output_uri = os.getenv("OUTPUT_URI")       # Keyframe output directory
   frame_rate = float(os.getenv("FRAME_RATE", "1.0"))  # Frame extraction frequency (frames/second)
   model_path = os.getenv("MODEL_PATH", "/mnt/data/keyframe_model")
   
   # 2. Use Daft to read the video and call the keyframe extraction operator
   df = daft.read_video(input_uri)
   df = df.with_column(
       "keyframes",
       df["video"].apply_udf("keyframe_extract", fps=frame_rate, model=model_path),
   )
   
   # 3. Write the results back to TOS
   df.write_parquet(output_uri)
   

2. Upload test.py to the TOS bucket in the same region as LAS, placing it at the designated path:

   tosutil cp ./test.py tos://test/test.py
   

   This example uses tos://test/ test.py as the script address and /test/ as the mount path within the task; for actual use, replace these with your own bucket name and directory.

Step 2: Create a task and mount the TOS directory #

Go to LAS Console > Data Processing > Task Management > Create Task and fill in the basic information and environment information as follows:

After mounting is complete, /test/ test.py inside the container at task startup will directly point to the script on TOS; any subsequent script changes only require re-uploading to TOS, without recreating the task.

Step 3: Fill in the environment variables as described in the Operator Marketplace #

Open the "Video Key Frame Extraction" page in the LAS Operator Marketplace, and copy the key-value pairs listed in the "Environment Variables" section directly into the task's environment variable form. For example, in this case (refer to the Operator Marketplace for actual values):

Step 4: Submit the task to the specified queue and specify the entry command #

Enter the actual execution statement in the entry command field. Since the script is already mounted at /test/ test.py, the most straightforward way to write it is:

python /test/test.py

In the Resources section on the right side of the page, select the queue prepared in Step 1 (for example, las-gpu-queue-a), and set the number of CPUs, memory, and GPUs for the Driver/Executor as needed. After confirming everything is correct, click Execute, and the task instance will immediately enter the queue for scheduling.

Step 5: View the execution results #

Each time the task is triggered, a record will be generated on the "Task Instance" page. The execution status can be confirmed from three aspects:

• Instance status: Running / Succeeded / Failed; when failed, hover the mouse to see a brief reason.
• Log: Click "Log" to view the standard output and error stack of the Driver / Executor, which is used to locate script-level errors.
• Daft UI: For Daft-type tasks, click "Daft UI" to jump to the Daft Web interface and view detailed metrics such as DAG, operator duration, and data volume distribution.

After the task succeeds, the extraction results will be written back to TOS according to OUTPUT_URI; use tosutil ls tos://your-bucket/keyframes/ or the object storage console to verify the output files.

Step 6: Adjust parameters and resources according to business requirements #

After running the first instance successfully, it is usually necessary to make adjustments based on the scale of the business. The common directions for parameter tuning are as follows:

• Queue management: Comparison, creation, and configuration changes of dedicated queues and compute queues.
• Development machine overview: Creation and usage of the operator code debugging environment.
• Task management: Basic task information and complete parameter descriptions for Daft, Python, and Spark task types.
• Workflow: Multi-task DAG orchestration and scheduling.
• Storage mounting instructions: Detailed rules for mounting TOS, vePFS, and NAS.