This topic explains how to create a VeArk batch inference task with TOS.

Background information

TOS

Torch Object Storage (TOS) is a distributed cloud storage service provided by Volcano Engine. TOS offers near-infinite scaling, cost efficiency, ease of use, and data security, reliability and availability. You can access TOS through RESTful APIs, software development kits (SDKs), and various tools. With TOS, you can manage and access your data over the network from any application, anytime, anywhere. For more information, see What is TOS.

VeArk

Volcano Engine Ark (VeArk) is an enterprise-grade, developer-centric platform for large language models (LLMs). It offers a wide range of features for training, inferencing, evaluating, and fine-tuning models, playing a pivotal role in the LLM ecosystem. For more information, see What is VeArk.

Batch inference is designed for non-real-time scenarios and enables the offline processing of large datasets in a single batch.

Creating VeArk batch inference tasks with data stored in TOS offers the following benefits:

Input data: You will not incur data transfer costs over the public network. VeArk is deeply integrated with TOS to deliver high performance and a seamless user experience.

Output data: You will not incur data transfer costs over the public network. You can conveniently leverage the full capabilities of TOS. For a list of supported features, see Console features overview.

Workflow

Prerequisites

Make sure that you have created a Volcano Engine account and completed real name verification.

Make sure that you have activated TOS for your account or Identity and Access Management users.

Steps

Step 1: Create a TOS bucket

Buckets are containers that hold your files or objects in TOS. You must create a bucket before uploading any objects.

Step 2: Upload inference files to TOS

You can upload your inference files to TOS using one of the following methods.

Upload via the console

Upload via tosutil

Upload via TOS Browser

Upload via SDK

Upload via API

Log in to the TOS console.

In the left navigation pane, click Buckets. In the buckets list, choose the name of the bucket that you want to upload your object to.

On the Objects page, click Upload file. For instructions on how to configure the relevant parameters, see Uploading objects.

Verify the files you want to upload, and click Upload.

Then, you are redirected to the Task progress management page. To view task progress and status, you can also click Task center in the upper-right corner of the console.

Log in to the VeArk console.

In the left navigation pane, select Inference > Batch inference.

On the Batch inference page, click Start batch inference.

On the Create batch inference page, configure the batch task parameters and then click Commit create. For more information on batch task parameters, see Batch inference.

Once the batch task is complete, you can view the results in one of two ways.

On the Batch inference page, click View in the Actions. This will take you to the TOS bucket where you can view and download the output files.

Click the batch task name to open the Overview page. In the File information section, click the link under Result file. This will also take you to the TOS bucket.

More use cases

You can also use TOS to host static websites or upload data from WeChat Mini Programs. See Use cases for more applications.

For guidance on common scenarios, see the following topics:

Common scenarios	Description
Uploading data from WeChat Mini Programs to TOS	WeChat Mini Programs, which can be used without downloading, are attracting a growing number of developers and users. One common concern for developers is how to upload files from a WeChat Mini Program to TOS.
Image processing	TOS allows you to process image files stored within it. You can initiate image processing by including the appropriate parameters in requests made through the GetObject API. Supported features include resizing, cropping, watermarking, and format conversion.
Video processing	TOS allows you to process video files stored within it. You can initiate video processing by including the appropriate parameters in requests made through the GetObject API. Supported video processing operations include extracting video frames and retrieving video metadata.
Hosting a static website	A static website consists of only static content, such as the homepage, images, and scripts. You can use TOS to host a static website. First, store all contents of the website in a TOS bucket. Next, configure rules to host the website so that it can be accessed via the bucket's domain name.
Mounting buckets	You can mount TOS buckets as local drives on Windows or Linux operating systems using Filesystem in Userspace (FUSE). This allows you to manage your buckets as if they were a local file system.

If you have any questions or suggestions about using TOS, please send them to us via a ticket.