Volcano Engine E-MapReduce (EMR) offers two types of payment options: Subscription and Pay-As-You-Go The cost depends on the instance type, storage type, and storage size. This chapter introduces how to optimize Volcano Engine EMR clusters and leverage features such as elastic scaling and EMR Stateless to reduce costs.

1 Cost optimization #

Volcano Engine EMR provides various features to help reduce costs. To fully utilize these features, consider the following factors.

1.1 Workload types #

You can run different applications and workload types on Volcano Engine EMR.

• For applications that run only during specific periods, you can use transient EMR clusters, which are based on the Volcano Engine EMR Stateless cloud-native open-source big data platform and offer exceptional elasticity. You can quickly create and release clusters, and view historical job information and more.
• For applications that run for extended periods, you can use long-running clusters.

1.2 Instance types #

Most Volcano Engine EMR clusters can run on general-purpose ECS instance types (g2i and g1ie). Compute-intensive clusters can benefit from compute instance types (c2i and c1ie). OLAP clusters and memory-intensive jobs can benefit from memory instance types (r1ie). The primary node typically does not require significant compute resources, so a general-purpose instance type can be selected.

1.3 Storage types #

Volcano EMR clusters support object storage TOS (Tinder Object Storage) as a storage medium. Volcano Engine Object Storage TOS is a distributed cloud storage service provided by Volcano Engine, offering massive capacity, security, low cost, ease of use, high reliability, and high availability. TOS supports three storage types: standard, infrequent access, and archive. You can optimize storage costs by tiering data based on data access frequency.

2 Storage optimization #

Storage optimization can improve job performance. Here are some strategies to help you optimize cluster storage.

• Data partitioning
  When partitioning data and reading data based on partitions, queries only read the required files. This reduces the amount of data scanned during queries.

• File size
  It is necessary to avoid the issue of having too many small files. Having too many small files increases the pressure on the NameNode, resulting in metadata bloat and slower lookup and indexing performance. The Hadoop analytics engine processes a large number of small files much slower than it processes an equivalent amount of data in large files. Each small file occupies a task, and starting a task consumes a significant amount of time, causing most of the job time to be spent on starting and releasing tasks. You can log in to the cluster and use the hadoop -fs count command to count the number of files.

  hadoop fs -count /datadir/xxx/xxxx
  

• Data compression
  Data compression can reduce storage space and decrease the number of bytes read from HDFS, which means the time required to read data will be reduced. This helps improve job execution performance.

• Storage format
  Parquet and ORC are column-based storage formats, while Apache Avro is a row-based storage format. Column-based storage formats are the best choice for queries that only access certain columns of the data, whereas row-based storage formats can be selected when querying all columns. Parquet and ORC provide better compression ratios than Avro.

3 Computation optimization #

In previous sections, we introduced storage cost and performance optimization. The following sections introduce features and methods for optimizing computation in Volcano Engine EMR.

• Auto scaling
  EMR provides auto scaling capabilities, automatically adjusting the management of computing resources based on your business requirements and strategies. When using an EMR cluster, if frequent peaks and valleys occur, automatically expanding temporary computing capacity during peak periods helps handle business computation surges, and automatically scaling down after the peak reduces the user's computing resource costs.
  EMR supports two types of elastic scaling rules: time-based and load-based. For details, refer to Elastic scaling.
• Stateless
  EMR Stateless externalizes state data such as user data, permission data, Hive metadata, log data, and elasticity rules, transforming long-running clusters into lightweight transient clusters. From cluster creation and waiting for task execution, to triggering cluster creation and release at the start and end of tasks, this approach delivers improved elasticity, scalability, and security, enabling the data platform architecture to evolve more effectively and achieve optimal cost optimization. Transient clusters greatly meet users' growing demands for component version upgrades, ultimate cost optimization, and more.
  EMR Stateless allows clusters to start upon job submission and be released upon job completion, turning long-running clusters into lightweight transient clusters. You only pay for cluster resources during the runtime, significantly optimizing computing costs.
  

4 What's next #

After cost evaluation and optimization are complete, the following section will introduce the data migration process. For details, see Data migration.