Docs

Elastic Compute Service

Method 1: Solve the local volume failure by replacing the faulty volume

Step 1: Isolate the local volume (optional)

This topic describes how to isolate a faulty local volume when a local volume failure event occurs.

Applicable scenarios

When you receive a DiskErrorDetected (disk exception) event, if you directly authorize Volcano Engine to replace the disk, your application will keep reading from and writing to the faulty local volume during processing. We recommend that you first isolate the faulty volume, make sure applications that run on the instance stop reading from and writing to the volume, then authorize Volcano Engine to replace the disk. This can help you to minimize data loss.

Usage notes

Currently, you can only isolate faulty volumes associated with Linux instances.

This feature is only available for the following instance type families:

System architecture	Instance type families
x86	Big data: Big Data d2 Big Data Storage-optimized d1s Big Data d2c Big Data d2s Big Data d2i tip If you need a trial of the Big Data d2i instance, please contact your account manager to apply. Local SSD Local SSD i2 Local SSD i3s

Procedure

Step 1: Obtain the serial number of the faulty hard disk

When a local volume mounted to an instance fails, Volcano Engine will synchronize the serial number (SN) of the faulty hard disk with you. Therefore, you can locate the faulty hard disk for the instance accurately and easily. You can call the DescribeSystemEvents operation, pass in the event ID of the hard disk exception, and obtain the SN of the faulty hard disk from the returned data.

If the instance with the faulty hard disk is a first-generation HDD instance (for example, Big Data Storage-optimized d1s), the value of the SystemEvents.ExtraInfo.Address parameter in the returned data is the SN of the faulty hard disk.

If the instance with the faulty hard disk is a second-generation HDD instance (for example, Big Data d2s, Big Data d2c, and Big Data d2), or an instance of Local SSD type (for example, Local SSD i2), the value of the SystemEvents.ExtraInfo.Serial parameter in the returned data is the SN of the faulty hard disk.

tip

For ease of use, we recommend that you record the obtained SN of the faulty disk. This is because you need to use the SN for "Step 2: Check the drive letter of the faulty local volume".

Step 2: Check the drive letter of the faulty local volume

You can view the drive letter of the faulty local volume to quickly locate the storage zone where the volume failure occurs.

Execute the following command to view the drive letter of the faulty local volume.

First-generation HDD instance

If your instance is a first-generation HDD instance (for example, Big Data Storage-optimized d1s), execute the following command to view the drive letter of the faulty hard disk.

warning

Replace $SN$ with the actual SN of the faulty hard disk.

ls -al /dev/disk/by-path/ | grep $SN$

Second-generation HDD or Local SSD instance

If your instance is a second-generation HDD instance (for example, Big Data d2s, Big Data d2c, and Big Data d2), or an instance of Local SSD type (for example, Local SSD i2), execute the following command to view the drive letter of the faulty hard disk.

warning

Replace $SN$ with the actual SN of the faulty hard disk.

lsblk -oNAME,SERIAL | grep $SN$

Step 3: View the mount point, file system, and UUID of the faulty local volume

Execute the following command to check the mount point of the faulty local volume. This example uses the faulty local volume "/dev/vda2".

df -h

tip

For ease of use, we recommend that you record the obtained mount point of the faulty local volume. This is because you need to use the mount point for "Step 4: Edit /etc/fstab file".

Execute the following command to view the file system and UUID of the faulty local volume.

blkid /dev/vda2

tip

For ease of use, we recommend that you record the obtained file system and UUID of the faulty local volume. This is because you need to use them in "Step 4: Edit /etc/fstab file".

Step 4: Edit /etc/fstab file

Execute the following command to edit the /etc/fstab file.

vim /etc/fstab

Press the i key to add the nofail parameter after the target local volume in the file.

Only the nofail parameter needs to be added and no other content requires adjustment.

Description of fstab file arguments

Parameter	Description
24fd1325-bbf3-4814-b7e9-041deba*****	File system UUID.
/	The mount point of the faulty local volume.
ext4	The file system type of the faulty local volume.
nofail	If a local volume is present in the file system but actually missing, this parameter will not interrupt the ECS instance startup process.

Press Esc and enter :wq to save the change and exit.

Step 5: Cancel the faulty local disk mount

Execute the following command to cancel mounting the faulty local volume.

Step 6: Isolate the faulty volume

In the left-side navigation pane, select "Instances & Images > Instances".

In the top menu bar, select the project and region of the resource that has triggered the event.

Find the target instance, and click the instance name to go to the instance details page.

Click the "System Events" tab, here you can filter event information based on a time range or event ID and take actions.

In the "Actions" column of the target event, select "Maintenance operation > Isolate the fault disk" to authorize Volcano Engine to isolate the faulty volume.

The isolation process may take several minutes. If it takes too long to complete, try refreshing the page.

After isolation is complete, you can hover your mouse over the "Isolate the fault disk" button.

If the Isolated state occurs, it means that the isolation succeeds.

If error feedback occurs, it means that the isolation fails.

Subsequent operation

After isolation is complete, you can select "Maintenance operation > Authorize maintenance" in the "Actions" column. Then, select the "Replace the faulty volume" to authorize and notify the O&M personnel of the Volcano Engine for maintenance. For details, refer to Replacing faulty volumes.