Docs

Elastic Compute Service

Local Disk issue

Method for upgrading FW version of NVME disks in Linux system

This topic mainly describes how to resolve the issue of probabilistic disk drop for NVMe disks in misaligned business scenarios by upgrading the NVMe disk firmware (FW) version online.

FW version of NVMe disk refers to the firmware version of an NVMe (Non-Volatile Memory Express) disk. It is the core program version for interaction between disk hardware and the system, affecting disk stability, performance, and other aspects. Firmware serves as the "operating system" of NVMe disks, and its version determines the disk's key capabilities.

Problem description

For Local SSD instances mounted with certain models of NVME disks, probabilistic disk drop may occur due to defects in the disk firmware version. As a result, the Linux operating system cannot recognize the corresponding local volume, and the instance cannot perform read or write operations on this local volume.

tip

During the upgrade for non-specific models of NVME disks, a prompt appears indicating that the local volume firmware version is already the latest and no upgrade is needed.

Causes

When two 512 B unaligned I/O operations (initiated separately by the primary CPU and the secondary CPU) access the same 4 KB physical unit (IU), data is retrieved from different locations within the IU that may overlap. As a result, this forms a cross-access region within the IU.

Specifically, disk physical space is divided into multiple 4 KB IUs, and the size of the I/O that is actually written to the disk is 512 B logical sectors. When a write I/O falls into a certain IU, if another write I/O also sequentially falls into the same IU, the disk adopts a per-IU queuing mechanism for logical I/O processing. During the simultaneous processing of these two I/Os, timing errors may cause queue abnormalities, causing disk status exceptions and I/O timeouts. At this time, the kernel tries to recover via a reset. However, the reset fails due to the disk in an abnormal status, resulting in a disk drop.

Usage note

Time of upgrading firmware version for one disk < 5s

Maximum I/O latency during firmware version upgrade for one disk< 5s

Procedure

warning

Disk I/O latency may increase during the upgrade. We recommend performing the upgrade during off-peak business hours or when the business is idle.

Execute the following command to upgrade the firmware version of a local volume online.

region=$(wget -q -O - http://100.96.0.96/volcstack/latest/region_id);bash -c "$(wget -q -O - http://nvme-tools-${region}.tos-${region}.ivolces.com/update_nvme_firmware.sh)"

tip

You can directly execute the above command without modifying any information.

Check the upgrade result:

If Update INTEL disk firmware success is prompted, this indicates the upgrade succeeds.

If INTEL disk firmware is 9CV10501, no update is needed or There is no INTEL NVMe disk, so no update is needed is prompted, this indicates the firmware version of the local volume firmware version is already the latest, and no upgrade is needed.

To obtain support when other messages are prompted, submit a ticket or contact your account manager.

Suggestion after upgrade

We recommend observing the stability of disk-dependent services within 24 hours after the upgrade. For example, check if the database has read/write latency and if the file storage response speed is normal to confirm no new exceptions.

Last updated: 2026.04.13 16:50:03