Disk I/O issue in Proxmox VM (Aborted Command) + XFS shut down

Support
1

Following this thread : https://forum.proxmox.com/threads/io_uring-on-kernel-7-0-6-2-pve-pve-9-2-3-guest-disk-i-o-errors-eio-filesystem-xfs-shutdown.184360/

I discovered a strange behavior on AlmaLinux 10 inside a Proxmox VM. It appears that kernel 6.12.0-211.22.1.el10_2.x86_64 has issue that triggers a disk failure (see screenshots).

Randomly, the filesystem (XFS) becomes unavailable and I can’t do anything more except stop and start the VM again.

I don’t know if it’s related to the storage architecture on Proxmox but I’m using LUKS + LVM-thin on 2 different hosts that are impacted.

Here is the way I was able to reproduce inside the VM :

root@zabbix :~$ fio --name=stress-write --ioengine=libaio --iodepth=64 --rw=randwrite \
    --bs=4k --direct=1 --size=20G --numjobs=8 --runtime=600 \
    --time_based --ramp_time=30 --group_reporting \
    --filename=/tmp/fio-test --output-format=normal,json \
    --output=fio-result.json
Jobs: 8 (f=0): [/(8)][-.-%][eta 10m:15s]
Jobs: 8 (f=0): [/(8)][-.-%][eta 10m:15s]
Jobs: 8 (f=0): [/(8)][-.-%][eta 10m:14s]
fio: io_u error on file /tmp/fio-test: Input/output error: write offset=20592291840, buflen=4096
fio: io_u error on file /tmp/fio-test: Input/output error: write offset=9765498880, buflen=4096
fio: io_u error on file /tmp/fio-test: Input/output error: write offset=11389923328, buflen=4096
fio: io_u error on file /tmp/fio-test: Input/output error: write offset=10865082368, buflen=4096
fio: io_u error on file /tmp/fio-test: Input/output error: write offset=4592693248, buflen=4096
fio: io_u error on file /tmp/fio-test: Input/output error: write offset=17240551424, buflen=4096
fio: io_u error on file /tmp/fio-test: Input/output error: write offset=10608463872, buflen=4096
fio: io_u error on file /tmp/fio-test: Input/output error: write offset=18193764352, buflen=4096
fio: io_u error on file /tmp/fio-test: Input/output error: write offset=10405310464, buflen=4096
fio: io_u error on file /tmp/fio-test: Input/output error: write offset=18420568064, buflen=4096
fio: io_u error on file /tmp/fio-test: Input/output error: write offset=20068622336, buflen=4096
fio: io_u error on file /tmp/fio-test: Input/output error: write offset=14732562432, buflen=4096
fio: io_u error on file /tmp/fio-test: Input/output error: write offset=19714949120, buflen=4096
fio: io_u error on file /tmp/fio-test: Input/output error: write offset=11894398976, buflen=4096
fio: io_u error on file /tmp/fio-test: Input/output error: write offset=13342396416, buflen=4096
fio: io_u error on file /tmp/fio-test: Input/output error: write offset=18269777920, buflen=4096
fio: io_u error on file /tmp/fio-test: Input/output error: write offset=21153767424, buflen=4096
fio: io_u error on file /tmp/fio-test: Input/output error: write offset=8210624512, buflen=4096
fio: io_u error on file /tmp/fio-test: Input/output error: write offset=5009162240, buflen=4096

I also open an issue here : 0000634: Disk I/O issue in Proxmox VM (Aborted Command) + XFS shut down - MantisBT

This is a critical problem (specific but critical).

2

Hello,

Thank you for the additional updates.

Based on the latest test results, this looks more like a regression in the Proxmox 7.x host kernel rather than an AlmaLinux 10.2 guest-only issue.

The key point is that the same VM and the same fio stress-write workload did not reproduce the issue when the Proxmox host was booted with 6.17.13-13-pve, while the issue appears with 7.0.x-pve kernels such as 7.0.6-2-pve and 7.0.12-1-pve.

Also, since the problem can now be reproduced with AlmaLinux 10.1 as well, AlmaLinux 10.2 no longer seems to be the only trigger.

From the guest side, XFS appears to be shutting down as a consequence of receiving EIO / Aborted Command from the virtual block device.

So I think it would be useful to focus the debugging on the Proxmox host kernel / QEMU / virtio-scsi / LVM-thin stack.

Related Proxmox bug report: