Extremely slow disk access

I’ve a fully up-to-date Fedora 36 system. Disk access is extremely slow. This already is the case for several months (maybe it started after I changed some disk configuration and added a new disk).

I ran several commands to gather more disk information.

output of several commands
$ inxi -D
Drives:
  Local Storage: total: 6.16 TiB used: 1.83 TiB (29.7%)
  ID-1: /dev/sda vendor: Crucial model: CT525MX300SSD1 size: 489.05 GiB
  ID-2: /dev/sdb vendor: Western Digital model: WD60EZAZ-00ZGHB0
    size: 5.46 TiB
  ID-3: /dev/sdc vendor: Samsung model: SSD 850 EVO 250GB size: 232.89 GiB


# pvdisplay
  --- Physical volume ---
  PV Name               /dev/mapper/luks-23d92739-0aa1-4ffc-af64-18fc524bad5b
  VG Name               fedora
  PV Size               487.85 GiB / not usable 3.00 MiB
  Allocatable           yes (but full)
  PE Size               4.00 MiB
  Total PE              124889
  Free PE               0
  Allocated PE          124889
  PV UUID               ySDzbo-2Fqo-Ncip-iToe-Wzdj-dyKy-vQvV9f
   
  --- Physical volume ---
  PV Name               /dev/mapper/luks-f29fc668-9a4c-4b15-87f2-c6981be1e840
  VG Name               fedora
  PV Size               232.88 GiB / not usable 4.00 MiB
  Allocatable           yes (but full)
  PE Size               4.00 MiB
  Total PE              59617
  Free PE               0
  Allocated PE          59617
  PV UUID               1cwrqY-rPwI-Tfsn-zu3X-dKG2-Js09-XlECQT
   
  --- Physical volume ---
  PV Name               /dev/mapper/luks-42ccf8b7-fcb4-416b-b707-9a97ec819239
  VG Name               fedora
  PV Size               <5.46 TiB / not usable <1.02 MiB
  Allocatable           yes 
  PE Size               4.00 MiB
  Total PE              1430787
  Free PE               502153
  Allocated PE          928634
  PV UUID               Haga1f-6yPV-0kx3-xQty-wQ5e-3F7F-9GwyTD
  
  
# vgdisplay -v
  --- Volume group ---
  VG Name               fedora
  System ID             
  Format                lvm2
  Metadata Areas        3
  Metadata Sequence No  19
  VG Access             read/write
  VG Status             resizable
  MAX LV                0
  Cur LV                3
  Open LV               3
  Max PV                0
  Cur PV                3
  Act PV                3
  VG Size               6.16 TiB
  PE Size               4.00 MiB
  Total PE              1615293
  Alloc PE / Size       1113140 / <4.25 TiB
  Free  PE / Size       502153 / <1.92 TiB
  VG UUID               4Xddzt-Vc2G-kGYU-PvAc-c9Ej-3BdN-WnxP8J
   
  --- Logical volume ---
  LV Path                /dev/fedora/root
  LV Name                root
  VG Name                fedora
  LV UUID                7fTsmn-GmgL-WpnK-Cx2k-alrK-Kf6E-TGo8A4
  LV Write Access        read/write
  LV Creation host, time localhost-live, 2017-04-25 19:45:34 +0200
  LV Status              available
  # open                 1
  LV Size                500.00 GiB
  Current LE             128000
  Segments               2
  Allocation             inherit
  Read ahead sectors     auto
  - currently set to     256
  Block device           253:3
   
  --- Logical volume ---
  LV Path                /dev/fedora/home
  LV Name                home
  VG Name                fedora
  LV UUID                WSVGmR-27ZT-lKoN-93uY-0uOD-nVM2-mW8IyW
  LV Write Access        read/write
  LV Creation host, time localhost-live, 2017-04-25 19:45:38 +0200
  LV Status              available
  # open                 1
  LV Size                3.75 TiB
  Current LE             983119
  Segments               5
  Allocation             inherit
  Read ahead sectors     auto
  - currently set to     256
  Block device           253:5
   
  --- Logical volume ---
  LV Path                /dev/fedora/swap
  LV Name                swap
  VG Name                fedora
  LV UUID                O4b7Vv-ph3p-dqqm-ilDp-5v4j-N8nc-QU6lmB
  LV Write Access        read/write
  LV Creation host, time localhost-live, 2017-04-25 19:45:52 +0200
  LV Status              available
  # open                 2
  LV Size                7.89 GiB
  Current LE             2021
  Segments               1
  Allocation             inherit
  Read ahead sectors     auto
  - currently set to     256
  Block device           253:1
   
  --- Physical volumes ---
  PV Name               /dev/mapper/luks-23d92739-0aa1-4ffc-af64-18fc524bad5b     
  PV UUID               ySDzbo-2Fqo-Ncip-iToe-Wzdj-dyKy-vQvV9f
  PV Status             allocatable
  Total PE / Free PE    124889 / 0
   
  PV Name               /dev/mapper/luks-f29fc668-9a4c-4b15-87f2-c6981be1e840     
  PV UUID               1cwrqY-rPwI-Tfsn-zu3X-dKG2-Js09-XlECQT
  PV Status             allocatable
  Total PE / Free PE    59617 / 0
   
  PV Name               /dev/mapper/luks-42ccf8b7-fcb4-416b-b707-9a97ec819239     
  PV UUID               Haga1f-6yPV-0kx3-xQty-wQ5e-3F7F-9GwyTD
  PV Status             allocatable
  Total PE / Free PE    1430787 / 502153


# df -h
Filesystem               Size  Used Avail Use% Mounted on
devtmpfs                 4.0M     0  4.0M   0% /dev
tmpfs                     16G   13M   16G   1% /dev/shm
tmpfs                    6.3G  2.3M  6.3G   1% /run
/dev/mapper/fedora-root  492G  158G  314G  34% /
tmpfs                     16G  768K   16G   1% /tmp
/dev/sda2                974M  268M  639M  30% /boot
/dev/loop4               128K  128K     0 100% /var/lib/snapd/snap/hello-world/29
/dev/sda1                200M  6.3M  194M   4% /boot/efi
/dev/mapper/fedora-home  3.7T  1.7T  1.9T  48% /home
/dev/loop2                56M   56M     0 100% /var/lib/snapd/snap/core18/2128
/dev/loop0               100M  100M     0 100% /var/lib/snapd/snap/core/11798
/dev/loop3                56M   56M     0 100% /var/lib/snapd/snap/core18/2246
/dev/loop1               100M  100M     0 100% /var/lib/snapd/snap/core/11993
/dev/loop5               384K  384K     0 100% /var/lib/snapd/snap/universal-update-utility/140
tmpfs                    3.2G  420K  3.2G   1% /run/user/1000


# fdisk -l
Disk /dev/sda: 489.05 GiB, 525112713216 bytes, 1025610768 sectors
Disk model: Crucial_CT525MX3
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disklabel type: gpt
Disk identifier: 4CFDCADD-9CCA-4B24-9738-C92FAF5C486D

Device       Start        End    Sectors   Size Type
/dev/sda1     2048     411647     409600   200M EFI System
/dev/sda2   411648    2508799    2097152     1G Linux filesystem
/dev/sda3  2508800 1025609727 1023100928 487.9G Linux filesystem


Disk /dev/sdb: 5.46 TiB, 6001175126016 bytes, 11721045168 sectors
Disk model: WDC WD60EZAZ-00Z
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 4096 bytes
I/O size (minimum/optimal): 4096 bytes / 4096 bytes
Disklabel type: gpt
Disk identifier: E5775F11-D269-49F4-BFF8-FF4A53ED9AE0

Device     Start         End     Sectors  Size Type
/dev/sdb1   2048 11721044003 11721041956  5.5T Linux filesystem


Disk /dev/sdc: 232.89 GiB, 250059350016 bytes, 488397168 sectors
Disk model: Samsung SSD 850 
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disklabel type: gpt
Disk identifier: E9CCB5D6-1528-48CA-994B-705B1F983AD5

Device     Start       End   Sectors   Size Type
/dev/sdc1   2048 488396799 488394752 232.9G Linux filesystem


Disk /dev/mapper/luks-23d92739-0aa1-4ffc-af64-18fc524bad5b: 487.85 GiB, 523825577984 bytes, 1023096832 sectors
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes


Disk /dev/mapper/fedora-swap: 7.89 GiB, 8476688384 bytes, 16556032 sectors
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes


Disk /dev/mapper/luks-42ccf8b7-fcb4-416b-b707-9a97ec819239: 5.46 TiB, 6001156704256 bytes, 11721009188 sectors
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 4096 bytes
I/O size (minimum/optimal): 4096 bytes / 4096 bytes


Disk /dev/mapper/fedora-root: 500 GiB, 536870912000 bytes, 1048576000 sectors
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 4096 bytes
I/O size (minimum/optimal): 4096 bytes / 4096 bytes


Disk /dev/mapper/luks-f29fc668-9a4c-4b15-87f2-c6981be1e840: 232.88 GiB, 250056015872 bytes, 488390656 sectors
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes


Disk /dev/zram0: 8 GiB, 8589934592 bytes, 2097152 sectors
Units: sectors of 1 * 4096 = 4096 bytes
Sector size (logical/physical): 4096 bytes / 4096 bytes
I/O size (minimum/optimal): 4096 bytes / 4096 bytes


Disk /dev/mapper/fedora-home: 3.75 TiB, 4123499954176 bytes, 8053710848 sectors
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 4096 bytes
I/O size (minimum/optimal): 4096 bytes / 4096 bytes


Disk /dev/loop0: 99.44 MiB, 104271872 bytes, 203656 sectors
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes


Disk /dev/loop1: 99.44 MiB, 104267776 bytes, 203648 sectors
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes


Disk /dev/loop2: 55.44 MiB, 58130432 bytes, 113536 sectors
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes


Disk /dev/loop3: 55.5 MiB, 58191872 bytes, 113656 sectors
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes


Disk /dev/loop4: 20 KiB, 20480 bytes, 40 sectors
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes


Disk /dev/loop5: 272 KiB, 278528 bytes, 544 sectors
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes



# lsblk
NAME                                          MAJ:MIN RM   SIZE RO TYPE  MOUNTPOINTS
loop0                                           7:0    0  99.4M  1 loop  /var/lib/snapd/snap/core/11798
loop1                                           7:1    0  99.4M  1 loop  /var/lib/snapd/snap/core/11993
loop2                                           7:2    0  55.4M  1 loop  /var/lib/snapd/snap/core18/2128
loop3                                           7:3    0  55.5M  1 loop  /var/lib/snapd/snap/core18/2246
loop4                                           7:4    0    20K  1 loop  /var/lib/snapd/snap/hello-world/29
loop5                                           7:5    0   272K  1 loop  /var/lib/snapd/snap/universal-update-utility/140
sda                                             8:0    0   489G  0 disk  
├─sda1                                          8:1    0   200M  0 part  /boot/efi
├─sda2                                          8:2    0     1G  0 part  /boot
└─sda3                                          8:3    0 487.9G  0 part  
  └─luks-23d92739-0aa1-4ffc-af64-18fc524bad5b 253:0    0 487.9G  0 crypt 
    ├─fedora-swap                             253:1    0   7.9G  0 lvm   [SWAP]
    ├─fedora-root                             253:3    0   500G  0 lvm   /
    └─fedora-home                             253:5    0   3.8T  0 lvm   /home
sdb                                             8:16   0   5.5T  0 disk  
└─sdb1                                          8:17   0   5.5T  0 part  
  └─luks-42ccf8b7-fcb4-416b-b707-9a97ec819239 253:2    0   5.5T  0 crypt 
    ├─fedora-root                             253:3    0   500G  0 lvm   /
    └─fedora-home                             253:5    0   3.8T  0 lvm   /home
sdc                                             8:32   0 232.9G  0 disk  
└─sdc1                                          8:33   0 232.9G  0 part  
  └─luks-f29fc668-9a4c-4b15-87f2-c6981be1e840 253:4    0 232.9G  0 crypt 
    └─fedora-home                             253:5    0   3.8T  0 lvm   /home
zram0                                         252:0    0     8G  0 disk  [SWAP]


# swapon -s
Filename				Type		Size		Used		Priority
/dev/dm-1                               partition	8278012		0		-2
/dev/zram0                              partition	8388604		0		100

# tune2fs -l /dev/mapper/fedora-home
tune2fs 1.46.5 (30-Dec-2021)
Filesystem volume name:   <none>
Last mounted on:          /home
Filesystem UUID:          d14fdcb1-3776-49be-86cd-9b3c3f7f6c69
Filesystem magic number:  0xEF53
Filesystem revision #:    1 (dynamic)
Filesystem features:      has_journal ext_attr resize_inode dir_index filetype needs_recovery extent 64bit flex_bg sparse_super large_file huge_file dir_nlink extra_isize
Filesystem flags:         signed_directory_hash 
Default mount options:    user_xattr acl
Filesystem state:         clean
Errors behavior:          Continue
Filesystem OS type:       Linux
Inode count:              251682816
Block count:              1006713856
Reserved block count:     41259609
Overhead clusters:        15847996
Free blocks:              541138310
Free inodes:              248047469
First block:              0
Block size:               4096
Fragment size:            4096
Group descriptor size:    64
Reserved GDT blocks:      591
Blocks per group:         32768
Fragments per group:      32768
Inodes per group:         8192
Inode blocks per group:   512
Flex block group size:    16
Filesystem created:       Tue Apr 25 19:45:38 2017
Last mount time:          Fri Nov  4 12:39:31 2022
Last write time:          Fri Nov  4 12:39:31 2022
Mount count:              832
Maximum mount count:      -1
Last checked:             Tue Apr 25 19:45:38 2017
Check interval:           0 (<none>)
Lifetime writes:          16 TB
Reserved blocks uid:      0 (user root)
Reserved blocks gid:      0 (group root)
First inode:              11
Inode size:	          256
Required extra isize:     32
Desired extra isize:      32
Journal inode:            8
First orphan inode:       15770695
Default directory hash:   half_md4
Directory Hash Seed:      b40bae50-55b2-457b-9fbb-efbf56c8eb2c
Journal backup:           inode blocks



# tune2fs -l /dev/mapper/fedora-root
tune2fs 1.46.5 (30-Dec-2021)
Filesystem volume name:   <none>
Last mounted on:          /
Filesystem UUID:          0bc96b6e-062e-44fd-895b-9e4ce733381a
Filesystem magic number:  0xEF53
Filesystem revision #:    1 (dynamic)
Filesystem features:      has_journal ext_attr resize_inode dir_index filetype needs_recovery extent 64bit flex_bg sparse_super large_file huge_file dir_nlink extra_isize
Filesystem flags:         signed_directory_hash 
Default mount options:    user_xattr acl
Filesystem state:         clean
Errors behavior:          Continue
Filesystem OS type:       Linux
Inode count:              32768000
Block count:              131072000
Reserved block count:     5510163
Free blocks:              87847483
Free inodes:              31617457
First block:              0
Block size:               4096
Fragment size:            4096
Group descriptor size:    64
Reserved GDT blocks:      974
Blocks per group:         32768
Fragments per group:      32768
Inodes per group:         8192
Inode blocks per group:   512
Flex block group size:    16
Filesystem created:       Tue Apr 25 19:45:34 2017
Last mount time:          Fri Nov  4 12:39:25 2022
Last write time:          Fri Nov  4 12:39:21 2022
Mount count:              845
Maximum mount count:      -1
Last checked:             Tue Apr 25 19:45:34 2017
Check interval:           0 (<none>)
Lifetime writes:          3574 GB
Reserved blocks uid:      0 (user root)
Reserved blocks gid:      0 (group root)
First inode:              11
Inode size:	          256
Required extra isize:     32
Desired extra isize:      32
Journal inode:            8
First orphan inode:       5507256
Default directory hash:   half_md4
Directory Hash Seed:      6facc129-2047-46fb-8ecb-d008a80cbf81
Journal backup:           inode blocks

Roughly my disk configuration is:
2 old SSD disks
1 large hard disk (6.0 TB)

Except the /boot and /boot/efi partitions, all partitions are encrypted.

The disks are part of LVM and ext4 formatted.

How can I find the root cause of the slow disk access? How can I improve the performance of disk access?

Is the speed significantly different between writing to /boot and /? I think the below commands should work to give you a rough estimate of the speed (each command writes a 200MB file; you can delete the file after the write test is done).

sudo dd if=/dev/zero of=/boot/test0 bs=1MiB count=200 oflag=direct
sudo dd if=/dev/zero of=/test1 bs=1MiB count=200 oflag=direct

If there is a big difference in speed between the above two commands, I would guess that the problem is due to the encryption overhead, not the “disks”.

$ sudo dd if=/dev/zero of=/boot/test0 bs=1MiB count=200 oflag=direct
200+0 records in
200+0 records out
209715200 bytes (210 MB, 200 MiB) copied, 0.458204 s, 458 MB/s

$ sudo dd if=/dev/zero of=/test1 bs=1MiB count=200 oflag=direct
200+0 records in
200+0 records out
209715200 bytes (210 MB, 200 MiB) copied, 0.88073 s, 238 MB/s

Once stuff is loaded, my system is quite fast, but loading applications the fist time takes a lot of time and for example doing a dnf update can take very long.

So 458 megabytes per second unencrypted but only 238 megabytes with the encryption. I’d question whether the encryption was really worth that much of a performance hit personally. But if you really want the encryption, there might not be a whole lot that you can do about it. You might check to see if there are different algorithms that work better on your hardware. But I’d be surprised if that made more than a 10%-20% difference.