6

I'm running all the commands on the Proxmox host, when I LUKS encrypt a disk I get about a 28% hit in read speeds and 66% hit in write speeds. I have aes-ni enabled and I should be getting good enough performance from hardware acceleration so I don't really know whats going on... Also it doesn't make sense the write speed is faster on an unencrypted drive.

root@red:/dev/mapper# lsblk
NAME    MAJ:MIN RM   SIZE RO TYPE MOUNTPOINT
sda       8:0    0 111.8G  0 disk
├─sda1    8:1    0  1007K  0 part
├─sda2    8:2    0   512M  0 part
└─sda3    8:3    0 111.3G  0 part
sdb       8:16   0 111.8G  0 disk
├─sdb1    8:17   0  1007K  0 part
├─sdb2    8:18   0   512M  0 part
└─sdb3    8:19   0 111.3G  0 part
sdc       8:32   0  12.8T  0 disk
sdd       8:48   0  12.8T  0 disk
nvme0n1 259:0    0 465.8G  0 disk

root@red:/dev/mapper# hdparm -t /dev/nvme0n1

/dev/nvme0n1:
 HDIO_DRIVE_CMD(identify) failed: Inappropriate ioctl for device
 Timing buffered disk reads: 4774 MB in  3.00 seconds = 1591.14 MB/sec

root@red:/dev/mapper# dd if=/dev/zero of=/dev/nvme0n1 oflag=direct bs=128k count=32k
32768+0 records in
32768+0 records out
4294967296 bytes (4.3 GB, 4.0 GiB) copied, 2.39636 s, 1.8 GB/s

Then I LUKS encrypt as follows:

root@red:~# cryptsetup luksFormat /dev/nvme0n1

WARNING!
========
This will overwrite data on /dev/nvme0n1 irrevocably.

Are you sure? (Type uppercase yes): YES
Enter passphrase for /dev/nvme0n1:
Verify passphrase:

root@red:~# cryptsetup luksOpen /dev/nvme0n1 Test
Enter passphrase for /dev/nvme0n1:

root@red:~# lsblk
NAME    MAJ:MIN RM   SIZE RO TYPE  MOUNTPOINT
sda       8:0    0 111.8G  0 disk
├─sda1    8:1    0  1007K  0 part
├─sda2    8:2    0   512M  0 part
└─sda3    8:3    0 111.3G  0 part
sdb       8:16   0 111.8G  0 disk
├─sdb1    8:17   0  1007K  0 part
├─sdb2    8:18   0   512M  0 part
└─sdb3    8:19   0 111.3G  0 part
sdc       8:32   0  12.8T  0 disk
sdd       8:48   0  12.8T  0 disk
nvme0n1 259:0    0 465.8G  0 disk
└─Test  253:0    0 465.8G  0 crypt


root@red:~# hdparm -t /dev/mapper/Test

/dev/mapper/Test:
 HDIO_DRIVE_CMD(identify) failed: Inappropriate ioctl for device
 Timing buffered disk reads: 3450 MB in  3.00 seconds = 1149.84 MB/sec

root@red:~# dd if=/dev/zero of=/dev/mapper/Test oflag=direct bs=128k count=32k
32768+0 records in
32768+0 records out
4294967296 bytes (4.3 GB, 4.0 GiB) copied, 6.83405 s, 628 MB/s
root@red:~#

Here is the a benchmark of cryptsetup

root@red:~# cryptsetup benchmark
#     Algorithm |       Key |      Encryption |      Decryption
        aes-cbc        128b      1011.6 MiB/s      3031.2 MiB/s
    serpent-cbc        128b        85.2 MiB/s       646.1 MiB/s
    twofish-cbc        128b       192.3 MiB/s       354.1 MiB/s
        aes-cbc        256b       792.4 MiB/s      2475.1 MiB/s
    serpent-cbc        256b        87.2 MiB/s       648.4 MiB/s
    twofish-cbc        256b       193.2 MiB/s       355.4 MiB/s
        aes-xts        256b      1921.2 MiB/s      1919.5 MiB/s
    serpent-xts        256b       626.2 MiB/s       634.2 MiB/s
    twofish-xts        256b       349.9 MiB/s       353.1 MiB/s
        aes-xts        512b      1780.3 MiB/s      1768.5 MiB/s
    serpent-xts        512b       614.3 MiB/s       637.7 MiB/s
    twofish-xts        512b       353.2 MiB/s       352.5 MiB/s

root@red:~# root@red:~# cryptsetup luksDump /dev/nvme0n1
LUKS header information
Version:        2
Epoch:          3
Metadata area:  16384 [bytes]
Keyslots area:  16744448 [bytes]
UUID:           XXXX
Label:          (no label)
Subsystem:      (no subsystem)
Flags:          (no flags)

Data segments:
  0: crypt
    offset: 16777216 [bytes]
    length: (whole device)
    cipher: aes-xts-plain64
    sector: 512 [bytes]

root@red:~# sort -u /proc/crypto | grep module
module       : aesni_intel
module       : crc32_pclmul
module       : crct10dif_pclmul
module       : cryptd
module       : ghash_clmulni_intel
module       : kernel
module       : serpent_avx2
module       : serpent_avx_x86_64
module       : serpent_generic
module       : serpent_sse2_x86_64
module       : twofish_avx_x86_64
module       : twofish_generic
module       : twofish_x86_64
module       : twofish_x86_64_3way
root@red:~#

I'm getting around 1700MB/s on the benchmark using aes-xts so I'm wondering why the drive speed is taking such a hit? Any help would be greatly appreciated!

Improve this question
1
  • Two random notes: 1) I'm unsure whether writing zeroes to a SSD is a good write performance test; it may be handled specially. 2) the output of fdisk -l /dev/nvme0n1could be interesting (block size, alignment, etc. Commented Jan 3, 2024 at 19:29

1 Answer 1

Reset to default
7

From cryptsetup-benchmark man page:

NOTE: This benchmark is using memory only and is only informative. You cannot directly predict real storage encryption speed from it.

This is also hinted at in the very first line of cryptsetup benchmark output (emphasis is mine):

root@archiso ~ # cryptsetup benchmark                                                                  
# Tests are approximate using memory only (no storage IO).
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
PBKDF2-sha1      1875806 iterations per second for 256-bit key
...

This is why you see such discrepancy when actually involving the storage layer with dd. Filesystem type and mount options will themselves add their own overhead, e.g. on Btrfs with zstd level 1 soft compression:

root@archiso ~ # hdparm -t /dev/nvme0n1 
/dev/nvme0n1:
 HDIO_DRIVE_CMD(identify) failed: Inappropriate ioctl for device
 Timing buffered disk reads: 4658 MB in  3.00 seconds = 1552.45 MB/sec

# cryptsetup luksFormat  /dev/nvme0n1p2
# cryptsetup open --allow-discards !$ luks

# hdparm -t /dev/mapper/luks
/dev/mapper/luks:
 HDIO_DRIVE_CMD(identify) failed: Inappropriate ioctl for device
 Timing buffered disk reads: 3638 MB in  3.00 seconds = 1212.52 MB/sec

# mkfs.btrfs --label btrfs -m dup --csum xxhash /dev/mapper/luks
# mount -t btrfs -o noatime,ssd,compress=zstd:1,autodefrag /dev/disk/by-label/btrfs /mnt
# dd if=/dev/zero of=/mnt/test oflag=direct bs=128k count=32k
32768+0 records in
32768+0 records out
4294967296 bytes (4.3 GB, 4.0 GiB) copied, 7.76791 s, 553 MB/s

Without compression I get results similar to yours:

# rm /mnt/test
# umount /mnt
# mount -t btrfs -o noatime,ssd,autodefrag /dev/disk/by-label/btrfs /mnt 
# dd if=/dev/zero of=/mnt/test oflag=direct bs=128k count=32k
32768+0 records in
32768+0 records out
4294967296 bytes (4.3 GB, 4.0 GiB) copied, 6.35182 s, 676 MB/s

This is on an Intel 660p 512G NVMe drive with Intel core i5-10210U processor and 2x4G LPDDR3 2133 MT/s Samsung memory. It is sad indeed that in order to get the benefits of software encryption we have to give up on so much performance from NVMe storage.

Cloudflare did some work on speeding up Linux disk encryption which got merged in Linux 5.9 and made generally available in 5.10.9, but there are reports that it actually harms performance once storage is involved, which I could corroborate in my own use case.

Assuming your software is up to date, you can use cryptsetup open with:

--perf-no_read_workqueue, --perf-no_write_workqueue

Bypass dm-crypt internal workqueue and process read or write requests synchronously.

NOTE: These options are available only for low-level dm-crypt performance tuning, use only if you need a change to default dm-crypt behaviour. Needs kernel 5.9 or later.

The aforementionned Reddit link suggests enabling the read option but leaving out the write one. As always, you should test and benchmark for yourself!

Improve this answer
3

You must log in to answer this question.

Start asking to get answers

Find the answer to your question by asking.

Ask question

Explore related questions

See similar questions with these tags.