When is `irqbalance` needed in a Linux VM under VMware?

Question

I have a Linux farm in VMware Enterprise 5.5. The VMs are (mostly) 64-bit amd64 Debian Jessie servers with SysVinit and not systemd. The VMs have open-vm-tools installed.

I paravirtualized their Ethernet and disk controllers.

Paravirtual drivers are ones where the virtualization platform does not have to emulate another device, such as an Intel E1000 NIC or a LSI Logic SAS SCSI adapter. These paravirtual drivers essentially cut the middleman out by ditching the emulation layer, which usually results in significant performance increases.

As lspci | egrep "PVSCSI|VMXNET" can show, ethernet and disks are now paravirtualized:

3:00.0 Serial Attached SCSI controller: VMware PVSCSI SCSI Controller (rev 02)
0b:00.0 Ethernet controller: VMware VMXNET3 Ethernet Controller (rev 01)

Doing cat to /proc/interrupts is is easy to show there are interrupts associated to them and to the functionalities paravirtualization depends on:

56:    6631557          0   PCI-MSI 1572864-edge      vmw_pvscsi
57:   72647654          0   PCI-MSI 5767168-edge      eth0-rxtx-0
58:   44570979          0   PCI-MSI 5767169-edge      eth0-rxtx-1
59:          0          0   PCI-MSI 5767170-edge      eth0-event-2
60:          1          0   PCI-MSI 129024-edge       vmw_vmci
61:          0          0   PCI-MSI 129025-edge       vmw_vmci

vmw_vmci: The Virtual Machine Communication Interface. It enables high-speed communication between host and guest in a virtual environment via the VMCI virtual device.

It seems obvious using itop they are fairly used monitoring a moderately busy SSL-enabled web front end:

INT                NAME          RATE             MAX
 57 [   0          0   ]   142 Ints/s     (max:   264)
 58 [   0          0   ]   155 Ints/s     (max:   185)
 59 [   0          0   ]   119 Ints/s     (max:   419)
 60 [   0          0   ]   133 Ints/s     (max:   479)

I am quite sure irqbalance is not needed in VMs with CPU affinity, and in single core VMs. The two servers where we have CPU affinity manually configured have indeed special needs, as in general cases, the literature says irqbalance is supposed to do a better job.

So my question is, when is irqbalance necessary to distribute interrupt load via the different CPUs for multi-CPU Linux VMs?

Note: I already consulted some papers, and a related (dated) serverfault post, they are not very clear about it. I also found an academic paper voicing similar concerns for Xen. vBalance: Using Interrupt Load Balance to Improve I/O Performance for SMP Virtual Machines

Answer 1

If all the following hold

• More than one CPU in the VM
• The VM is pinned (via the host) to specific dedicated CPUs (not shared with other VMs) with a 1-1 mapping of VM CPUs to host CPUs
• The VM has dedicated (e.g. via passthrough) access to storage/network hardware

then in-VM IRQ rebalancing still makes sense.

Without multiple CPUs within the VM, in-VM IRQ rebalancing obviously serves no purpose. For the other points things become tricky because the "real" CPUs your VM is sitting on can be shuffling around underneath it and the VM's OS doesn't know which of the virtual interrupts are going to be handled by which of the real CPUs. Additionally, if the real CPU is being shared between multiple VMs you don't actually know what other work it is doing or when the virtual CPU is going to get around to being serviced so the "virtual rebalancing" could actually be making things worse...

PS: Two years ago isn't that old! Some information is timeless...

PPS: VMCI is vestigial and isn't supported on ESXi 6 or later.