notes.neulen.net

Virtualization

Mar 16, 20254 min read

Enabling IOMMU

To enable IOMMU - which is an essential virtualization feature for the passthrough of PCIe devices - it can be necessary to edit the grub file of proxmox to enable this feature during startup. Frist of all the CPU and the motherboard used have to support virtualization (e.g. VTd / VTx for Intel CPUS) and IOMMU. This has to be enabled in the BIOS since it is by mostly deactivated by default. If proxmox can not find the IOMMU feature the grub file has to be edited. To check if IOMMU is available following command can be used:

dmesg | grep -e DMAR -e IOMMU

It should output something like this:

[    0.028730] DMAR: IOMMU enabled

If this is not the case the grub file located at /etc/default/grub has to be edited. The file should be edited to include the following line. Make sure to edit it based on the CPU manufacturer (Intel / AMD):

GRUB_CMDLINE_LINUX_DEFAULT="quiet <intel/amd>_iommu=on"

After this change the system has to be updated using the following command and rebooted.

Debian

update-grub

Fedora

sudo grub2-mkconfig -o /etc/grub2-efi.cfg

Now IOMMU should be available if it is supported by the system.

Split up IOMMU Groups

Sources:

It is important to check which devices are on which IOMMU group when setting up a virtualization environment. Using the follwing command the groups will be listed:

for d in /sys/kernel/iommu_groups/*/devices/*; do n=${d#*/iommu_groups/*}; n=${n%%/*}; printf 'IOMMU group %s ' "$n"; lspci -nns "${d##*/}"; done

En exemplary output could look like this:

IOMMU group 0 00:01.0 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Renoir PCIe Dummy Host Bridge [1022:1632]
IOMMU group 10 16:00.0 USB controller [0c03]: Advanced Micro Devices, Inc. [AMD] Device [1022:43bc] (rev 02)
IOMMU group 11 16:00.1 SATA controller [0106]: Advanced Micro Devices, Inc. [AMD] Device [1022:43b8] (rev 02)
IOMMU group 12 16:00.2 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD] Device [1022:43b3] (rev 02)
IOMMU group 13 20:04.0 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD] 300 Series Chipset PCIe Port [1022:43b4] (rev 02)
IOMMU group 14 20:05.0 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD] 300 Series Chipset PCIe Port [1022:43b4] (rev 02)
IOMMU group 15 20:06.0 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD] 300 Series Chipset PCIe Port [1022:43b4] (rev 02)
IOMMU group 16 20:07.0 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD] 300 Series Chipset PCIe Port [1022:43b4] (rev 02)
IOMMU group 17 25:00.0 Ethernet controller [0200]: Realtek Semiconductor Co., Ltd. RTL8111/8168/8411 PCI Express Gigabit Ethernet Controller [10ec:8168] (rev 15)
IOMMU group 18 26:00.0 Ethernet controller [0200]: Realtek Semiconductor Co., Ltd. RTL8125 2.5GbE Controller [10ec:8125] (rev 04)
IOMMU group 19 30:00.0 VGA compatible controller [0300]: Advanced Micro Devices, Inc. [AMD/ATI] Cezanne [Radeon Vega Series / Radeon Vega Mobile Series] [1002:1638] (rev c9)
IOMMU group 1 00:01.1 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD] Renoir PCIe GPP Bridge [1022:1633]
...
...

In this example can be seen that IOMMU support is enabled since there are actual groups assigned, indicated by the numbers behind IOMMU group. If you only see an * IOMMU is not successfull enabled. In this case check the chapter Enabling IOMMU.

Another problem that can occur is that multiple devices are in one single IOMMU group. In that case it would only be possible to passthrough all of those devices to a virtual machine. Sometome this is not feasible - for instance when th onboard network card and a PCIe network card are in one IOMMU group. The virtualization host would not be accessible if its network interface would be passed through to the virtual machine. In this case it is possible to pslit up the IOMMU groups - which is an official feature, but it should be used with care, since it can have security and stability implications.

To split up the IOMMU groups grub has to be edited. Add the following option to the line containing GRUB_CMDLINE_LINUX_DEFAULT in the file /etc/default/grub:

pcie_acs_override=downstream,multifunction

It should then look somehting like this:

GRUB_CMDLINE_LINUX_DEFAULT="quiet amd_iommu=on pcie_acs_override=downstream,multifunction"

After adding this option run this:

update-grub

After a restart the IOMMU groups should now be split up and ready to be passed to virtual machines as individual devices.

Passing devices through in FreeBSD

Take a look at PCIe Passthrough.

Graph View

Backlinks