< img height="1" width="1" style="display:none" src="https://www.facebook.com/tr?id=177525256911397&ev=PageView&noscript=1" />

The IOMMU issue. Boot and USB problems.

2

27leguas

Member
What IOMMU controller does?

Some PCI devices can only handle 32 bit address through DMA access (it's a legacy matter). So some PCI devices can only use the first 4GB of memory. Even on modern 64 bits systems.
When the addressable range of a device (through DMA) is limited and no IOMMU exists, the device might not be able to reach all of physical memory. In this case a region of system memory that the device can address is reserved (within the first 4GB of memory), and the device is programmed to DMA to this reserved area. The processor then copies the result to the target memory that was beyond the “reach” of the device. That takes a lot of processor resources. This method is known as bounce buffering (also known as SWIOTLB or software IOMMU).
Hardware IOMMUs are physical devices that translate (remaps) device DMA addresses (32 bit) to 64 bit physical addresses so devices could access all of physical memory without processor help.
Some IOMMU provide address translation, some other isolation (checking if an address is allowed) and most modern IOMMU both characteristics.

Fixes for boot and USB problems related to IOMMU in Linux
Different ways to fix USB and boot issues in Linux on AMD Ryzen processors:
  1. Disable IOMMU in BIOS
    1. Linux emulates the IOMMU (software IOMMU) if there is more than 4GB of RAM.
    2. Pros: Easy. No Linux setup needed.
    3. Cons: Lose of performance when Virtual Machines used.
  2. Kernel parameter: iommu=soft OR amd_iommu=off
    1. Is equivalent to the previous option. We are telling the kernel not to use the IOMMU hardware but instead emulate it through software. Is the default mode for Intel processors.
    2. Pros: You don't need to disable IOMMU in bios. Is one suitable option if you have dual boot and the other O.S. needs the IOMMU enabled.
    3. Cons: Lose of features when Virtual Machines used
  3. Kernel parameter: amd_iommu=force_enable iommu=pt
    1. amd_iommu=on is not supported but amd_iommu=force_enable is. Look like just putting iommu=pt alone works. Requires further testing! Tested!
    2. Pros: Works. At least for me. Not a software emulation. 100% hardware accelerated. Passthrough enabled
    3. Cons: iommu=pt is incompatible with Secure Memory Encryption (RAM encryption present on Ryzen Pro processors)
  4. Kernel parameter: amd_iommu=force_enable
    1. Same as fix #3 but without passthrough
  5. Kernel parameter: pci=noats
    1. Not tested. Almost no information. Comments about that solution are welcome.
Disabling or software emulating IOMMU (solution 1 and 2) fixes boot and USB problems but disables some important functions for virtualization.
If you don't use Virtual Machines its OK, but solution 3 is the one that don't penalize virtualization features.

To apply solution number 3 follow these instructions:

Edit /etc/default/grub
and add amd_iommu=force_enable iommu=pt
at the end of the line GRUB_CMDLINE_LINUX_DEFAULT

my line looks like:
Code: 
GRUB_CMDLINE_LINUX_DEFAULT="quiet splash amd_iommu=force_enable iommu=pt"
after saving changes you will have to do a update-grub:
Code: 
$ sudo update-grub
Reboot
Enter BIOS and enable:
  • IOMMU
  • SVM mode (virtualization support) Advanced->CPU configuration->SVM mode
Save settings and reboot

After rebooting your Ubuntu will boot OK, your rear USBs will work
and your virtual machines will be able to benefit from a working IOMMU with passthru option. Passthru enables virtual machines to directly use hardware component from host (a second GPU, NVME drive, Ethernet, WiFi, USB ...)

Tested on Ubuntu 20.04 Kernel 5.13.0-28 with BIOS 2.20.1275

Related documentation

Official kernel supported options:

Here you have many excellent documents for understanding what IOMMU does and its different modes:
https://lenovopress.com/lp1467.pdf
https://www.amd.com/system/files/TechDocs/48882_IOMMU.pdf
 
Last edited:
2

27leguas

Member
How to tell if you are running on an emulated IOMMU or a Hardware one?

Run dmesg command.
If you can find "PCI-DMA: Disabling IOMMU", then you aren't using any kind of IOMMU (if you have less than 4GB of RAM you don't need it)

If you can find "PCI-DMA: Using software bounce buffering for IO (SWIOTLB)", then you are running on software emulated IOMMU

If one or more of these lines are present
"PCI-DMA: using GART IOMMU" (old IOMMU mode) or
"AMD-Vi: IOMMU performance counters supported" or
"Adding to iommu group" , then you are running on hardware IOMMU
If you find "iommu: Default domain type: Passthrough" then you have passthrough enabled IOMMU

How to check if you are running on hardware IOMMU and it is initialized successfully?

The only way you have to ensure that you are running on hardware IOMMU (in passthrough mode or not) and it is initialized correctly is using this script for listing IOMMU groups:
As described in https://wiki.archlinux.org/title/PCI_passthrough_via_OVMF#Ensuring_that_the_groups_are_valid
Code: 
#!/bin/bash
shopt -s nullglob
for g in `find /sys/kernel/iommu_groups/* -maxdepth 0 -type d | sort -V`; do
    echo "IOMMU Group ${g##*/}:"
    for d in $g/devices/*; do
        echo -e "\t$(lspci -nns ${d##*/})"
    done;
done;
Output for the HM50 with kernel parameters amd_iommu=force_enable iommu=pt and IOMMU and SVM enabled in BIOS:
Code: 
IOMMU Group 0:
    00:01.0 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Renoir PCIe Dummy Host Bridge [1022:1632]
IOMMU Group 1:
    00:01.2 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD] Renoir PCIe GPP Bridge [1022:1634]
IOMMU Group 2:
    00:01.3 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD] Renoir PCIe GPP Bridge [1022:1634]
IOMMU Group 3:
    00:02.0 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Renoir PCIe Dummy Host Bridge [1022:1632]
IOMMU Group 4:
    00:02.1 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD] Renoir PCIe GPP Bridge [1022:1634]
IOMMU Group 5:
    00:02.2 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD] Renoir PCIe GPP Bridge [1022:1634]
IOMMU Group 6:
    00:02.3 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD] Renoir PCIe GPP Bridge [1022:1634]
IOMMU Group 7:
    00:08.0 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Renoir PCIe Dummy Host Bridge [1022:1632]
    00:08.1 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD] Renoir Internal PCIe GPP Bridge to Bus [1022:1635]
    00:08.2 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD] Renoir Internal PCIe GPP Bridge to Bus [1022:1635]
    06:00.0 VGA compatible controller [0300]: Advanced Micro Devices, Inc. [AMD/ATI] Renoir [1002:1636] (rev c3)
    06:00.1 Audio device [0403]: Advanced Micro Devices, Inc. [AMD/ATI] Device [1002:1637]
    06:00.2 Encryption controller [1080]: Advanced Micro Devices, Inc. [AMD] Family 17h (Models 10h-1fh) Platform Security Processor [1022:15df]
    06:00.3 USB controller [0c03]: Advanced Micro Devices, Inc. [AMD] Renoir USB 3.1 [1022:1639]
    06:00.4 USB controller [0c03]: Advanced Micro Devices, Inc. [AMD] Renoir USB 3.1 [1022:1639]
    06:00.5 Multimedia controller [0480]: Advanced Micro Devices, Inc. [AMD] Raven/Raven2/FireFlight/Renoir Audio Processor [1022:15e2] (rev 01)
    06:00.6 Audio device [0403]: Advanced Micro Devices, Inc. [AMD] Family 17h (Models 10h-1fh) HD Audio Controller [1022:15e3]
    06:00.7 Signal processing controller [1180]: Advanced Micro Devices, Inc. [AMD] Raven/Raven2/Renoir Sensor Fusion Hub [1022:15e4]
    07:00.0 SATA controller [0106]: Advanced Micro Devices, Inc. [AMD] FCH SATA Controller [AHCI mode] [1022:7901] (rev 81)
    07:00.1 SATA controller [0106]: Advanced Micro Devices, Inc. [AMD] FCH SATA Controller [AHCI mode] [1022:7901] (rev 81)
IOMMU Group 8:
    00:14.0 SMBus [0c05]: Advanced Micro Devices, Inc. [AMD] FCH SMBus Controller [1022:790b] (rev 51)
    00:14.3 ISA bridge [0601]: Advanced Micro Devices, Inc. [AMD] FCH LPC Bridge [1022:790e] (rev 51)
IOMMU Group 9:
    00:18.0 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Renoir Device 24: Function 0 [1022:1448]
    00:18.1 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Renoir Device 24: Function 1 [1022:1449]
    00:18.2 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Renoir Device 24: Function 2 [1022:144a]
    00:18.3 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Renoir Device 24: Function 3 [1022:144b]
    00:18.4 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Renoir Device 24: Function 4 [1022:144c]
    00:18.5 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Renoir Device 24: Function 5 [1022:144d]
    00:18.6 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Renoir Device 24: Function 6 [1022:144e]
    00:18.7 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Renoir Device 24: Function 7 [1022:144f]
IOMMU Group 10:
    01:00.0 Non-Volatile memory controller [0108]: Micron/Crucial Technology Device [c0a9:540a] (rev 01)
IOMMU Group 11:
    02:00.0 Ethernet controller [0200]: Realtek Semiconductor Co., Ltd. RTL8111/8168/8411 PCI Express Gigabit Ethernet Controller [10ec:8168] (rev 15)
IOMMU Group 12:
    03:00.0 Network controller [0280]: Intel Corporation Wi-Fi 6 AX200 [8086:2723] (rev 1a)
IOMMU Group 13:
    04:00.0 Ethernet controller [0200]: Intel Corporation Device [8086:15f3] (rev 03)
IOMMU Group 14:
    05:00.0 USB controller [0c03]: VIA Technologies, Inc. VL805 USB 3.0 Host Controller [1106:3483] (rev 01)
An IOMMU group is the smallest set of physical devices that can be passed to a virtual machine. For instance, in the example above, both the GPU in 06:00.0 and the front USB controller in 6:00.3 and 6:00.4 belong to IOMMU group 7 and can only be passed together. The back USB, the ethernets and WiFi controller, however, has each one its own group (group 11,12,13,14) which is separate from other devices meaning that any of them could be passed to a VM without affecting the others.

Groups may differ depending on BIOS version.
 
Last edited:
2

27leguas

Member
Who is the culprit?

Having to explicitly enable or disable AMD IOMMU through parameters in order to work properly is not normal, but is the case with HM50, HM80, and other Ryzen computers.

The culprit of this malfunction could be one of these candidates (or a combination)
  • Linux kernel: May be kernel developers aren't initializing and handling AMD IOMMU the right way. (it's totally different from Intel IOMMU) Look like AMD_IOMMU management development is a mess in linux kernel.
  • AMD BIOS (AGESA). Can a BIOS bug prevent IOMMU from working properly? Yes it could.
  • USB controller firmware. (VIA VL805). This controller is present in numerous computers (Raspberry Pi) and its firmware has a long bug record.
As pointed out by "joey" in this link: https://bbs.archlinux.org/viewtopic.php?id=214293
some people have fixed their problems with IOMMU and USB by updating VIA VL805 firmware. But could not be our case because this people have USB controllers that don't work from the very first moment and our does for a few seconds and then stops.


In this links you can find very specific information about VIA VL805 firmware and upgrading tools:
NOTE: Upgrading VIA VL805 firmware is a very dangerous procedure for your computer. I strongly discourage regular users from trying it.

This script checks which firmware version our VIA VL805 controller has:
Code: 
#!/bin/sh
# Version

lspci -d 1106:3483

if [ "$USER" != "root" ]
then
  echo root needed to report version. use sudo
#  exit 1
fi
lspci -d 1106:3483 -xxx | awk '/^50:/ { print "VL805 FW version: " $5 $4 $3 $2 }'
(from https://github.com/jpmorrison/VL805/blob/main/bin/vl805.sh)

Results from my computer:
Code: 
$ sudo lspci -d 1106:3483 -xxx | awk '/^50:/ { print "VL805 FW version: " $5 $4 $3 $2 }'
VL805 FW version: 000137b2
Look like our VL805 firmware is outdated by just one revision (there is only one later version "0138a1" from Raspberry Pi)
 
Last edited:
2

27leguas

Member
I have found a very good article that explains very clear what IOMMU does and in witch scenarios is good to enable it nowadays.

32/64 bit, IOMMU and SWIOTLB in Linux | xillybus.com

xillybus.com xillybus.com
IOMMUs were originally promoted along with 64-bit x86 processors in order to ensure proper operation of pre-PCI and PCI / PCIe devices that support DMA with 32-bit addresses only.
...
The main reason IOMMUs are rarely necessary for their original purpose (solving the 32-bit address issue) is that the vast majority of peripherals support 64-bit DMA access nowadays. Those that don’t, are very likely to get buffers in the 32-bit range, as the Linux kernel driver API allows allocating memory in the 32-bit region explicitly, and there are good chances that such memory will be available.
...
On top of all this, there’s SWIOTLB, which is a software solution for the 32/64-bit DMA problem for systems not supporting IOMMU (or when it’s disabled), which is explained further below.
...
For applications that require strict low latency (in terms of tens of microseconds) for DMA accesses, the IOMMU’s impact may be beyond acceptable. ...when a maximal latency needs to be assured. Such halt may, for example, cause a buffer overflow in data acquisition devices running at several GByte/s.
...
For most desktop applications, there is usually no perceived difference between having the IOMMU active or not.
...
First, what not to do: Don’t set iommu=off. This does indeed turn off IOMMU, but it also turns off SWIOMMU(SWIOTLB) partly: It prevents it from initializing, but not from attempts of using this feature. SWIOMMU is discussed below, but to make a long story short, it’s a feature that is unlikely to do any good. But if it’s not initialized and then later some driver attempt to allocate a DMA buffer uncleverly, a kernel panic occurs instead of what would normally end as a plain I/O error, or maybe a kernel oops at most. That is, a complete computer freeze. This is a bug in the kernel that seems to have slipped away because of the rather obscure scenario for making it show.
...
Therefore, rather use intel_iommu=off and/or amd_iommu=off to disable the IOMMU, depending on the processor architecture (or use both, it doesn’t hurt).
...
Given the IOMMUs’ address translation features, they became a crucial component for implementing virtualization that allows the guest operating system to access a DMA-capable device directly.
...
The use of IOMMUs for virtualization led to an expansion of their capabilities, in particular in the field of isolation: Without an IOMMU, any PCI / PCIe device may read from or write to any physical address of the host. A faulty device may therefore turn the computer unstable by randomly writing data to random positions in the RAM. Since the IOMMU keeps track of each device’s allocated DMA ranges by virtue of translation tables, it may also block any attempt to access an address that isn’t mapped for it.
...
The IOMMU’s isolation capability is hence primarily useful for ensuring that data doesn’t leak between a guest OS and other guests or the host OS.
...
IOMMUs are in fact not required in most common uses of virtual machines, in particular on plain desktop computers for running multiple operating systems on one machine.
...
Using IOMMUs for virtualization is hence required only when the guest needs a direct access to a PCI / PCIe device. For example, when a non-standard PCI / PCIe device needs to be accessed by the guest. Another case could be a GPU that is used for acceleration by the guest, and hence the direct access is necessary for performance reasons.
...
The SWIOTLB feature (which would have been better called SWIOMMU) was introduced to solve the 64/32-bit DMA problem for platforms without an IOMMU.
...
The price for this loan is that the data is copied by the CPU once for each DMA transaction.
As the vast majority of PCI / PCIe devices support 64-bit DMA, SWIOTLB rarely comes to action, and when it does, it’s usually because of a bug or poor system configuration. Even worse, it may cause kernel panics in some unexpected situations (that is, a complete system freeze).
The initialization of the SWIOTLB feature is announced with a kernel message like this:"PCI-DMA: Using software bounce buffering for IO (SWIOTLB)"
Click to expand...
In short:
If you virtualize, and ONLY if your virtual machines need direct access to a PCI device, then you need IOMMU
otherwise you actually don't need IOMMU or its software version (SWIOTLB) and it's better to disable it.
BUT we can't prevent some pieces of kernel trying to use SWIOTLB unless you compile your own kernel with some special options.
SO we have to activate some sort of IOMMU or SWIOTLB in order to prevent mishandle of some DMA mapping requests that aren't aware of SWIOTLB absence.

Unless you compile your own linux kernel you can't disable SWIOTLB completely. Otherwise you have to use IOMMU or SWIOTLB and do it the right way. You have to choose if you want to use the hardware or software flavor.
When you disable IOMMU in BIOS you are forcing Linux to use de software version of IOMMU (SWIOTLB), as you also do when using intel_iommu=off , amd_iommu=off, iommu=soft
 
Last edited:
2

27leguas

Member
I have done some tests with kernel parameters.
Here are de results:

BIOS: IOMMU enabled, SVM enabled
Kernel parameters: iommu=soft
relevant dmesg lines: works OK
Code: 
[    0.000000] Command line: BOOT_IMAGE=/boot/vmlinuz-5.13.0-28-generic root=UUID=xxxx ro quiet splash iommu=soft
[    0.266850] iommu: Default domain type: Translated
[    0.289591] PCI-DMA: Using software bounce buffering for IO (SWIOTLB)
[    0.865064] AMD-Vi: AMD IOMMUv2 driver by Joerg Roedel <jroedel@suse.de>
[    0.865090] AMD-Vi: AMD IOMMUv2 functionality not available on this system
BIOS: IOMMU enabled, SVM enabled
Kernel parameters: amd_iommu=off
relevant dmesg lines: works OK
Code: 
[    0.000000] Command line: BOOT_IMAGE=/boot/vmlinuz-5.13.0-28-generic root=UUID=xxxx ro quiet splash amd_iommu=off
[    0.267061] iommu: Default domain type: Translated
[    0.289433] PCI-DMA: Using software bounce buffering for IO (SWIOTLB)
[    0.789097] AMD-Vi: AMD IOMMUv2 driver by Joerg Roedel <jroedel@suse.de>
[    0.789100] AMD-Vi: AMD IOMMUv2 functionality not available on this system

BIOS: IOMMU enabled, SVM enabled
Kernel parameters: amd_iommu=on (possibly not recognized. Same as no parameter?)
relevant dmesg lines: USB stops working (last lines)
Code: 
[    0.000000] Command line: BOOT_IMAGE=/boot/vmlinuz-5.13.0-28-generic root=UUID=xxxx ro quiet splash amd_iommu=on
[    0.086753] AMD-Vi: ivrs, add hid:AMDI0020, uid:\_SB.FUR0, rdevid:160
[    0.086755] AMD-Vi: ivrs, add hid:AMDI0020, uid:\_SB.FUR1, rdevid:160
[    0.086756] AMD-Vi: ivrs, add hid:AMDI0020, uid:\_SB.FUR2, rdevid:160
[    0.086757] AMD-Vi: ivrs, add hid:AMDI0020, uid:\_SB.FUR3, rdevid:160
[    0.266225] iommu: Default domain type: Translated
[    0.289039] pci 0000:00:00.2: AMD-Vi: IOMMU performance counters supported
[    0.289073] AMD-Vi: Lazy IO/TLB flushing enabled
[    0.289089] pci 0000:00:01.0: Adding to iommu group 0
[    0.289097] pci 0000:00:01.2: Adding to iommu group 1
[    0.289104] pci 0000:00:01.3: Adding to iommu group 2
[    0.289116] pci 0000:00:02.0: Adding to iommu group 3
[    0.289123] pci 0000:00:02.1: Adding to iommu group 4
[    0.289130] pci 0000:00:02.2: Adding to iommu group 5
[    0.289137] pci 0000:00:02.3: Adding to iommu group 6
[    0.289151] pci 0000:00:08.0: Adding to iommu group 7
[    0.289157] pci 0000:00:08.1: Adding to iommu group 7
[    0.289163] pci 0000:00:08.2: Adding to iommu group 7
[    0.289176] pci 0000:00:14.0: Adding to iommu group 8
[    0.289182] pci 0000:00:14.3: Adding to iommu group 8
[    0.289209] pci 0000:00:18.0: Adding to iommu group 9
[    0.289216] pci 0000:00:18.1: Adding to iommu group 9
[    0.289222] pci 0000:00:18.2: Adding to iommu group 9
[    0.289228] pci 0000:00:18.3: Adding to iommu group 9
[    0.289236] pci 0000:00:18.4: Adding to iommu group 9
[    0.289242] pci 0000:00:18.5: Adding to iommu group 9
[    0.289249] pci 0000:00:18.6: Adding to iommu group 9
[    0.289258] pci 0000:00:18.7: Adding to iommu group 9
[    0.289266] pci 0000:01:00.0: Adding to iommu group 10
[    0.289274] pci 0000:02:00.0: Adding to iommu group 11
[    0.289281] pci 0000:03:00.0: Adding to iommu group 12
[    0.289288] pci 0000:04:00.0: Adding to iommu group 13
[    0.289296] pci 0000:05:00.0: Adding to iommu group 14
[    0.289304] pci 0000:06:00.0: Adding to iommu group 7
[    0.289306] pci 0000:06:00.1: Adding to iommu group 7
[    0.289310] pci 0000:06:00.2: Adding to iommu group 7
[    0.289313] pci 0000:06:00.3: Adding to iommu group 7
[    0.289316] pci 0000:06:00.4: Adding to iommu group 7
[    0.289320] pci 0000:06:00.5: Adding to iommu group 7
[    0.289324] pci 0000:06:00.6: Adding to iommu group 7
[    0.289327] pci 0000:06:00.7: Adding to iommu group 7
[    0.289331] pci 0000:07:00.0: Adding to iommu group 7
[    0.289334] pci 0000:07:00.1: Adding to iommu group 7
[    0.291341] pci 0000:00:00.2: AMD-Vi: Found IOMMU cap 0x40
[    0.291348] AMD-Vi: Extended features (0x206d73ef22254ade): PPR X2APIC NX GT IA GA PC GA_vAPIC
[    0.291353] AMD-Vi: Interrupt remapping enabled
[    0.291354] AMD-Vi: Virtual APIC enabled
[    0.291354] AMD-Vi: X2APIC enabled
[    0.291575] PCI-DMA: Using software bounce buffering for IO (SWIOTLB)
[    0.291577] software IO TLB: mapped [mem 0x00000000c3e59000-0x00000000c7e59000] (64MB)
[    0.300916] perf/amd_iommu: Detected AMD IOMMU #0 (2 banks, 4 counters/bank).
[    0.819448] AMD-Vi: AMD IOMMUv2 driver by Joerg Roedel <jroedel@suse.de>
[   27.974676] xhci_hcd 0000:05:00.0: WARNING: Host System Error
[   27.974726] xhci_hcd 0000:05:00.0: AMD-Vi: Event logged [IO_PAGE_FAULT domain=0x000f address=0xfed8a000 flags=0x0000]
BIOS: IOMMU enabled, SVM enabled
Kernel parameters: amd_iommu=on iommu=pt (although amd_iommu=on parameter may be not recognized, iommu=pt forces amd iommu to be enabled)
relevant dmesg lines: works OK
Code: 
[    0.000000] Command line: BOOT_IMAGE=/boot/vmlinuz-5.13.0-28-generic root=UUID=xxxx ro quiet splash amd_iommu=on iommu=pt
[    0.087464] AMD-Vi: ivrs, add hid:AMDI0020, uid:\_SB.FUR0, rdevid:160
[    0.087465] AMD-Vi: ivrs, add hid:AMDI0020, uid:\_SB.FUR1, rdevid:160
[    0.087466] AMD-Vi: ivrs, add hid:AMDI0020, uid:\_SB.FUR2, rdevid:160
[    0.087467] AMD-Vi: ivrs, add hid:AMDI0020, uid:\_SB.FUR3, rdevid:160
[    0.266833] iommu: Default domain type: Passthrough (set via kernel command line)
[    0.292571] pci 0000:00:00.2: AMD-Vi: IOMMU performance counters supported
[    0.292605] AMD-Vi: Lazy IO/TLB flushing enabled
[    0.292622] pci 0000:00:01.0: Adding to iommu group 0
[    0.292630] pci 0000:00:01.2: Adding to iommu group 1
[    0.292637] pci 0000:00:01.3: Adding to iommu group 2
[    0.292648] pci 0000:00:02.0: Adding to iommu group 3
[    0.292655] pci 0000:00:02.1: Adding to iommu group 4
[    0.292662] pci 0000:00:02.2: Adding to iommu group 5
[    0.292669] pci 0000:00:02.3: Adding to iommu group 6
[    0.292683] pci 0000:00:08.0: Adding to iommu group 7
[    0.292690] pci 0000:00:08.1: Adding to iommu group 7
[    0.292696] pci 0000:00:08.2: Adding to iommu group 7
[    0.292707] pci 0000:00:14.0: Adding to iommu group 8
[    0.292714] pci 0000:00:14.3: Adding to iommu group 8
[    0.292739] pci 0000:00:18.0: Adding to iommu group 9
[    0.292745] pci 0000:00:18.1: Adding to iommu group 9
[    0.292753] pci 0000:00:18.2: Adding to iommu group 9
[    0.292759] pci 0000:00:18.3: Adding to iommu group 9
[    0.292766] pci 0000:00:18.4: Adding to iommu group 9
[    0.292772] pci 0000:00:18.5: Adding to iommu group 9
[    0.292779] pci 0000:00:18.6: Adding to iommu group 9
[    0.292785] pci 0000:00:18.7: Adding to iommu group 9
[    0.292792] pci 0000:01:00.0: Adding to iommu group 10
[    0.292800] pci 0000:02:00.0: Adding to iommu group 11
[    0.292807] pci 0000:03:00.0: Adding to iommu group 12
[    0.292816] pci 0000:04:00.0: Adding to iommu group 13
[    0.292823] pci 0000:05:00.0: Adding to iommu group 14
[    0.292831] pci 0000:06:00.0: Adding to iommu group 7
[    0.292834] pci 0000:06:00.1: Adding to iommu group 7
[    0.292840] pci 0000:06:00.2: Adding to iommu group 7
[    0.292843] pci 0000:06:00.3: Adding to iommu group 7
[    0.292846] pci 0000:06:00.4: Adding to iommu group 7
[    0.292849] pci 0000:06:00.5: Adding to iommu group 7
[    0.292852] pci 0000:06:00.6: Adding to iommu group 7
[    0.292856] pci 0000:06:00.7: Adding to iommu group 7
[    0.292859] pci 0000:07:00.0: Adding to iommu group 7
[    0.292861] pci 0000:07:00.1: Adding to iommu group 7
[    0.293245] pci 0000:00:00.2: AMD-Vi: Found IOMMU cap 0x40
[    0.293246] AMD-Vi: Extended features (0x206d73ef22254ade): PPR X2APIC NX GT IA GA PC GA_vAPIC
[    0.293251] AMD-Vi: Interrupt remapping enabled
[    0.293252] AMD-Vi: Virtual APIC enabled
[    0.293252] AMD-Vi: X2APIC enabled
[    0.293380] PCI-DMA: Using software bounce buffering for IO (SWIOTLB)
[    0.293383] software IO TLB: mapped [mem 0x00000000c3d8a000-0x00000000c7d8a000] (64MB)
[    0.294000] perf/amd_iommu: Detected AMD IOMMU #0 (2 banks, 4 counters/bank).
[    1.063461] AMD-Vi: AMD IOMMUv2 driver by Joerg Roedel <jroedel@suse.de>
 
Last edited:
2

27leguas

Member
BIOS: IOMMU enabled, SVM enabled
Kernel parameters: amd_iommu=force_enable
relevant dmesg lines: works OK
Code: 
[    0.000000] Command line: BOOT_IMAGE=/boot/vmlinuz-5.13.0-28-generic root=UUID=xxxx ro quiet splash amd_iommu=force_enable
[    0.086622] AMD-Vi: ivrs, add hid:AMDI0020, uid:\_SB.FUR0, rdevid:160
[    0.086623] AMD-Vi: ivrs, add hid:AMDI0020, uid:\_SB.FUR1, rdevid:160
[    0.086624] AMD-Vi: ivrs, add hid:AMDI0020, uid:\_SB.FUR2, rdevid:160
[    0.086625] AMD-Vi: ivrs, add hid:AMDI0020, uid:\_SB.FUR3, rdevid:160
[    0.266635] iommu: Default domain type: Translated
[    0.291865] pci 0000:00:00.2: AMD-Vi: IOMMU performance counters supported
[    0.291900] AMD-Vi: Lazy IO/TLB flushing enabled
[    0.291917] pci 0000:00:01.0: Adding to iommu group 0
[    0.291924] pci 0000:00:01.2: Adding to iommu group 1
[    0.291931] pci 0000:00:01.3: Adding to iommu group 2
[    0.291943] pci 0000:00:02.0: Adding to iommu group 3
[    0.291950] pci 0000:00:02.1: Adding to iommu group 4
[    0.291957] pci 0000:00:02.2: Adding to iommu group 5
[    0.291964] pci 0000:00:02.3: Adding to iommu group 6
[    0.291978] pci 0000:00:08.0: Adding to iommu group 7
[    0.291984] pci 0000:00:08.1: Adding to iommu group 7
[    0.291990] pci 0000:00:08.2: Adding to iommu group 7
[    0.292001] pci 0000:00:14.0: Adding to iommu group 8
[    0.292008] pci 0000:00:14.3: Adding to iommu group 8
[    0.292033] pci 0000:00:18.0: Adding to iommu group 9
[    0.292040] pci 0000:00:18.1: Adding to iommu group 9
[    0.292046] pci 0000:00:18.2: Adding to iommu group 9
[    0.292052] pci 0000:00:18.3: Adding to iommu group 9
[    0.292058] pci 0000:00:18.4: Adding to iommu group 9
[    0.292065] pci 0000:00:18.5: Adding to iommu group 9
[    0.292072] pci 0000:00:18.6: Adding to iommu group 9
[    0.292078] pci 0000:00:18.7: Adding to iommu group 9
[    0.292086] pci 0000:01:00.0: Adding to iommu group 10
[    0.292093] pci 0000:02:00.0: Adding to iommu group 11
[    0.292100] pci 0000:03:00.0: Adding to iommu group 12
[    0.292107] pci 0000:04:00.0: Adding to iommu group 13
[    0.292115] pci 0000:05:00.0: Adding to iommu group 14
[    0.292122] pci 0000:06:00.0: Adding to iommu group 7
[    0.292126] pci 0000:06:00.1: Adding to iommu group 7
[    0.292129] pci 0000:06:00.2: Adding to iommu group 7
[    0.292132] pci 0000:06:00.3: Adding to iommu group 7
[    0.292135] pci 0000:06:00.4: Adding to iommu group 7
[    0.292138] pci 0000:06:00.5: Adding to iommu group 7
[    0.292142] pci 0000:06:00.6: Adding to iommu group 7
[    0.292145] pci 0000:06:00.7: Adding to iommu group 7
[    0.292149] pci 0000:07:00.0: Adding to iommu group 7
[    0.292152] pci 0000:07:00.1: Adding to iommu group 7
[    0.293895] pci 0000:00:00.2: AMD-Vi: Found IOMMU cap 0x40
[    0.293901] AMD-Vi: Extended features (0x206d73ef22254ade): PPR X2APIC NX GT IA GA PC GA_vAPIC
[    0.293907] AMD-Vi: Interrupt remapping enabled
[    0.293908] AMD-Vi: Virtual APIC enabled
[    0.293909] AMD-Vi: X2APIC enabled
[    0.294202] PCI-DMA: Using software bounce buffering for IO (SWIOTLB)
[    0.294204] software IO TLB: mapped [mem 0x00000000c3d90000-0x00000000c7d90000] (64MB)
[    0.294598] perf/amd_iommu: Detected AMD IOMMU #0 (2 banks, 4 counters/bank).
[    1.113151] AMD-Vi: AMD IOMMUv2 driver by Joerg Roedel <jroedel@suse.de>
 
2

27leguas

Member
Conclusions:
  • iommu=soft AND amd_iommu=off have the same effect: disables hardware IOMMU and enables SWIOTLB (software emulated IOMMU)
  • amd_iommu=on is not recognized. The same as putting nothing. USB stops working
  • iommu=pt enables amd iommu and passtrough
  • amd_iommu=force_enable is the option to use if you want hardware iommu but not passtrhough
  • There is a strong possibility that all our problems are a silly bug in linux kernel where auto detection of AMD_IOMMU skips some important component initialization that "amd_iommu=force_enable" doesn't. Its clear that with no parameters AMD_IOMMU is detected and kernel tries to use it because dmesg outputs the same as forcing it's use but very different to disabling hardware IOMMU.
Unanswered questions:
  • In all cases we have the "PCI-DMA: Using software bounce buffering for IO (SWIOTLB)" present in dmesg. Is supposed to be there when using hardware IOMMU? Is it possible that hardware and software IOMMU could be enabled simultaneously?
  • There is any performance difference between iommu=soft or amd_iommu=off and amd_iommu=force_enable?
To-Do:
 
Last edited:
You must log in or register to reply here.
Top Bottom