QEMU GPU vfio-pci pass through

@tlosm
Quote:May be a problem on a real PegasosII, but @balaton fixed all problems (by converting 64 bit BARs to 32 bit, interrupt related fixes, etc.) in his bboot.fth and BBoot, and probably in QEmu itself as well.
There should be no problems using the QEmu emulation of the PegasosII any more with PCIe gfx cards, at least not with AmigaOS 4.x, MorphOS may be different.
Using BBoot on a real PegasosII is probably not possible, but using bboot.fth should be, and it might even help getting PCIe gfx cards working with a PCI<->PCIe bridge.
The real AmigaOne SE/XE/µA1 systems have at least 100 times as much problems, hardware limits and bugs as the QEmu emulation of it, but QEmu doesn't emulate the hardware limits and bugs of the PegasosII and AmigaOne SE/XE/µA1

Several people got Radeon HD/RX PCIe cards working with vfio-pci with QEmu, and all used the PegasosII emulation since it used to be the best one for AmigaOS 4.x.
Now, with NVRAM support etc. added, the AmigaOne XE emulation is probably better than the PegasosII one for AmigaOS 4.x.
Sam4x0 emulation is too slow, one of the reasons is the missing "real" MMU, using TLB caches instead, which cause a lot of exceptions, and which probably can't be fixed/worked around in the QEmu emulation.
However, only @geenaam got vfio-pci working with usable speeds, for everyone else it was extremely slow, the real Radeon PCIe gfx card was even much slower than using SM501/502 software emulation.

What @white has to fix are the iommu groups, the Radeon gfx card has to be in a separate group with nothing else (required by the host) in it.
The easiest, and maybe only, option is trying to use a different PCIe slot for the Radeon gfx card.

@joerg
Quote:

Just to clarify a bit: I use bboot on real pegasos2 and it works fine. It speed up loading of os4 for about 10 seconds (30s vs 40s with os4s loader)

And, while maybe not very important at the moment because of "beta":

Quote:

In beta kernel for os4 all this also fixed already, and i can use real RadeonRx and RadeonHd via bridges, but.. that not released for now, sadly.

@joerg

Quote:

Swapping PCI slots is one way. The other is to use ACS-overwrite. This requires both a new kernel and some lines in the grub boot. I don't have the link handy that tells how to do that, butas for the ACS overwrite/patch, you just need ot install linux-zen as the kernel (I think I did that on arch linux using pacman -Sy linux-zen or similar... can be found online). Think it is similar on ubuntu-based systems.

After that, and reboot, etc, you will see only a single pci device per IOMMU group (you will have many groups). You can then just make a '-device ' entry in the qemu command line for just the video part and not bother with the audio if you don't want.

@themamboman

CachyOS based on Arch and its Linux kernels also have similar functionality.

https://wiki.cachyos.org/cachyos_basic/why_cachyos/
https://github.com/CachyOS/kernel-patches

The Linux kernel for the desktop in popular distributions is best recompiled yourself.
The versions included in the distributions are too (so to speak...) neutral.

Current situation
It seems that ACS with the Liquorix kernel is now working or so it seems.
I always write what I do so as to keep a reference page in the Forum.

Steps:
uname -r
6.14. 0-1-liquorix-amd64

then use the Kernel Stub:

sudo kernelstub --add-options "pcie_acs_override=downstream,multifunction" <<<----- THIS ENABLES ACS IOMMU with Liquorix kernel

sudo update-initramfs -u


reboot

It seems that ACS is now enabled with the Liquorix Kernel

for d in $(find /sys/kernel/iommu_groups/ -type l | sort -n -k5 -t/); do n=${d#*/iommu_groups/*}; n=${n%%/*}; printf 'IOMMU Group %s ' "$n"; lspci -nns "${d##*/}"; done;

New IOMMU Result:

IOMMU Group 0 00:00.0 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Starship/Matisse Root Complex [1022:1480]
IOMMU Group 1 00:01.0 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Starship/Matisse PCIe Dummy Host Bridge [1022:1482]
IOMMU Group 2 00:01.2 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD] Starship/Matisse GPP Bridge [1022:1483]
IOMMU Group 3 00:02.0 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Starship/Matisse PCIe Dummy Host Bridge [1022:1482]
IOMMU Group 4 00:03.0 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Starship/Matisse PCIe Dummy Host Bridge [1022:1482]
IOMMU Group 5 00:03.1 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD] Starship/Matisse GPP Bridge [1022:1483]
IOMMU Group 6 00:04.0 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Starship/Matisse PCIe Dummy Host Bridge [1022:1482]
IOMMU Group 7 00:05.0 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Starship/Matisse PCIe Dummy Host Bridge [1022:1482]
IOMMU Group 8 00:07.0 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Starship/Matisse PCIe Dummy Host Bridge [1022:1482]
IOMMU Group 9 00:07.1 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD] Starship/Matisse Internal PCIe GPP Bridge 0 to bus[E:B] [1022:1484]
IOMMU Group 10 00:08.0 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Starship/Matisse PCIe Dummy Host Bridge [1022:1482]
IOMMU Group 11 00:08.1 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD] Starship/Matisse Internal PCIe GPP Bridge 0 to bus[E:B] [1022:1484]
IOMMU Group 12 00:14.0 SMBus [0c05]: Advanced Micro Devices, Inc. [AMD] FCH SMBus Controller [1022:790b] (rev 61)
IOMMU Group 12 00:14.3 ISA bridge [0601]: Advanced Micro Devices, Inc. [AMD] FCH LPC Bridge [1022:790e] (rev 51)
IOMMU Group 13 00:18.0 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Matisse/Vermeer Data Fabric: Device 18h; Function 0 [1022:1440]
IOMMU Group 13 00:18.1 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Matisse/Vermeer Data Fabric: Device 18h; Function 1 [1022:1441]
IOMMU Group 13 00:18.2 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Matisse/Vermeer Data Fabric: Device 18h; Function 2 [1022:1442]
IOMMU Group 13 00:18.3 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Matisse/Vermeer Data Fabric: Device 18h; Function 3 [1022:1443]
IOMMU Group 13 00:18.4 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Matisse/Vermeer Data Fabric: Device 18h; Function 4 [1022:1444]
IOMMU Group 13 00:18.5 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Matisse/Vermeer Data Fabric: Device 18h; Function 5 [1022:1445]
IOMMU Group 13 00:18.6 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Matisse/Vermeer Data Fabric: Device 18h; Function 6 [1022:1446]
IOMMU Group 13 00:18.7 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Matisse/Vermeer Data Fabric: Device 18h; Function 7 [1022:1447]
IOMMU Group 14 01:00.0 USB controller [0c03]: Advanced Micro Devices, Inc. [AMD] Device [1022:43ee]
IOMMU Group 15 01:00.1 SATA controller [0106]: Advanced Micro Devices, Inc. [AMD] Device [1022:43eb]
IOMMU Group 16 01:00.2 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD] Device [1022:43e9]
IOMMU Group 17 02:00.0 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD] Device [1022:43ea]
IOMMU Group 18 02:08.0 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD] Device [1022:43ea]
IOMMU Group 19 02:09.0 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD] Device [1022:43ea]
IOMMU Group 20 03:00.0 VGA compatible controller [0300]: Advanced Micro Devices, Inc. [AMD/ATI] Tahiti XT [Radeon HD 7970/8970 OEM / R9 280X] [1002:6798]
IOMMU Group 21 03:00.1 Audio device [0403]: Advanced Micro Devices, Inc. [AMD/ATI] Tahiti HDMI Audio [Radeon HD 7870 XT / 7950/7970] [1002:aaa0]
IOMMU Group 22 05:00.0 Ethernet controller [0200]: Realtek Semiconductor Co., Ltd. RTL8125 2.5GbE Controller [10ec:8125] (rev 04)
IOMMU Group 23 06:00.0 VGA compatible controller [0300]: NVIDIA Corporation Device [10de:2882] (rev a1)
IOMMU Group 24 06:00.1 Audio device [0403]: NVIDIA Corporation Device [10de:22be] (rev a1)
IOMMU Group 25 07:00.0 Non-Essential Instrumentation [1300]: Advanced Micro Devices, Inc. [AMD] Starship/Matisse PCIe Dummy Function [1022:148a]
IOMMU Group 26 08:00.0 Non-Essential Instrumentation [1300]: Advanced Micro Devices, Inc. [AMD] Starship/Matisse Reserved SPP [1022:1485]
IOMMU Group 27 08:00.1 Encryption controller [1080]: Advanced Micro Devices, Inc. [AMD] Starship/Matisse Cryptographic Coprocessor PSPCPP [1022:1486]
IOMMU Group 28 08:00.3 USB controller [0c03]: Advanced Micro Devices, Inc. [AMD] Matisse USB 3.0 Host Controller [1022:149c]
IOMMU Group 29
08:00.4 Audio device [0403]: Advanced Micro Devices, Inc. [AMD] Starship/Matisse HD Audio Controller [1022:1487]
white@pop-os:~$

Note:
I guess I need to reconfigure the ALIAS now

Now the Radeon card is here IOMMU group 20 and 21:

IOMMU Group 20 03:00.0 VGA compatible controller [0300]: Advanced Micro Devices, Inc. [AMD/ATI] Tahiti XT [Radeon HD 7970/8970 OEM / R9 280X] [1002:6798]
IOMMU Group 21 03:00.1 Audio device [0403]: Advanced Micro Devices, Inc. [AMD/ATI] Tahiti HDMI Audio [Radeon HD 7870 XT / 7950/7970] [1002:aaa0]

--------------------------------------------------------------

If you want, you can also manually edit this file:
/etc/kernelstub

{
"default": {
"kernel_options": [
"quiet",
"splash"
],
"esp_path": "/boot/efi",
"setup_loader": false,
"manage_mode": false,
"force_update": false,
"live_mode": false,
"config_rev": 3
},
"user": {
"kernel_options": [
"vfio-pci.ids",
"pcie_acs_override=downstream,multifunction"
],
"esp_path": "/boot/efi",
"setup_loader": true,
"manage_mode": true,
"force_update": false,
"live_mode": false,
"config_rev": 3
}
}


@Balaton
now the IOMMU groups are divided
despite this

the result is this:
sudo qemu-system-ppc -M pegasos2 -bios /home/white/Scaricati/pegasos2.rom -m 1024 -serial stdio -device VGA,romfile="" -drive if=none,id=hd,file=/home/white/Scaricati/32gb.raw,format=raw -device ide-hd,drive=hd,bus=ide.1 -netdev user,id=tap0 -device rtl8139,netdev=tap0 -vga none -drive if=none,id=hd1,file=/home/white/Scaricati/coffin.raw,format=raw -device ide-hd,drive=hd1,bus=ide.1 -device vfio-pci,host=03:00.0,bus=pci.0 -device vfio-pci.host=03:00.1,bus=pci.0 -accel tcg

qemu-system-ppc: -device vfio-pci,host=03:00.0,bus=pci.0: vfio 0000:03:00.0: Could not open '/dev/vfio/20': No such file or directory

I guess there is something to fix although IOMMU now seems to work fine.

But a step forward has been made now this can be applied to any Linux distribution with Liquorix kernel.

additional information if it may be useful:
sudo dmesg | grep -i -e DMAR -e IOMMU

result:
[ 0.000000] Command line: initrd=\EFI\Pop_OS-826942db-e6c1-41bf-881f-dbe3ff07ca95\initrd.img root=UUID=826942db-e6c1-41bf-881f-dbe3ff07ca95 ro vfio-pci.ids pcie_acs_override=downstream,multifunction amd_iommu=on iommu=pt vfio.pci-ids=1002:6798,1002:aaa0
[ 0.000000] Warning: PCIe ACS overrides enabled; This may allow non-IOMMU protected peer-to-peer DMA
[ 0.010008] Kernel command line: audit=0 intel_pstate=disable amd_pstate=disable initrd=\EFI\Pop_OS-826942db-e6c1-41bf-881f-dbe3ff07ca95\initrd.img root=UUID=826942db-e6c1-41bf-881f-dbe3ff07ca95 ro vfio-pci.ids pcie_acs_override=downstream,multifunction amd_iommu=on iommu=pt vfio.pci-ids=1002:6798,1002:aaa0
[ 0.586095] iommu: Default domain type: Passthrough (set via kernel command line)
[ 0.611288] pci 0000:00:00.2: AMD-Vi: IOMMU performance counters supported
[ 0.611335] pci 0000:00:00.0: Adding to iommu group 0
[ 0.611357] pci 0000:00:01.0: Adding to iommu group 1
[ 0.611373] pci 0000:00:01.2: Adding to iommu group 2
[ 0.611391] pci 0000:00:02.0: Adding to iommu group 3
[ 0.611413] pci 0000:00:03.0: Adding to iommu group 4
[ 0.611428] pci 0000:00:03.1: Adding to iommu group 5
[ 0.611446] pci 0000:00:04.0: Adding to iommu group 6
[ 0.611463] pci 0000:00:05.0: Adding to iommu group 7
[ 0.611485] pci 0000:00:07.0: Adding to iommu group 8
[ 0.611500] pci 0000:00:07.1: Adding to iommu group 9
[ 0.611520] pci 0000:00:08.0: Adding to iommu group 10
[ 0.611536] pci 0000:00:08.1: Adding to iommu group 11
[ 0.611560] pci 0000:00:14.0: Adding to iommu group 12
[ 0.611574] pci 0000:00:14.3: Adding to iommu group 12
[ 0.611635] pci 0000:00:18.0: Adding to iommu group 13
[ 0.611649] pci 0000:00:18.1: Adding to iommu group 13
[ 0.611662] pci 0000:00:18.2: Adding to iommu group 13
[ 0.611676] pci 0000:00:18.3: Adding to iommu group 13
[ 0.611690] pci 0000:00:18.4: Adding to iommu group 13
[ 0.611705] pci 0000:00:18.5: Adding to iommu group 13
[ 0.611719] pci 0000:00:18.6: Adding to iommu group 13
[ 0.611733] pci 0000:00:18.7: Adding to iommu group 13
[ 0.611752] pci 0000:01:00.0: Adding to iommu group 14
[ 0.611769] pci 0000:01:00.1: Adding to iommu group 15
[ 0.611786] pci 0000:01:00.2: Adding to iommu group 16
[ 0.611803] pci 0000:02:00.0: Adding to iommu group 17
[ 0.611820] pci 0000:02:08.0: Adding to iommu group 18
[ 0.611837] pci 0000:02:09.0: Adding to iommu group 19
[ 0.611871] pci 0000:03:00.0: Adding to iommu group 20
[ 0.611892] pci 0000:03:00.1: Adding to iommu group 21
[ 0.611912] pci 0000:05:00.0: Adding to iommu group 22
[ 0.611934] pci 0000:06:00.0: Adding to iommu group 23
[ 0.611951] pci 0000:06:00.1: Adding to iommu group 24
[ 0.611966] pci 0000:07:00.0: Adding to iommu group 25
[ 0.611986] pci 0000:08:00.0: Adding to iommu group 26
[ 0.612005] pci 0000:08:00.1: Adding to iommu group 27
[ 0.612024] pci 0000:08:00.3: Adding to iommu group 28
[ 0.612043] pci 0000:08:00.4: Adding to iommu group 29
[ 1.612534] iommu ivhd0: AMD-Vi: Event logged [IOTLB_INV_TIMEOUT device=0000:03:00.0 address=0x1002348a0]
[ 1.612653] perf/amd_iommu: Detected AMD IOMMU #0 (2 banks, 4 counters/bank).

ll /dev/vfio/
totale 0
drwxr-xr-x 2 root root 60 apr 5 04:17 ./
drwxr-xr-x 20 root root 4040 apr 5 04:18 ../
crw-rw-rw- 1 root root 10, 196 apr 5 04:17 vfio

lsmod | grep vfio

vfio_pci_core 94208 0
vfio_iommu_type1 45056 0
vfio 61440 2 vfio_pci_core,vfio_iommu_type1
iommufd 131072 1 vfio

------------------------------------------------------------

Next step

Using a distribution with GRUB Ubuntu+Kernel Liquorix
clean installation
is to use the commands directly in GRUB
without Aliases for the GPU

waiting for your advice

@Balaton
@All

Vfio 0000:41:00.0: failed to open /dev/vfio/32: No such file or directory - QEMU

Here is a link that is related to the error:

If you take a look maybe it's the solution but I can't understand it

https://stackoverflow.com/questions/61 ... ch-file-or-directory-qemu

example:

You need bind your PCI device to the vfio-pci driver.

Unbind pci device from its previous driver

echo "0000:03:00.0" > /sys/devices/pci0000:00/0000:03:00.0/driver/unbind

Bind to vfio-pci driver

echo 1002 6798 > /sys/bus/pci/drivers/vfio-pci/new_id

Where 8086 is the vendor id of the PCI device, and a36d is device id of the PCI device <<<----- WHAT SHOULD I PUT THIS HERE ? (vendor id)

Check and vfio group should be present, and Qemu can run.

ls -l /dev/vfio/

Can someone give me an example for the syntax that would be useful for the Radeon GPU I'm using ?

probably I have to fix the creation of:
vfio-pci
/dev/vfio/20

or the correct procedure is the one mentioned above but I can't figure out how to bind the GPU with:
Bind to vfio-pci driver


In the meantime I'm going to make myself some spaghetti with clams, garlic and parsley
And I'll think about it a bit


HELP

@white
I think slowing a bit and thinking and knowing which command you do why helps. It's hard to guess what your problem is but I think you should verify if the card is still assigned to vfio-pci. I've posted the modprobe.d config file that would avoid needing unbind or driverctl but you did not try that. Here it is again:

put that in /etc/modprobe.d/vfio.conf then reboot and verify with lspci -v that the AMD GPU is using vfio-pci driver. If that works that's all, no need to manually unbind, rebind or use driverctl when you make sure that the correct driver is bound to the card and not try to fix up later.

@Balaton

I'm following your advice.
But the question is:
Since I installed the Liquorix kernel
do I lose the settings made previously in:
/etc/modprobe.d/vfio.conf

or in:
/etc/modprobe.d/local.conf

Because until the update the command returned:

lspci -k | grep -E "vfio-pci|radeon"

Kernel modules: radeon, amdgpu
Kernel driver in use: vfio-pci

And to know otherwise I update the kernel first and then I do everything else.

I installed "Arch" and I will follow this guide:

https://www.youtube.com/watch?v=uOuzFd8Gd2o

I will try with only the Radeon GPU.

and I will also swap the slots of the two GPUs to see if it is better.

@joerg

"Thank you for the information, Joerg, much appreciated. I've missed a few years of developments due to family and work, and your input has been very thorough and has filled all the gaps. Thanks again!"

I can't install the latest version of qemu with "Arch" the "10.0.0-rc3"
I would need to understand what dependencies are needed
for example pacman -S meson also installs "ninja build"
and this once the "make install" is done it stops at the "1500" line

with:
./configure --target-list=ppc-softmmu --enable-lto --enable-sdl --enable-gtk

clearly if I do pacman -S qemu install version 9.2.3
but I would like to know if anyone knows the dependencies needed in "Arch" for the compilation of "qemu beta"
So as to have the possibility to compile other beta versions in the future.


Thanks.

@white
How do we know without you telling what error you get? But if QEMU compiled you don't have to install it, you can run it from build dir: cd build; ./qemu-system-ppc...

@Balaton

Thanks

Here where it stops:

[1497/2698] Compiling C object libblockdev.a.p/block_export_fuse.c.o
FAILED: libblockdev.a.p/block_export_fuse.c.o
cc -m64 -Ilibblockdev.a.p -I. -I.. -Isubprojects/libvduse -I../subprojects/libvduse -Iqapi -Itrace -Iui -Iui/shader -I/usr/include/fuse3 -I/usr/include/glib-2.0 -I/usr/lib/glib-2.0/include -I/usr/include/sysprof-6 -flto=auto -fdiagnostics-color=auto -Wall -Winvalid-pch -Werror -std=gnu11 -O2 -g -fstack-protector-strong -Wempty-body -Wendif-labels -Wexpansion-to-defined -Wformat-security -Wformat-y2k -Wignored-qualifiers -Wimplicit-fallthrough=2 -Winit-self -Wmissing-format-attribute -Wmissing-prototypes -Wnested-externs -Wold-style-declaration -Wold-style-definition -Wredundant-decls -Wshadow=local -Wstrict-prototypes -Wtype-limits -Wundef -Wvla -Wwrite-strings -Wno-missing-include-dirs -Wno-psabi -Wno-shift-negative-value -isystem /home/white/qemu/linux-headers -isystem linux-headers -iquote . -iquote /home/white/qemu -iquote /home/white/qemu/include -iquote /home/white/qemu/host/include/x86_64 -iquote /home/white/qemu/host/include/generic -iquote /home/white/qemu/tcg/i386 -pthread -mcx16 -msse2 -D_GNU_SOURCE -D_FILE_OFFSET_BITS=64 -D_LARGEFILE_SOURCE -fno-strict-aliasing -fno-common -fwrapv -ftrivial-auto-var-init=zero -fzero-call-used-regs=used-gpr -fPIE -MD -MQ libblockdev.a.p/block_export_fuse.c.o -MF libblockdev.a.p/block_export_fuse.c.o.d -o libblockdev.a.p/block_export_fuse.c.o -c ../block/export/fuse.c
In file included from ../block/export/fuse.c:33:
/usr/include/fuse3/fuse.h:959:5: error: redundant redeclaration of ‘fuse_main_real_versioned’ [-Werror=redundant-decls]
959 | int fuse_main_real_versioned(int argc, char *argv[],
| ^~~~~~~~~~~~~~~~~~~~~~~~
/usr/include/fuse3/fuse.h:885:5: note: previous declaration of ‘fuse_main_real_versioned’ with type ‘int(int, char **, const struct fuse_operations *, size_t, struct libfuse_version *, void *)’ {aka ‘int(int, char **, const struct fuse_operations *, long unsigned int, struct libfuse_version *, void *)’}
885 | int fuse_main_real_versioned(int argc, char *argv[],
| ^~~~~~~~~~~~~~~~~~~~~~~~
cc1: all warnings being treated as errors
ninja: build stopped: subcommand failed.
make[1]: *** [Makefile:168: run-ninja] Error 1
make[1]: uscita dalla directory «/home/white/qemu/build»
make: *** [GNUmakefile:6: build] Error 2


Note:
Generally in Ubuntu and derivatives there are only small changes in the requirements for compilation.
But in the end it compiles easily.

---------------------------------------------------

I tried this by intuition:
pacman -Ss libblockdev
then I tried:
sudo pacman -S libblockdev-crypto libblockdev-fs libblockdev-loop libblockdev-mdraid libblockdev-nvme libblockdev-part libblockdev-swap udisks2

but it doesn't seem to work

@white
Your libfuse installation seems to be broken, I don't know how could configure detect it but then it does not work. Either install the appropriate devel package for fuse or add --disable-fuse to configure as fuse is not needed anyway.

@Balaton
Thanks for your suggestion it is right and it works.
Now it was important to create the machine that works
Tonight under cover of darkness we start working on it for the vfio PCI

./configure --target-list=ppc-softmmu --enable-lto --enable-sdl --enable-gtk --disable-fuse





Have a nice day

@Balaton

I reversed the GPUs in the PCIe slots
With Arch at the moment these are the results:

./test-vfio.sh
result:
[3.834645] VFIO - User Level meta-driver version: 0.3

./test-iommu.sh
result with ACS active the card is isolated in group 15:

IOMMU Group 15:
06:00.0 VGA compatible controller [0300]: Advanced Micro Devices, Inc. [AMD/ATI] Tahiti XT [Radeon HD 7970/8970 OEM / R9 280X] [1002:6798]
06:00.1 Audio device [0403]: Advanced Micro Devices, Inc. [AMD/ATI] Tahiti HDMI Audio [Radeon HD 7870 XT / 7950/7970] [1002:aaa0]

The time-out has disappeared:
sudo dmesg | grep -i -e DMAR -e IOMMU

[ 0.655585] iommu: Default domain type: Translated
[ 0.655587] iommu: DMA domain TLB invalidation policy: lazy mode
[ 0.684690] pci 0000:00:00.2: AMD-Vi: IOMMU performance counters supported
[ 0.684736] pci 0000:00:00.0: Adding to iommu group 0
[ 0.684754] pci 0000:00:01.0: Adding to iommu group 1
[ 0.684767] pci 0000:00:01.2: Adding to iommu group 2
[ 0.684784] pci 0000:00:02.0: Adding to iommu group 3
[ 0.684802] pci 0000:00:03.0: Adding to iommu group 4
[ 0.684814] pci 0000:00:03.1: Adding to iommu group 5
[ 0.684830] pci 0000:00:04.0: Adding to iommu group 6
[ 0.684847] pci 0000:00:05.0: Adding to iommu group 7
[ 0.684865] pci 0000:00:07.0: Adding to iommu group 8
[ 0.684877] pci 0000:00:07.1: Adding to iommu group 9
[ 0.684894] pci 0000:00:08.0: Adding to iommu group 10
[ 0.684907] pci 0000:00:08.1: Adding to iommu group 11
[ 0.684930] pci 0000:00:14.0: Adding to iommu group 12
[ 0.684941] pci 0000:00:14.3: Adding to iommu group 12
[ 0.684998] pci 0000:00:18.0: Adding to iommu group 13
[ 0.685010] pci 0000:00:18.1: Adding to iommu group 13
[ 0.685022] pci 0000:00:18.2: Adding to iommu group 13
[ 0.685033] pci 0000:00:18.3: Adding to iommu group 13
[ 0.685045] pci 0000:00:18.4: Adding to iommu group 13
[ 0.685057] pci 0000:00:18.5: Adding to iommu group 13
[ 0.685069] pci 0000:00:18.6: Adding to iommu group 13
[ 0.685081] pci 0000:00:18.7: Adding to iommu group 13
[ 0.685109] pci 0000:01:00.0: Adding to iommu group 14
[ 0.685123] pci 0000:01:00.1: Adding to iommu group 14
[ 0.685137] pci 0000:01:00.2: Adding to iommu group 14
[ 0.685141] pci 0000:02:00.0: Adding to iommu group 14
[ 0.685146] pci 0000:02:08.0: Adding to iommu group 14
[ 0.685150] pci 0000:02:09.0: Adding to iommu group 14
[ 0.685154] pci 0000:03:00.0: Adding to iommu group 14
[ 0.685159] pci 0000:03:00.1: Adding to iommu group 14
[ 0.685163] pci 0000:05:00.0: Adding to iommu group 14
[ 0.685187] pci 0000:06:00.0: Adding to iommu group 15
[ 0.685203] pci 0000:06:00.1: Adding to iommu group 15
[ 0.685215] pci 0000:07:00.0: Adding to iommu group 16
[ 0.685229] pci 0000:08:00.0: Adding to iommu group 17
[ 0.685242] pci 0000:08:00.1: Adding to iommu group 18
[ 0.685255] pci 0000:08:00.3: Adding to iommu group 19
[ 0.685268] pci 0000:08:00.4: Adding to iommu group 20
[ 0.688112] perf/amd_iommu: Detected AMD IOMMU #0 (2 banks, 4 counters/bank).

Finally with:

find /sys/kernel/iommu_groups/ -type l | sort -n -t/ -k5

result:

/sys/kernel/iommu_groups/0/devices/0000:00:00.0
/sys/kernel/iommu_groups/1/devices/0000:00:01.0
/sys/kernel/iommu_groups/2/devices/0000:00:01.2
/sys/kernel/iommu_groups/3/devices/0000:00:02.0
/sys/kernel/iommu_groups/4/devices/0000:00:03.0
/sys/kernel/iommu_groups/5/devices/0000:00:03.1
/sys/kernel/iommu_groups/6/devices/0000:00:04.0
/sys/kernel/iommu_groups/7/devices/0000:00:05.0
/sys/kernel/iommu_groups/8/devices/0000:00:07.0
/sys/kernel/iommu_groups/9/devices/0000:00:07.1
/sys/kernel/iommu_groups/10/devices/0000:00:08.0
/sys/kernel/iommu_groups/11/devices/0000:00:08.1
/sys/kernel/iommu_groups/12/devices/0000:00:14.0
/sys/kernel/iommu_groups/12/devices/0000:00:14.3
/sys/kernel/iommu_groups/13/devices/0000:00:18.0
/sys/kernel/iommu_groups/13/devices/0000:00:18.1
/sys/kernel/iommu_groups/13/devices/0000:00:18.2
/sys/kernel/iommu_groups/13/devices/0000:00:18.3
/sys/kernel/iommu_groups/13/devices/0000:00:18.4
/sys/kernel/iommu_groups/13/devices/0000:00:18.5
/sys/kernel/iommu_groups/13/devices/0000:00:18.6
/sys/kernel/iommu_groups/13/devices/0000:00:18.7
/sys/kernel/iommu_groups/14/devices/0000:01:00.0
/sys/kernel/iommu_groups/14/devices/0000:01:00.1
/sys/kernel/iommu_groups/14/devices/0000:01:00.2
/sys/kernel/iommu_groups/14/devices/0000:02:00.0
/sys/kernel/iommu_groups/14/devices/0000:02:08.0
/sys/kernel/iommu_groups/14/devices/0000:02:09.0
/sys/kernel/iommu_groups/14/devices/0000:03:00.0
/sys/kernel/iommu_groups/14/devices/0000:03:00.1
/sys/kernel/iommu_groups/14/devices/0000:05:00.0
/sys/kernel/iommu_groups/15/devices/0000:06:00.0
/sys/kernel/iommu_groups/15/devices/0000:06:00.1
/sys/kernel/iommu_groups/16/devices/0000:07:00.0
/sys/kernel/iommu_groups/17/devices/0000:08:00.0
/sys/kernel/iommu_groups/18/devices/0000:08:00.1
/sys/kernel/iommu_groups/19/devices/0000:08:00.3
/sys/kernel/iommu_groups/20/devices/0000:08:00.4

note:

I guess something has changed
but the result should be something like:

./test-vfio.sh
result:
[3.834645] VFIO - User Level meta-driver version: 0.3
----------------------------------------------------------------

I guess something changed
but the result should be something like:

vfio-pci enabled etc. etc.

if you need more information let me know. Thanks.

qesudo qemu-system-ppc -M pegasos2 -bios /home/white/Scaricati/pegasos2.rom -m 1024 -serial stdio -device VGA,romfile="" -drive if=none,id=hd,file=/home/white/Scaricati/32gb.raw,format=raw -device ide-hd,drive=hd,bus=ide.1 -netdev user,id=tap0 -device rtl8139,netdev=tap0 -vga none -drive if=none,id=hd1,file=/home/white/Scaricati/coffin.raw,format=raw -device ide-hd,drive=hd1,bus=ide.1 -device vfio-pci,host=06:00.0,bus=pci.0 -device vfio-pci.host=06:00.1,bus=pci.0 -accel tcg
mu:

result:
qemu-system-ppc: -device vfio-pci,host=06:00.0,bus=pci.0: vfio 0000:06:00.0: Could not open '/dev/vfio/15': No such file or directory

GRUB:
GRUB_CMDLINE_LINUX_DEFAULT="rd.driver.pre-vfio-pci amd_iommu=on iommu=pt video=efifb:off loglevel=3 quiet"

VFIO.conf:
options vfio-pci ids=1002:6798,1002:aaa0
softdep drm pre: vfio-pci
softdep snd_hda_intel pre: vfio-pci
blacklist radeon
blacklist amdgpu

/etc/mkinitcpio.conf

MODULES=(vfio_pci vfio vfio_iommu_type1)
HOOKS=(base udev autodetect microcode modconf kms keyboard keymap consolefont block filesystems fsck)

sudo mkinitcpio -p linux

==> Building image from preset: /etc/mkinitcpio.d/linux.preset: 'default'
==> Using default configuration file: '/etc/mkinitcpio.conf'
-> -k /boot/vmlinuz-linux -g /boot/initramfs-linux.img
==> Starting build: '6.14.2-arch1-1'
-> Running build hook: [base]
-> Running build hook: [udev]
-> Running build hook: [autodetect]
-> Running build hook: [microcode]
-> Running build hook: [modconf]
-> Running build hook: [kms]
-> Running build hook: [keyboard]
-> Running build hook: [keymap]
-> Running build hook: [consolefont]
==> WARNING: consolefont: no font found in configuration
-> Running build hook: [block]
-> Running build hook: [filesystems]
-> Running build hook: [fsck]
==> Generating module dependencies
==> Creating zstd-compressed initcpio image: '/boot/initramfs-linux.img'
-> Early uncompressed CPIO image generation successful
==> Initcpio image generation successful
==> Building image from preset: /etc/mkinitcpio.d/linux.preset: 'fallback'
==> Using default configuration file: '/etc/mkinitcpio.conf'
-> -k /boot/vmlinuz-linux -g /boot/initramfs-linux-fallback.img -S autodetect
==> Starting build: '6.14.2-arch1-1'
-> Running build hook: [base]
-> Running build hook: [udev]
-> Running build hook: [microcode]
-> Running build hook: [modconf]
-> Running build hook: [kms]
==> WARNING: Possibly missing firmware for module: 'ast'
-> Running build hook: [keyboard]
==> WARNING: Possibly missing firmware for module: 'xhci_pci_renesas'
-> Running build hook: [keymap]
-> Running build hook: [consolefont]
==> WARNING: consolefont: no font found in configuration
-> Running build hook: [block]
==> WARNING: Possibly missing firmware for module: 'aic94xx'
==> WARNING: Possibly missing firmware for module: 'bfa'
==> WARNING: Possibly missing firmware for module: 'qed'
==> WARNING: Possibly missing firmware for module: 'qla1280'
==> WARNING: Possibly missing firmware for module: 'qla2xxx'
==> WARNING: Possibly missing firmware for module: 'wd719x'
-> Running build hook: [filesystems]
-> Running build hook: [fsck]
==> Generating module dependencies
==> Creating zstd-compressed initcpio image: '/boot/initramfs-linux-fallback.img'
-> Early uncompressed CPIO image generation successful
==> Initcpio image generation successful

----------------------------------------------------------------------
Ok I reviewed the /etc/mkinitcpio.conf passage:

here is the result

./test-vfio.sh

[sudo] password di white:
[ 3.826200] VFIO - User Level meta-driver version: 0.3
[ 3.865513] vfio-pci 0000:06:00.0: vgaarb: deactivate vga console
[ 3.865518] vfio-pci 0000:06:00.0: vgaarb: VGA decodes changed: olddecodes=io+mem,decodes=io+mem:owns=io+mem
[ 3.865951] vfio_pci: add [1002:6798[ffffffff:ffffffff]] class 0x000000/00000000
[ 3.913622] vfio_pci: add [1002:aaa0[ffffffff:ffffffff]] class 0x000000/00000000
[white@Amiga Scaricati]$

@Balaton

Now qemu starts loading but the black screen remains

Or sometimes from this error:

CS:IP = 0000C000:000039C7 WRITE ACCESS TO UNEMULATED IOPORT=00000000 DATA=00006E70
CS:IP = 0000C000:000039CB READ ACCESS FROM UNEMULATED IOPORT=00000004
CS:IP = 0000C000:00003760 READ ACCESS FROM UNEMULATED IOPORT=0000004C

Is it useful to use the FHT file?
Thanks.

Note:
I am currently using version 9.2.3 and not the latest beta "10 RC3" now I compile it.

@white
Quote:
This command is unnecessarily complicated and also wrong. You're adding here a VGA card next to the passed through card so when the BIOS tries to access the VGA resources of the passed through card it gets to the emulated VGA not the ATI card. Therefore it can't init the ATI card and you get no picture. Try this simpler command instead that should do the same:

@Balaton
I changed the syntax:
sudo qemu-system-ppc -M pegasos2 -m 1024 -bios /home/white/Downloaded/pegasos2.rom -serial stdio -drive media=disk,format=raw,file=/home/white/Downloaded/32gb.raw -drive media=disk,format=raw,file=/home/white/Downloaded/coffin.raw -netdev user,id=net -device rtl8139,netdev=net -vga none -device vfio-pci,host=06:00.0,bus=pci.0,x-vga=on,multifunction=on -device vfio-pci,host=06:00.1,bus=pci.0

but it never starts
the "qemu monitor console" window opens

and it tells me this:

PegasosII Boot Strap (c) 2002-2003 bplan GmbH
Running on CPU PVR:80020102
Enable L1 ICache... Done.
Clean/Flush Block enabled
Reading W83194 : FAILED.
Setting Front Side Bus to 133MHz... FAILED.
Configuring DDR... Done.
Configuring PCI0... Done.
Configuring PCI1... Done.
Configuring ETH... Done.
Releasing IDE reset ... Done.
Configuring Legacy Devices
Initializing KBD... Done.
Testing 10000000 Bytes, Pass: 00000000 Failed: 00000000
RAM TEST (fill linear)... Done.
FFFFFFFF

SmartFirmware:
cpu0: PowerPC,74x7 CPUClock 1533 Mhz BUSClock 133 Mhz (Version 0x8002,0x0102)
no/bad nvramrc - performing default startup script
channel 0 unit 0 : ata | QEMU HARDDISK | 2.5+
channel 0 unit 1 : ata | QEMU HARDDISK | 2.5+
channel 1 unit 0 : atapi | QEMU DVD-ROM | 2.5+
ATA device not present or not responding
INTERNAL ERROR: 0000000E=UNIMPLEMENTED EXTENDED OPCODE

EAX=0000 EBX=0000 ECX=0000 EDX=0000 ESP=0000 EBP=5555 ESI=6666 EDI=7777
AX=0008 BX=0004 CX=3A98 DX=0014 SP=FFC8 BP=FFB6 SI=0008 DI=A298
DS=C000 ES=BAD0 SS=1000 CS=C000 IP=3288 NV UP -- PL NZ NA PO NC
CS:IP = 0F
STACK: 0008 0008 15B9 0004 162B 07F2 02B7 0000
00: FE00 F000 FE01 F000 FE02 F000 FE03 F000
10: FE04 F000 FE05 F000 FE06 F000 FE07 F000
20: FE08 F000 FE09 F000 FE0A F000 FE0B F000
30: FE0C F000 FE0D F000 FE0E F000 FE0F F000
40: FE10 F000 FE11 F000 FE12 F000 FE13 F000
50: FE14 F000 FE15 F000 FE16 F000 FE17 F000
60: FE18 F000 FE19 F000 FE1A F000 FE1B F000
70: FE1C F000 FE1D F000 FE1E F000 FE1F F000
80: FE20 F000 FE21 F000 FE22 F000 FE23 F000
90: FE24 F000 FE25 F000 FE26 F000 FE27 F000
A0: FE28 F000 FE29 F000 FE2A F000 FE2B F000
B0: FE2C F000 FE2D F000 FE2E F000 FE2F F000
C0: FE30 F000 FE31 F000 FE32 F000 FE33 F000
D0: FE34 F000 FE35 F000 FE36 F000 FE37 F000
E0: FE38 F000 FE39 F000 FE3A F000 FE3B F000
F0: FE3C F000 FE3D F000 FE3E F000 FE3F F000
00: FE40 F000 FE41 F000 FE42 F000 FE43 F000
10: FE44 F000 FE45 F000 FE46 F000 FE47 F000
20: FE48 F000 FE49 F000 FE4A F000 FE4B F000
30: FE4C F000 FE4D F000 FE4E F000 FE4F F000
Failed to emulate CS:IP [C000:3288]=0F,BC,D8,8D,36,95
UNHANDLED INT 10 FUNCTION 0100 WITHIN EMULATION
EA: BYTE READ FROM UNINITIALIZED LOW MEM 0040:0085
UNHANDLED INT 10 FUNCTION 0300 WITHIN EMULATION
UNHANDLED INT 10 FUNCTION 1301 WITHIN EMULATION
UNHANDLED INT 10 FUNCTION 0300 WITHIN EMULATION
UNHANDLED INT 10 FUNCTION 1301 WITHIN EMULATION
UNHANDLED INT 10 FUNCTION 0300 WITHIN EMULATION
UNHANDLED INT 10 FUNCTION 1301 WITHIN EMULATION
UNHANDLED INT 10 FUNCTION 0300 WITHIN EMULATION
UNHANDLED INT 10 FUNCTION 1301 WITHIN EMULATION
UNHANDLED INT 10 FUNCTION 0300 WITHIN EMULATION
UNHANDLED INT 10 FUNCTION 1301 WITHIN EMULATION
UNHANDLED INT 10 FUNCTION 0300 WITHIN EMULATION
UNHANDLED INT 10 FUNCTION 1301 WITHIN EMULATION
UNHANDLED INT 10 FUNCTION 0300 WITHIN EMULATION
UNHANDLED INT 10 FUNCTION 1301 WITHIN EMULATION
UNHANDLED INT 10 FUNCTION 0300 WITHIN EMULATION
UNHANDLED INT 10 FUNCTION 1301 WITHIN EMULATION
entering main read/eval loop...
UNHANDLED INT 10 FUNCTION 0300 WITHIN EMULATION
UNHANDLED INT 10 FUNCTION 1301 WITHIN EMULATION
UNHANDLED INT 10 FUNCTION 0300 WITHIN EMULATION
UNHANDLED INT 10 FUNCTION 1301 WITHIN EMULATION
UNHANDLED INT 10 FUNCTION 0300 WITHIN EMULATION
UNHANDLED INT 10 FUNCTION 1301 WITHIN EMULATION
UNHANDLED INT 10 FUNCTION 0300 WITHIN EMULATION
UNHANDLED INT 10 FUNCTION 1301 WITHIN EMULATION
UNHANDLED INT 10 FUNCTION 0300 WITHIN EMULATION
UNHANDLED INT 10 FUNCTION 1301 WITHIN EMULATION
UNHANDLED INT 10 FUNCTION 0300 WITHIN EMULATION
UNHANDLED INT 10 FUNCTION 1301 WITHIN EMULATION
UNHANDLED INT 10 FUNCTION 0300 WITHIN EMULATION
UNHANDLED INT 10 FUNCTION 1301 WITHIN EMULATION
entering main read/eval loop...
UNHANDLED INT 10 FUNCTION 0300 WITHIN EMULATION
UNHANDLED INT 10 FUNCTION 1301 WITHIN EMULATION
qemu: Slirp: Failed to send packet, ret: -1


it is better if I use beta 10 ?

Thanks