@joerg
OK so those would be something like generalised protection bits in the MMU. Since there's no cache in the emulated PPC most of these don't make sense so they likely do nothing with endian being the only useful one but since one can use ppc64le guests I think those are supported. (But does that even exist on G4 and before we're using here? I think those are big endian only anyway.) I think you meant these should be handled on the host not in emulation. But since vfio pass thorugh is in QEMU for a while and works for other machines including PPC64 machines emulated by QEMU I think problems with the basics in that is very unlikely. I'd expect problems within pegasos2 emulation (the parts I've written) which is just mv64361, vt8231 and the board code connecting these together. These are not fully emulated and there could be errors in there. In this case it's likely something with raising or acking interrrupts or tracking their state between multiple interrupts sources. It's hard to figure out though because this is scattered between different parts of code and I also don't know how AmigaOS usees this so what to check.

How does AmigaOS ack interrupts on pegasos2? Does it use the MV64361 regs for that or the ISA PIC regs? Maybe this is somewhere in the kernel and likely only who ported it to pegasos2 knows.

@joerg
QEMU has softmmu to emulate MMU/TLB/SR, don't know what WIMGE is nor what all this has to do with caches.

There can be a reason for sm502 driver limitations but that does not change the end result that it's limited. I don't understand what you refer to but I think the separate alpha is only used for blending output with a video layer like genlock/color keying and in that mode sm501 may be limited to 16 bit. But for just display output it does support 32 bit RGBA mode (with the A part ignored) up to 1280x1024 so what prevents the sm502 driver to support that without compositing as it does in 16 bit mode?

@MartinW
There's no specific reason to try pass through I think just we were curious if it worked and how usable that would be. If there's any reason then that's the limited sm502 driver that made us look for alternatives and since reaching 3D with software (either emulating a 3D capable card or writing virtio drivers) may take a long time using pass through could be the shortest path to that now.

There's probably no reason to do any of this and it is just fun playing with these. Also nobody is pushing anobody to do what they don't want to do, I hope everybody just does what they find fun to do and is best at doing but if we put that together something useful may come out of it so we can have more fun with it in the future.

@MartinW
I never used vfio but read that problems with starting guests more than once can happen if cards are not reset properly and that's why there are quirks in Linux kernel sources for this. Maybe you should check host logs for clues and check what the kernel driver sources have about your card.

If this testing revealed bugs in the driver then debugging and fixing that would make it better for real hardware or in general so even if that would not fix the issues with emulation I think it's not wasted time.

Interrupts on pegasos2 are edge sensitive by default unless something changes that in PIC. MorphOS changes that using an old way that wasn't emulated but was fixed in QEMU 8.0 but I don't think other guests change it. One could check with info PIC and looking up the register docs for PIC to see what values mean.

What I was trying to find out yesterday is what's missing from QEMU virtual open firmware for AmigaOS to boot. My boot loader would mostly be ready if that worked but I could not find out what it's missing. I can boot AmigaOS with it using the pegasos2.rom but with VOF AmigaOS does not start but crashes around the time it tries to access PCI devices. Not sure it it's still in the loader or in openfirmware.resource within the kernel but since it works with original firmware I think it's not something missing from the boot loader but something missing from the device tree under VOF. It does instatiate rtas and VOF has that as Linux also uses it and it works with that but AmigaOS does not seem to call the rtas yet but crashes between instantiating it and using it for PCI access. Does anybody know what it may do in that part and what it might need from the device tree at that point?

When using pegaoss2.rom I get:


when using VOF:

Both using my boot loader. It seems to get into a write loop or maybe pegasos2 firmware does something about setting up some MMU mappings on claim maybe? I'm stuck at this point at the moment.

Additionally if I break at the address giving me invalid 0 opcode at 0x0182c350 there seems to be code there at first but maybe something overwrites it so could be this is around when init-ing memory and uses some missing value that should come from the device tree but what?

The whole interrupt routing on pegasos2 is quote complex so I'm not sure I even understand it fully and it's quite possible we don't emulate that correctly somewhere. AFAIU the CPU interrupt input is connected to the MV64361 which has 2 independent PCI buses that each can have 4 IRQ pins but for the AGP port only 2 are connected. These PCI interrupts are connected to the GPP 12-15 pins of the MV64361 which are general purpose inputs that can be programmed to generate an interrupt. This should be emulated unless I've made a mistake writing that. However there is also the VT8231 south bridge that has a completely different IRQ routing as it's a PC south bridge containing usual PC hardware for ISA including 2 ISA PICs and peripherals routed there. The VT8231 also has 4 inputs for PCI interrupts and the interrupts lines for both buses area also connected here. These inputs can be routed to ISA interrupts within VT8231 with the other internal peripherals individually routable as well. The pegasos2.rom firmware routes everything to IRQ 9 though and guests don't change this so we also only emulate that. The VT8231 is also a PCI device itself and is connected to the pci.1 bus of the MV64361 so when an ISA interrupt is raised VT8231 raises its PCI interrupt EDIT: the VT8231 IRQ output is connected to GPP 31 of MV64361 instead, this is also correclty emulated, which is noticed by MV64361 and raises the CPU interrupt. That part probably works but then the interrupt should be also cleared at some point.

Clearing (aka ack'ing) interrupts is another mess. One can poke the PIC regs directly but there are also some virtual ack registers of MV64361 that can be read or written to ack an ISA interrupt within the VT8231 which I'm not sure is used but checking it now it's only implemented for pci.1 now and maybe it's not even correct in all cases. Since I don't know what AmigaOS and its drivers do I don't know what to check so some info on how interrupt handling is done and which way to ack interrupts is used by AmigaOS might help to debug and find where an interrupt could be missed. (If all the above works which I'm not sure it does yet then there's still a possibility that due to sharing interrupts there's some mixup happening when multiple sources change the same interrupt but I also don't know how to debug that. Problem with debugging these is one can't just add logs as there'll be millions of thees so it would be hard to find anything in there unless one knows what to look for in the first place. So to get further with this better understanding is needed what is happening and where could it fail then we can try to check that theory.

On MorphOS the network does not work from the start but it was found that disabling sound makes it work and starting sound after the network is up keeps it working so it's probably really an issue with sound emulation there that prevents network starting so maybe not related to interrupts. MorphOS uses the interrupt controllers differently so maybe it has different problems so testing with one OS may not give meaningful results for the other as these work differently.

@kas1e: Are you saying the network problem seen on AmigaOS (hang after lot of network traffic) can be reproduced with Linux guest as well? I've missed that part. Is there a way to easily reproduce this issue? Maybe it's easier to debug with Linux as we can get debug logs from the driver there or add more debug if needed.

The network related problems under AmigaOS and MorphOS probably have different causes. Under AmigaOS it's likely related to the IRQ issue which is just triggered by network traffic generating lot of interrupts but the same could be caused by other interrupts sources but network seems to be one of the biggest sources of interrupts.

@MartinW or others testing patches: Can you please check if info irq in QEMU monitor shows any difference with these patches after about the same things done in guest like booting and playing the startup sound and maybe downloading a file or so. If it hangs before that then when it hangs. Go to QEMU monitor and do info irq a few times and see if the numbers are changing. Also note what graphics card is used and with or without irq. Maybe that could show something about the IRQ source but since everything is mapped to IRQ 9 it probably won't but at least we could see how much interrupts are generated in each case.

@Hans, @MartinW

So there might be a problem with at least RX when used with shared interrupts that's the case on Pegasos2 but probably nobody tried this on real hardware yet. Or did @kas1e test that already? I've lost track. This might be fixable in the driver and until then one can disable the interrupts for the gfx card as a work around. Correct?

However does the same IRQ related hang happen with old Radeon or HD cards passed through? If those drivers don't have this problem and should work with shared interrupts then that still confirms we have some problems with interrupts in emulation. But I still don't understand what exactly and what causes it.

QEMU does not emulate caches so there should be no issues with cache coherency or such. Unless the driver tries to read data too early before the card finished writing it I don't think that could be a problem. It's more likely that interrupts are either missed or cleared by acknowledging other interrupts while other devices still should hold it raised. In current master everything is mapped to the 4 PCI interrupt lines and code in PCI emulation should track their state and keep everthing as it should but I did not write that code so don't know how it works however as it's generic PCI emulation used by all other machines it should not have serious bugs but maybe in pegasos2 we use it wrong. My original patches on OSDN try to track each interrupt source separately and combine them to the one interrupt so this should not have the above problems but maybe not all sources go through this so it could still miss some changes. The one patch work around was originally for MorphOS which expects level sensitive interrupts so that tries to convert level triggered interrupts to edge triggered by generating an edge when interrupts are still held high after a source is cleared. This may have a side effect of refreshing some state somewhere that helps on AmigaOS which supposed to handle edge sensitive interrupts. So either something works differently on real hardware than we expect or there's some problem emulating it correctly. (I'll continue in different post as this is getting too long.)

@MartinW
Quote:
After "git checkout pegasos2" "git status" should say On branch pegasos2 not detached head but if in "git log" you see the 3 patches (1 + 2 reverts) on top then that's probably OK just not sure what you did then.

Quote:
Then did this patch do anything at all? Does disabling GPU interrupts help with qemu master without any patches or some patch is still needed and it looks like the previous is better? This suggests I'll have to check how vfio handles interrupts as that may somehow bypass the IRQ routing these patches change but I don't see how.

Quote:
I'm just guessing but on real hardware IRQ may be better but on emulation an interrupt stops translated code, goes back to emulator and looks up a different translated block with the IRQ handler then after running that it goes back to QEMU again and does the same to go back to the code that was running whereas a jump or function call in translated code can jump to the next block directly so IRQs are kind of expensive on QEMU so a lot of them can be bad for performance.

Quote:
I know nothing about resizable BARs so can't comment on that but as said before the PCI bus memory and io spaces are mapped to system memory in windows so the CPU will only see BARs that are mapped in these windows. You can map card resources anywhere in PCI address space but unless they are in the windows the CPU will not see them. You can check this in info mtree which shows the PCI address spaces for both PCI buses and you should see there BARs that are mapped too and in the system memory or address space you see the windows with addresses that show what part of the pci regions are visible there. The memory windows are 1 to 1 mapped so for window at 0xc0000000 you should also map the bar to 0xc0000000 in PCI address space. The io bars start from 0 so an io BAR mapped at 0x1000 will be visible at window + 0x1000. Hope this makes sense but should be understandable looking at info mtree output. The only difficulty may be the lot of address spaces. There's one system address space, there are memory and io spaces for each PCI bus and QEMU also has an unused system io space that's just listed as i/o which is used on PC where there's a system io space but PPC is all memory mapped has no separate io bus. Also QEMU list all of these twice for some reason once as address-space and once as memory but I'm not sure what's the difference.

Quote:
I'm now not sure the BARs need to be mapped by the firmware because trying to find why my boot loader works with pegasos2.rom but not with VOF I first suspected that but now I've checked and while the pegasos2 firmware adds assigned-addresses property with addresses for sm501 it does not program the BARs and the AmigaOS driver still maps them. But this may be different for VGA cards that SmartFirmware wants to use for output so it will map them but not sure what happens if they aren't mapped. I think your script just patched assigned-addresses property and that was enough so I think the AmigaOS drivers would program the card and just need the assigned-addresses from the firmware but I'm still not sure. I've tried adding assigned-addresses to VOF but that did not help with my boot loader so maybe something else is missing from the device tree but that's unrelated to your gfx card BARs.

Quote:
Indeed, this is a fun way to learn a lot of stuff, that's why I also like to do it.

@kas1e
That bug report was network not working _under MorphOS_ so now that you've confirmed it works with AmigaOS can you also try booting the MorphOS demo iso and see if doing something like ifconfig or ping in shell works with that card? On QEMU this will only pop up a requester about needing bsdsockets.library but this may be because something is missing on QEMU which MorphOS expects (such as the on-board network ports) but I'm interested if not using the on-board ports but PCI network card works with MorphOS on real PegasosII.

The hangs we see under AmigaOS is due to a problem with handling multiple devices using the same interrupt in QEMU but that's likely different from the issue reported with MorphOS.

@MartinW
Maybe AmigaOS programs the BARs as long as there are addresses assigned in the device tree so it may be enough to set that and not actually map-in the BAR but I don't know. I may try that with my boot loader though as that would simplify it too if it only has to patch the device tree and not touch the cards.

@MartinW
I have no better idea than to check info mtree and info qtree or info pci in QEMU monitor and try to map the BAR in the appropriate pci1-mem or pci0-mem window. The machine and OS would only see the part of VRAM through the window so you could try to fit the most of it and the regs BAR in the window which is where using pci.8 may help becuase then you don't need to care about other PCI devices only the memory BARs of the gfx card. Info mtree shows what's mapped where and info pci shows what the BAR registers are set to. If they are ffffffff then it's not mapped and you had to write something there to map them. There's some way to do this from forth maybe using map-in but I don't know how that works so you have to search for docs.

@geennaam
Quote:

As for what the quirks do you'd need to check source and maybe some VGA docs but if you look at info metree outputs (like those posted by @derfs above) you can see that these overlap some of the io ports and the register BAR to fix up some issues. I did not look closer but comments suggests there are some ways in ATI cards to access io regs in memory BAR and that is not handled by vfio apparently with some help from QEMU but I'm not sure if it's because these are mapped at different address on the host or something else.

Anybody who has a real pegasos2 and a PCI network card (any PCI network card that has a driver may do) could please help to test if this problem reported here works on real hardware: https://osdn.net/projects/qmiga/ticket/48376
The real pegasos2 has two on board network ports which we don't emulate so maybe that has something to do with it but maybe also nobody tried with a PCI network card as there are already enough ports without that so could be this also happens on real machnine.

@MartinW
Quote:

I've ported my original series to current master and uploaded it in the pegasos2 branch here:
https://osdn.net/projects/qmiga/scm/git/qemu/summary
It's the 3 commits (2 reverts + 1 patch with the rest of the changes) on top of current master. If you cloning from that repo make sure to checkout the pegasos2 branch. (OSDN has been very slow for me recently at least the web, do you get the same? The git part still seems to work.) So this was my original approach which was then not accepted but simplified (maybe too much). So I'd like to know if this worked. Can you please test this removeing the current work around patch and using this instead? Try everything that caused a hang so far, including using pci.0 or more devices on pci.1 such as network vfio sound and usb (e.t. -device usb-mouse) together to see if it's stable. Please let me know which version is better, the current workaround or these 3 patches.
Edit: If you have problems getting the patches from the OSDN repo let me know and I can upload the mbox somewhere.

@white
Quote:

Of course you have to set DISPLAY in the remote shell not the local one. Normally DISPLAY is set to :0 and things just work if you start apps locally, If you ssh to your Linux host and want firefox running there to open window on AmiCygnix then you have to set DISPLAY accordingly in that shell where you start firefox. But you can't access 10.0.2.15 from there as that's behind the -netdrev user firewall/NAT so you do the same as you'd do in a router and port forward the 10.0.2.15:6000 port to localhost:6010 on your host with the hostfwd option. Therefore you have to connect there from the display variable which is DISPLAY=localhost:10. I don't know what's so hard to get on this, I think everybody reading this already understands and you did try everything except this one which should work. So just try what I'm sathing:
use -netdev user,id=mynet0,hostfwd=tcp::6010-:6000 then in the ssh shell before starting firefox do export DISPLAY=localhost:10. If you get problems with authentication either disable that in AmiCygnix or copy the MIT key from Linux to AmiCygnix and add it to both display :10 and display :0 and one of those should work. If it does not work with :10 try port 6009 and DISPLAY=localhost:9 as :10 may clash with ssh X forwarding but if you have that it should just work without any of the above.

@Hypex
Doing this from firmware should not be necessary as neither Linux nor MorphOS need the firmware to find and drive PCI devices so it's something that's missing from AmigaOS and thus needs to be patched up in the boot loader (the same way as support for 64 bit BARs but that may only be missing in the pegasos2 version because they did not expect PCIe cards used there). The QEMU built in VOF is very minimal and does not support most of OpenFirmware just enough to boot some boot loader to bring up the OS which then should handle stuff itself so it only provides the client interface, the device tree and runtime services (RTAS) but no full scripting or support for operating devices that OF can do.

There's no need to reset devices in the boot loader becuse that's done by QEMU and maybe we only need to care about display and ide to get them mapped. We also don't need to scan the PCI bus as the device tree is provided by VOF and the devices are listed there so the boot loader just needs to check the device tree for pcu devices and map the BARs for those that the OS needs but does not map itself (it can also fix up 64bit BARs at the same time). I have most of what I need for this but this needs to be implemented which takes some time to do.

It's not a question of configuration to emulate something else such as X1000 but heving device models for the componentes that machine has. QEMU can emulate PPC64 and the necessary ISA levels (BookE version) that may be needed but otherwise it may be difficult to get info on what other hardware that machine contains and likely those are not emulated so you'd have to write emulation for those before you can connect them the way they are connected in the machine (which may be handled in an undocumented FPGA so you may have a hard time finding that out). And what would that bring you anyway that you can't achieve with already existing machines? Could it run any more OS or sofrware? I don't think so so just improving existing machine emulation would make more sense than doing yet another emulation.

I don't think QEMU has any limitations on graphics and using other OSes you can use other resolutions and color depths but the AmigaOS sm502 driver seems to be limited partly by graphics.library partly by itself as far as I got from the comments here earlier so that could be solved by either improving ati-vga to by at least minimally usable or write a minimal driver to drive any of the QEMU emualted gfx cards at least as a simple frame buffer and that should solve this problem but this needs somebody who knows AmigaOS and can write a driver for it (or it takes me time to learn it which I don't have when still working on somethnig else).

I experimented some more with my boot loader and I think the problem is that AmigaOS does not seem to scan and map PCI devices itself but relies on the firmware to do that. Both Linux and MorphOS would do that themselves so they work with current virtual open firmware but for AmigaOS looks like I'll need to do that from the boot loader also. This is more work but if I do that patching 64bit bars at the same time will be easy so I had to do this anyway sometimes just hoped it would work without this but it seems it doesn't. The kernel probably starts but does not find the gfx card when it does not have mapped BARs. This might also be similar issue when a PCIe card is not mapped by the firmware so you may need to take care of that from the Forth sctipt or wait for me to finish this in the boot loader.

@MartinW
I also have no idea why it won't work on pci.0 as all PCI interrupts are tied together but the twp are different buses (these are created in hw/pci-host/mv64361.c that models the Marvell Discovery II system controller which has two PCI buses). So maybe the current way of keeping track of interrupts that relies on the irq counter in PCI fails. If you read the links in post #812 it was discussed there how we got to this solution but maybe it's not quite clear from all the emails. I wanted to test my original solution what I first posted but that does not apply any more so I had to port that to current master or maybe you can go back to the version before th ecurrent fix was committed and try to apply the original series on that, this was somewhere this January I think but if you can't do this easily then don't spend too much time with it as that series would need to be rewritten anyway so maybe I should do that first and only test that. But it's not likely to get that in 8.1 so either it will stay as it is or I can try to submit the work around you're using now as that may still be better tha nleaving this broken. If we found this eariler we could have try to rewrite this but that's now only possible after the freeze for 8.1 is over so I don't bother much but maybe if I'll have some time I'll try to make a patch to test and see if that can be submitted still.