@Mlehto
Unless you use the yet unreleased fixed kernel on a real PegasosII then the kernel up to and including 4.1FEUpdate2 has a known problem with configuring interrupts on PegasosII that BBoot does fix. This was found on QEMU but affects real PegasosII as well so using BBoot on that might help too (and it also boots faster). This can lead to lost interrupts if multiple devices use the same interrupt and that happens more on PegasosII due to how the firmware assigns interrupts. The
BBoot README has some info on how to use it on real PegasosII. Removing a card may only help by reducing number of interrupts so the problem happens less frequently but it's not really fixed. Only updated kernel or BBoot fixes that config issue.
On QEMU this problem is fixed using BBoot too but interrupts still don't seem to work with passed through graphics cards so that's a different and unrelated issue that is not fixed yet which likely does not affect real machines but specific to vfio-pci.