I have both RX580 (8GB) and R550 that I can use for my X5000 primary slot. Currently I'm on the 550. It seems plenty good enough and the 580 doesn't really gain me anything but just uses extra power.
The secondary card for MOS is a different matter entirely. I have two that are listed as compatible, an HD5450 and an R5 230. The 5450 doesn't work at all in the primary or the secondary slot. The R5 works in the primary slot but not in the x4 slot. I'm using a x4 -> x16 riser and have tried two types, neither makes any difference.
Sadly, there's no support on this. I posted to the MOS forum and there were a few responses initially but nothing of any use and then it filtered down the list of active topics and out of sight never to be seen again, so that's that. I'm not prepared to just keep throwing money at it until something works, but it would have been nice to replace the Powerbook laptop as that this is horrible to use.
I went ahead and ordered the RX550 to get the ball rolling. Seems the most logical for the X5K to me. Only $75 from Amazon and should be quite nice. I have plenty of PSU and larger than stock case so could have went 580 but unless there are real benefits, that I could not see from anyone's posts, why bother.
I will deal with the 2nd GPU later after I get the RX550 installed and all working well.
So, don't take the fact that my 5450 didn't work as none of them will work. The 5450 is listed as supported generally (under MOS) so it just seems to be bad luck that mine doesn't (the card works fine generally - it's now in my workbench PC).
I have every confidence it would have worked under linux on the X5K, but I have no desire to run linux on the machine. I simply don't see the point. I have a Mac Mini right next to it on the desk and access to plenty of far more powerful linux machines if I need it. The X5K is purely for things I can't run elsewhere - OS4 and I had rather hoped MOS.
I actually think the limit may be coming from the amount of memory on the second GPU, or the x4 slot. Not sure which it is but whenever I see success stories it's often with 256MB cards. I don't know if that's exclusively the case though so don't read too much into it. Obviously without support, I'm guessing
[EDIT] Maybe I spoke too soon on the benefits or not of the RX580. I just installed Astromenace and that could definitely use more than the 550. If I get time maybe I'll put the 580 in later and see what difference it makes. It's still a tough call unless you already have one as they're not exactly cheap. I already had one from a previous machine.
Well I may try the same thing. After plopping my order in for a 550, I did have 2nd thoughts and my huge Thermaltake View 71 was staring at me saying, "I have plenty of room for a bigger card and plenty of power as well, so splurge a little!" haha
I went ahead and ordered an MSI RX580. We shall see if I chose well or poorly. haha