As I said, I'm not much of a game player.
For XYZ I was thinking of actually having to lift the controller off
the table for the Y direction, and forward-back for Z. Perhaps for a
game a glove would work better than a visor for what I was thinking
about.
If USB is used for this more than 1 port would be required, however
Xmos can be programmed to use several lines in parallel, (and then
easily modified for some other purpose) and thus avoid the conversions
from serial to parallel to get a full binary word to - - wherever it
needs to go. Since the inputs are talking to a CPU even further
processing could be done on-chip before presenting to data to the rest
of the system.