Does anybody know what Linux versions worked on real pegasos2?

Are you sure IUtility->CallHook() doesn't include m68k support?
That would be very strange.

The removed amiga.lib/libamiga.a CallHook() didn't support emulated m68k code, but of course the utility.library one does.

It's most likely not related to varargs at all, but that only utility.library CallHookPkt() includes the required check and different way to execute the Hook function depending on if it's emulated m68k code or PPC native code.

Varargs throws out a lot of type-safety - you could pass pointers, floats, etc., instead of ints and it will compile without issue. Misuse of varargs, such as omitting arguments, can introduce odd crashes due to stack corruption, or reading invalid pointers.

The vararg mandates that the parameter be passed on the stack, thus slowing the function call. On some architecture the difference can be significative. On x86 it's not very important because of its lack of register, on SPARC, for example, it can be important. Up to 5 parameters are passed on registers and if your function uses few locals, no stack adjustment is made. If your function is a leaf function (i.e. doesn't call another function), there is no window adjustment also. The cost of the call is therefor very small. With a vararg, the normal sequence of passing parameters on the stack, stack adjustement and window management is made or your function wouldn't be able to get at the parameters. This increases the cost of the call significantly.

A function using varargs cannot correctly verify its arguments

To try to avoid this, creators of varargs routines rely on conventions to manage this. For example, args sometimes get passed in as pairs - a type constant followed by the actual value ending with a sentinel:
SomeFunc(kInt, 3, kDouble, 2.0, kString, "blah", kLastArg);
Other times a descriptor is passed in first which describes the arguments:
printf("%d %f %s\n", 3, 2.0, "blah");
Other times, the first argument is a command:
DispatchMessage(kPaintWindow, &bounds, blackColor);
No matter what, it is possible to break the callee by passing in something that doesn't follow the convention. The problem is that there is nothing that can be done to enforce the convention. Without enforcement, you get a crash if you're lucky. If you're not lucky, you get a time bomb set in your heap that will go off only if the conditions are right.
In the last example, you're about as safe as you can get as long as you don't actually called DispatchMessage(), but instead use author-supplied macros like:
#define PaintWindow(bounds, color) DispatchMessage(kPaintWindow, bounds, color)
because then at least DispatchMessage is called correctly as long as the macro matches convention, and it is a single point of failure (which means a single point for a fix, which we like). I would turn around and argue that a single API, DispatchMessage, would be better off being multiple APIs.
Here's the skinny - if you're using varargs, you're probably doing something wrong out of the gate. Varargs promises a lot, but delivers very little.

The reason for CallHookA() in amiga.lib can only be AmigaOS 1.x compatibility, as written in the NOTES of the autodoc.

CallHookPkt() was added in utility.library V36 (AmigaOS 2.0), using the libamiga.a CallHook() instead was required for software which should work on AmigaOS 1.x as well.