Servo tuning f-error spikes

I can’t think of a proper way to measure the backlash though.

You can check backlash with a dial indicator measuring movement of the table for x.y or mount it in the spindle for checking the z.  Apply a force manually to see if the needle deflects. You can also check for lost motion when stepping electronically with a small increment and see if it registers on the indicator when reversing. 
Most likely not your issue here.
 

Disconnect all wires that correspond to speed reference to the motors - motor should stand still or move with smallest velocity as possible.
As for interface - you control your servos with analog voltage reference as I understood.

I think you have big values of "P" parameters in ini file, try do decreese them.

As for interface - you control your servos with analog voltage reference as I understood.

Yes, that’s correct, the servos are controlled with 7i77 through +-10V. When the 10V output and the encoder input of X are disconnected, it slowly drifts. Only visible with dial indicator. Or if I wait some time then its visible by eye.

I think you have big values of "P" parameters in ini file, try do decreese them.

P of X is 2, but with P=1 i get the same results. Should I decrease it even more?
P of Y is at 6 and the Y axis f-error spike is below 0.01 mm.

Could maybe a loose head of the linear encoder cause such a problem? This just came to my mind. For the controller this would look the same as mechanical backlash.

I will check that tomorrow. The machine stands at a friend of mine, I sadly don´t have the space at home.

I tried pushing the X-axis by hand with an indicator on it, but I did this while the machine was on. No movement of the needle at all. I will try again tomorrow with the machine of.
 

Increase P untill it starts to vibrate when trying to jog, then lower the value a bit till there are no vibrations at stand still.
Loose P values will cause those spikes as the motor will react slowly to commands.

If X is much worse than Y or Z with similar LinuxCNC PID settings
it may be analog drive tuning related. You might try some small drive
setting changes (marking the original POT settings first) to see their
effect on the error plot.

Also, I had problem with one channel of analogue control on MESA. PCW advised me to make some re-soldering of Chanel. That was very similar problems as yours.
Try to change channel to another one.

Problem with encoder could be but you can check it by osciloscope (in linuxcnc) and look that channels where encoders are connected.