PC for Mesa 5i25T

I would first verify that you have 5V at the 7I77 in case this is an analog
output voltage issue.

Another possibility is encoder count rate issues, especially if there is significant
quadrature distortion. You can try disabling the encoder filter to test this.

 

In the end, the problem was in the switched analog signal and the GND of the analog signal. After swapping it works without a problem.

Yeah, a shorted analog output is going to drag the (relatively weak) local
power supply down.