I finally could solve the problem, so if someone has the same problem in the future maybe this post can help you as well:
I first added some markers to the stepper shaft so I can directly track the rotation, and disconnected them from the axes. From here it could be seen that the problem really comes from the steppers and not from the mechanics.
So the first thing I did was to try a new PC with less jitter. That did not help.
I tried to change some more timing settings in stepconf, however that did not help eighter.
What solved the problem was following:
On the stepper driver there is the possibility to select by a dip swich if a step should be triggered on rising or falling edge. I changed this from rising to falling and suddenly the problem was gone.
So my guess is, that the signal looks bad and somehow the stepper triggers multiple steps when the signal overshoots or something like that, unfortunately I do not have an oscilloscope at the location the cnc is at.
