Could it be the stepper drivers?

I have been slowly loosing my mind converting an old isel desktop cnc to a modern controller and I hoping someone here can help me.

I have tried a few different controllers from arduino with estlcam, esp32 with fluidNC, parallel port BOB and now a Mesa 7i96S, but with all of them I got some step skipping. Specifically when making small circles like with spiral drilling or adaptive slotting. With the Mesa I'm pretty sure the problem isn't on the side of step generation and I have tried two different stepper motors, so it isn't that. The only thing that I haven't tried multiple versions of are the stepper drivers and since they are a generic version of DM542 it's not hard to see why I'm suspicious, but I'm still not sure it makes sense. Or more precisely I'm not sure if buying a leadshine DM542 or EM542 would solve the problem, or would I have to spend noticeably more?

I have set the pulse and direction times per specification as well as double that and it didn't change much, and I have played around with the wiring a lot and it didn't solve anything. The only thing that does noticeably affect the result is the microstep setting, but the higher the number the better the result. Attached is an image of the same toroidal path around a square with the microstep setting written on each one.
 
This was with the BOB, with the Mesa I could also do 12800 and got even better results, but not perfect. Also running with the same setting again gives a very similar result, so the mistake isn't random. Also it can run straight paths in any direction and at any speed without skipping a step.

My theory is that the drivers don't react to direction changes as fast as they claim, but I would like to hear some sort of conformation that that's possible, or some encouraging experiences with leadshine drivers before I have to hide more packets from my gf.

Thanks for the recommendation, but it didn't change much. With 150mm/s/s it would only reach about 600mm/min in that small circle, but it still consistently skipped steps. When I lowered the feed to 270mm/min it continued on the same trajectory. If I lower it significantly more, it'll take hours to finish.

The steppers are rated to 2,8A and the drivers are set to 3A peak. I tried 3,3A, but they would occasionally alarm out. I doubt that it's a problem with the limitations of the motors, as the machine could do all of this with the old control board and drivers.

They start at the green dot and follow the arrows.
 

The fact that increasing microsteps reduces the errors
suggests  that either you are at the torque/speed limits of the
drive/motors (despite what many say, increasing the microstep
ratio increases the stall torque because the smoother motion
is less likely to excite resonances that induce stalls) or
there is an electrical issue that is adding steps, say a missing ground...

I guess a mechanical issue (binding or slippage) is possible as well

Try reducing accelleration  by 50% as thats where the drive is working (before cutting forces)
If it works, increase slowly to get reasonable performance
But as said earlier there could be noise

So this is a picture of the driver and the whole wiring mess.
 
 

I know it's a bit of a mess, but I've changed so many things I've given up on order until I get a working version.
I took four strand shielded cable from work to connect the mesa with the drivers and I grounded the shielding on the mesa side. Since we make industrial encoders with up to 20m of such cable, I figured it would be perfect. They aren't twisted pairs as I've seen recommended, but that isn't a problem for any of our customers.

I highly doubt that torque is a limitation, since it can move back and forth in a straight line at 1000mm/s/s, but skips steps at 150mm/s/s when making a circle. Also I crashed quite a few times with the old controller and drivers so I know how much damage it does before it skips a step because of torque.

The toolpath I use for testing is generated in Estlcam which can do simplified adaptive slotting with a lot of small circles.
 
It's good for testing, because it produces an obvious error, but the same happens with other toolpaths with arcs (with or without G2/G3). I first noticed it when I was making 24 parts at once and the fist operation looked great, but the second one was a chamfer that showed a noticeable drift from the first to the last part. I tried rehoming and starting from the 12th part and it started perfectly and then drifted off, so it's a systemic mistake.

My thinking is that the mistake only happens when an axis changes direction in an arc and one microstep is made in the wrong direction. This would explain why higher microstepping reduces the error and why there seems to be a weird consistency to the drift.

If you look at the "1600" route it does not look like a direction issue,
but extra steps added to what should be a stationary axis. How else do
you get a slanted cut that is supposed to have only 1 axis in motion?
This seems more like a noise/crosstalk issue.

I would definitely get rid of the flying ground leads. If you want to ground
the STEP/DIR cable shields, ground them at the 7I96S TB1 GND pins.