Tracking down stuttering (deb-12, LinuxCNC-2.9)

I'm converting a Langmuir MR-1 gantry mill to LinuxCNC.  I originally started with the current 2.8-series boot image, but decided to move to 2.9 to take advantage of python3.  My machine has debian12, LinuxCNC 2.9.  I'm currently using pyqtvcp, but had the same problem with axis.  My hardware is interfaced through a Mesa 7i96S.  My latency histogram is decent but not awesome with the servo-thread showing min(us) of -42.6 and max(us) of 42.3 while the machine is under heavy load.  I've set isolcpus=2,3 (my machine uses a 4-thread/2-core Intel i5).

I am using the built-in wifi and ethernet, but have turned of all other system devices in the bios (touch screen, sound, etc).

When I'm jogging I'm getting inconsistent speeds and I can also see the current velocity number in the UI dropping when I hear the stepper stutter.  A theory is that the 7i96S is finishing a command and that the next one hasn't been enqueued yet, so that is when I'm hearing the stutter.  I can also hear the same issues on long rapid commands.

I didn't have these issues when using the same PC, Mesa, and machine on LinuxCNC 2.8.x.  I know that I'm on the bleeding edge using Deb-12 and LinuxCNC 2.9, though both are close to release.

What can I look for to track these issues down?

Since it is happening on long programmed rapids (G0 across the full diagonal of the machine) I expect that it will also happen when running programs.

I haven't run a real program on it since moving to Debian 12, I'm still trying to dial it in and get my probing routines working.

Since it happens on long G0 commands typed into the MDI I don't think it is a keyboard issue.

Your theory about overruns  with the mesa card can't be vailid. If it does happen, the card will report an erorr finishing read and stop communicating.
There are pins for tmax (the longest time to read and write that are in timer ticks that can be used to check  (divide by the frequency in latency-histogram to convert to usec.)

There is also a packet errror pin that counts how often it happens

I have done a lot of work with bookworm. You might like to read my how to guide.
docs.google.com/document/d/1jeV_4VKzVmOI.../edit?usp=share_link

A good general approach to trouble-shooting is to do a binary search.

As a first step, to check if the motion issue is in step generation or external
hardware you can plot the following error with halscope

"When I'm jogging I'm getting inconsistent speeds and I can also see the current velocity number in the UI dropping when I hear the stepper stutter. "

If the velocity changes, this is not due to motion hardware but likely the
keyboard repeat jogging issue tommylight mentioned.

RodW: Thanks for that document! I used it during my initial Debian 12 setup and it pointed me towards isolcpus setting and tracking down some keyboard options. It would be great if your document was stickied, I found it references in a few threads and it took me longer to chase it down.

I think there are really two issues here. One is that the steppers don't sound right when running under 2.9 but sounded fine under 2.8 with the same hardware. In 2.9 I can hear a second order inconsistent ticking on long rapids (not jogging) done with G0 commands. I don't think this is missed steps, but I haven't done a test yet (it's in my plans next time I get to the machine). My test plan is to do 100 3" back and forths along a 1/2/3 block and see if I end up at the same place at the end of them.

The other issue is that on long jogs (with keyboard) I get inconsistent speeds. This is likely the keyboard issue, though I've tried to solve it by setting kbpoll to low values (tried 1 and 4) and by turning keyboard repeat to high values. My next thing was going to be setting up a debugger to track key-down and key-up events to see if I'm getting spurious key-up events in the middle of a long hold.

I have a long history (about 15 years) with EMC2 and LinuxCNC and have converted a number of machines with both steppers and servos -- this is the first one that is really vexing me. The same hardware (mesa card, power supply, stepper drivers, and steppers) seemed like it behaved better under 2.8. I might setup dual boot with both environments so that I can do a direct comparison.