Tracking down stuttering (deb-12, LinuxCNC-2.9)
04 May 2023 20:00 #270668
by awetmore
Tracking down stuttering (deb-12, LinuxCNC-2.9) was created by awetmore
I'm converting a Langmuir MR-1 gantry mill to LinuxCNC. I originally started with the current 2.8-series boot image, but decided to move to 2.9 to take advantage of python3. My machine has debian12, LinuxCNC 2.9. I'm currently using pyqtvcp, but had the same problem with axis. My hardware is interfaced through a Mesa 7i96S. My latency histogram is decent but not awesome with the servo-thread showing min(us) of -42.6 and max(us) of 42.3 while the machine is under heavy load. I've set isolcpus=2,3 (my machine uses a 4-thread/2-core Intel i5).
I am using the built-in wifi and ethernet, but have turned of all other system devices in the bios (touch screen, sound, etc).
When I'm jogging I'm getting inconsistent speeds and I can also see the current velocity number in the UI dropping when I hear the stepper stutter. A theory is that the 7i96S is finishing a command and that the next one hasn't been enqueued yet, so that is when I'm hearing the stutter. I can also hear the same issues on long rapid commands.
I didn't have these issues when using the same PC, Mesa, and machine on LinuxCNC 2.8.x. I know that I'm on the bleeding edge using Deb-12 and LinuxCNC 2.9, though both are close to release.
What can I look for to track these issues down?
I am using the built-in wifi and ethernet, but have turned of all other system devices in the bios (touch screen, sound, etc).
When I'm jogging I'm getting inconsistent speeds and I can also see the current velocity number in the UI dropping when I hear the stepper stutter. A theory is that the 7i96S is finishing a command and that the next one hasn't been enqueued yet, so that is when I'm hearing the stutter. I can also hear the same issues on long rapid commands.
I didn't have these issues when using the same PC, Mesa, and machine on LinuxCNC 2.8.x. I know that I'm on the bleeding edge using Deb-12 and LinuxCNC 2.9, though both are close to release.
What can I look for to track these issues down?
Please Log in or Create an account to join the conversation.
- tommylight
-
- Offline
- Moderator
-
Less
More
- Posts: 18562
- Thank you received: 6218
04 May 2023 21:28 #270676
by tommylight
Replied by tommylight on topic Tracking down stuttering (deb-12, LinuxCNC-2.9)
Only while jogging or even when running programs?
Please Log in or Create an account to join the conversation.
04 May 2023 22:12 #270680
by awetmore
Replied by awetmore on topic Tracking down stuttering (deb-12, LinuxCNC-2.9)
Since it is happening on long programmed rapids (G0 across the full diagonal of the machine) I expect that it will also happen when running programs.
I haven't run a real program on it since moving to Debian 12, I'm still trying to dial it in and get my probing routines working.
I haven't run a real program on it since moving to Debian 12, I'm still trying to dial it in and get my probing routines working.
Please Log in or Create an account to join the conversation.
- tommylight
-
- Offline
- Moderator
-
Less
More
- Posts: 18562
- Thank you received: 6218
04 May 2023 22:23 #270681
by tommylight
Replied by tommylight on topic Tracking down stuttering (deb-12, LinuxCNC-2.9)
You have to check that as the stuttering is usually a keyboard issue that shows up only when jogging.
Please Log in or Create an account to join the conversation.
04 May 2023 22:27 #270682
by awetmore
Replied by awetmore on topic Tracking down stuttering (deb-12, LinuxCNC-2.9)
Since it happens on long G0 commands typed into the MDI I don't think it is a keyboard issue.
Please Log in or Create an account to join the conversation.
- tommylight
-
- Offline
- Moderator
-
Less
More
- Posts: 18562
- Thank you received: 6218
04 May 2023 22:37 #270684
by tommylight
Things to check:
Power supply, if there is a LED see if it changes intensity while machine is doing a rapid
Drives, check current settings, to much current skips steps same as to little current, but makes more noise
In the ini file, set steplen and stepspace to at least 5000 while testing.
Also ini file, dirsetup and dirhold to 20000
Replied by tommylight on topic Tracking down stuttering (deb-12, LinuxCNC-2.9)
Yes, that is correct.Since it happens on long G0 commands typed into the MDI I don't think it is a keyboard issue.
Things to check:
Power supply, if there is a LED see if it changes intensity while machine is doing a rapid
Drives, check current settings, to much current skips steps same as to little current, but makes more noise
In the ini file, set steplen and stepspace to at least 5000 while testing.
Also ini file, dirsetup and dirhold to 20000
Please Log in or Create an account to join the conversation.
05 May 2023 00:44 #270689
by rodw
Replied by rodw on topic Tracking down stuttering (deb-12, LinuxCNC-2.9)
Your theory about overruns with the mesa card can't be vailid. If it does happen, the card will report an erorr finishing read and stop communicating.
There are pins for tmax (the longest time to read and write that are in timer ticks that can be used to check (divide by the frequency in latency-histogram to convert to usec.)
There is also a packet errror pin that counts how often it happens
I have done a lot of work with bookworm. You might like to read my how to guide.
docs.google.com/document/d/1jeV_4VKzVmOI.../edit?usp=share_link
There are pins for tmax (the longest time to read and write that are in timer ticks that can be used to check (divide by the frequency in latency-histogram to convert to usec.)
There is also a packet errror pin that counts how often it happens
I have done a lot of work with bookworm. You might like to read my how to guide.
docs.google.com/document/d/1jeV_4VKzVmOI.../edit?usp=share_link
Please Log in or Create an account to join the conversation.
05 May 2023 01:57 #270691
by PCW
Replied by PCW on topic Tracking down stuttering (deb-12, LinuxCNC-2.9)
A good general approach to trouble-shooting is to do a binary search.
As a first step, to check if the motion issue is in step generation or external
hardware you can plot the following error with halscope
As a first step, to check if the motion issue is in step generation or external
hardware you can plot the following error with halscope
Please Log in or Create an account to join the conversation.
05 May 2023 14:26 #270730
by PCW
Replied by PCW on topic Tracking down stuttering (deb-12, LinuxCNC-2.9)
"When I'm jogging I'm getting inconsistent speeds and I can also see the current velocity number in the UI dropping when I hear the stepper stutter. "
If the velocity changes, this is not due to motion hardware but likely the
keyboard repeat jogging issue tommylight mentioned.
If the velocity changes, this is not due to motion hardware but likely the
keyboard repeat jogging issue tommylight mentioned.
Please Log in or Create an account to join the conversation.
05 May 2023 16:50 #270736
by awetmore
Replied by awetmore on topic Tracking down stuttering (deb-12, LinuxCNC-2.9)
RodW: Thanks for that document! I used it during my initial Debian 12 setup and it pointed me towards isolcpus setting and tracking down some keyboard options. It would be great if your document was stickied, I found it references in a few threads and it took me longer to chase it down.
I think there are really two issues here. One is that the steppers don't sound right when running under 2.9 but sounded fine under 2.8 with the same hardware. In 2.9 I can hear a second order inconsistent ticking on long rapids (not jogging) done with G0 commands. I don't think this is missed steps, but I haven't done a test yet (it's in my plans next time I get to the machine). My test plan is to do 100 3" back and forths along a 1/2/3 block and see if I end up at the same place at the end of them.
The other issue is that on long jogs (with keyboard) I get inconsistent speeds. This is likely the keyboard issue, though I've tried to solve it by setting kbpoll to low values (tried 1 and 4) and by turning keyboard repeat to high values. My next thing was going to be setting up a debugger to track key-down and key-up events to see if I'm getting spurious key-up events in the middle of a long hold.
I have a long history (about 15 years) with EMC2 and LinuxCNC and have converted a number of machines with both steppers and servos -- this is the first one that is really vexing me. The same hardware (mesa card, power supply, stepper drivers, and steppers) seemed like it behaved better under 2.8. I might setup dual boot with both environments so that I can do a direct comparison.
I think there are really two issues here. One is that the steppers don't sound right when running under 2.9 but sounded fine under 2.8 with the same hardware. In 2.9 I can hear a second order inconsistent ticking on long rapids (not jogging) done with G0 commands. I don't think this is missed steps, but I haven't done a test yet (it's in my plans next time I get to the machine). My test plan is to do 100 3" back and forths along a 1/2/3 block and see if I end up at the same place at the end of them.
The other issue is that on long jogs (with keyboard) I get inconsistent speeds. This is likely the keyboard issue, though I've tried to solve it by setting kbpoll to low values (tried 1 and 4) and by turning keyboard repeat to high values. My next thing was going to be setting up a debugger to track key-down and key-up events to see if I'm getting spurious key-up events in the middle of a long hold.
I have a long history (about 15 years) with EMC2 and LinuxCNC and have converted a number of machines with both steppers and servos -- this is the first one that is really vexing me. The same hardware (mesa card, power supply, stepper drivers, and steppers) seemed like it behaved better under 2.8. I might setup dual boot with both environments so that I can do a direct comparison.
The following user(s) said Thank You: rodw
Please Log in or Create an account to join the conversation.
Time to create page: 0.213 seconds