Can't eek out that last bit of jitter

More
13 Mar 2025 07:06 #323804 by royka
Replied by royka on topic Can't eek out that last bit of jitter
I've never used rtai, but run latency-history and htop, sort by priority (F6?) and you'll see. I think you could use the same script.
Bad RAM memory could be a cause, usually using 2 ram modules (of the same) is better so it can use DDR. But if one of the two is bad removing one or the other might give a better result.
HDD can also be a cause, if you still use that buy a cheap SSD.
Disconnect any CD/DVD drives, SD-reader units.

Please Log in or Create an account to join the conversation.

More
14 Mar 2025 18:44 - 14 Mar 2025 18:50 #323950 by royka
Replied by royka on topic Can't eek out that last bit of jitter
I think that you should enable HPC (High Performance Computing) to prevent it from lowering the cpu frequency. Possibly also core performance boost. And to see if the kernel parameter works: cat /sys/devices/system/cpu/smt/active
Last edit: 14 Mar 2025 18:50 by royka.

Please Log in or Create an account to join the conversation.

More
16 Mar 2025 22:59 #324074 by webbyguy
Replied by webbyguy on topic Can't eek out that last bit of jitter
Thanks royka,
Do you happen to know the names of the base and servo processes? I see migration/0 through migration/5 and rtkit-daemon listed as RT in htop. They don't seem to increase in time though, so I'd guess they aren't being used. There are some -20 priority processes too.

I tried removing each ram module, and it seemed to work for a bit, but I'd restart the latency-histogram (without restarting anything else), and it would stop helping, so I guess really some other random thing was happening that I can't find.

I am using a cheap SSD. I tried playing around with bios settings for it, and it seems to make no difference. I don't have any CD/DVD or SD-readers.

Please Log in or Create an account to join the conversation.

More
16 Mar 2025 23:15 - 16 Mar 2025 23:20 #324075 by webbyguy
Replied by webbyguy on topic Can't eek out that last bit of jitter
I no longer have the HPC option in my bios settings.  I'm guessing it went away when I switched to the X6 1090T processor (instead of the fx 8350).  The 1090T didn't have hyperthreading, so I'm not using it.  With the real time kernel, for some reason /sys/devices/system/cpu/smt/active isn't there, but I'm sure I don't have hyperthreading now.

I tried the core performance boost.  It seemed to work for a bit, but restarting without changing settings made it go away.  I seem to have some random really good performance every once in a while, but it isn't repeatable.  Even restarting just the latency-histogram makes it go away sometimes.

Here's one of the times it looked pretty good for awhile.
 

I also tried adding the following commands to try to help things, but it is hard to tell if they really did anything.

swap off --all
systemctl --user stop pulseaudio.socket
systemctl --user stop pulseaudio.service


I also moved the rcu_sched to processor affinity 1,2,3,4,5.  That helped until I started firefox, and I could never get it back.


I think I'm gonna just try to move foreward without getting that last bit of jitter out.  I'm consistently below 40us, which sounds like it is good enough.  (I'll come back to it if I get some motors moving.) 

Oh, and for completeness, I tried the blocking of sound modules described here:
forum.linuxcnc.org/18-computer/39371-res...th-linuxcnc?start=90
Attachments:
Last edit: 16 Mar 2025 23:20 by webbyguy.

Please Log in or Create an account to join the conversation.

More
17 Mar 2025 00:20 - 17 Mar 2025 00:25 #324080 by royka
Replied by royka on topic Can't eek out that last bit of jitter
The process with the PREEMPT_RT kernel is rtapi. The one with -98 priority is the base thread and the one with -99 is the servo thread.
Some settings you could try:
Maximum Payload = Auto > to max
AMD 880 HD Audo = Enabled > Disabled
PME Wake Up From S5 = Enabled > Disabled
PS2 Keyboard PowerOn = Any Key > Disabled
Stroke Key = Wake Key > Disabled
PS2 Mouse PowerOn = Enabled > Disabled
USB Device Wake Up From S3/S4 = Enabled > Disabled
Unpopulated Links = Keep Link ON > Off

And the following kernel parameters instead:
skew_tick=1 nosmt=force kthread_cpus=0-1 irqaffinity=0-1 rcu_nocbs=2-7 nohz_full=2-7 isolcpus=2-7
intel_pstate=disable cpufreq.off=1 cpuidle.off=1 processor.max_cstate=0 idle=mwait clocksource=tsc
tsc=reliable nosoftlockup nowatchdog default_hugepagesz=1G hugepages=4 transparent_hugepage=never
threadirqs forcedeth.use_msi=0 mce=ignore_ce nmi_watchdog=0 audit=0 mitigations=off
Last edit: 17 Mar 2025 00:25 by royka.
The following user(s) said Thank You: besriworld

Please Log in or Create an account to join the conversation.

More
17 Mar 2025 01:22 #324083 by webbyguy
Replied by webbyguy on topic Can't eek out that last bit of jitter
So far, that works!  I don't think I ever would have gotten there without your help.  Thank you.

I didn't end up deleting everything in my grub setup, but I put in everything you added.  The 1090T only has 6 cores, so I reduced the 7s to 5s.  I ended up with:

 smt=1 ro quiet skew_tick=1 nosmt=force mds=nosmt mitigations=auto,nosmt irqaffinity=0-1 kthread_cpus=0 rcu_nocb_poll rcu_nocbs=2-5 nohz=on nohz_full=2-5 isolcpus=2-5 intel_pstate=disable amd_pstate=disable idle=mwait clocksource=tsc tsc=reliable cpufreq.off=1 cpuidle.off=1 intel_idle.max_cstate=0 amd_idle.max_cstate=0 processor.max_cstate=0 nowatchdog nosoftlockup acpi_irq_nobalance noirqbalance default_hugepagesz=1G hugepages=4 transparent_hugepage=never threadirqs forcedeth.use_msi=0 mce=ignore_ce nmi_watchdog=0 audit=0 initrd=/install/gtk/initrd.gz


Not only does it work, it is also running much cooler.  Now the CPU is around 31C, and before it was near 50C.

This result is using the A880GZ with integrated HD4250 graphics.  (I didn't need the R5 340X, but I may try it out just to give others info.)
 

Thank you so much!
 
Attachments:
The following user(s) said Thank You: besriworld, royka

Please Log in or Create an account to join the conversation.

More
17 Mar 2025 01:45 #324084 by webbyguy
Replied by webbyguy on topic Can't eek out that last bit of jitter
For anyone wondering when I tried with the R5 340X graphics card, the max base jitter went up to +/-30us.  So I guess that isn't a good graphics card for linuxcnc, or maybe it just doesn't work with the A880GZ and linuxcnc.

Also, in case anyone is interested in, here is a screen shot including the watch of /proc/interrupts when using the integrated HD4250 graphics card with the biostar A880gz .

 
Attachments:
The following user(s) said Thank You: besriworld

Please Log in or Create an account to join the conversation.

More
17 Mar 2025 01:48 #324085 by webbyguy
Replied by webbyguy on topic Can't eek out that last bit of jitter
Oh, and on that screenshot, I did NOT run

swapoff --all
systemctl --user stop pulseaudio.socket
systemctl --user stop pulseaudio.service
sudo sysctl kernel.sched_rt_runtime_us=-1

I guess they were unnecessary. (Although maybe I had added that last one to startup in some way before, but I don't remember.)
The following user(s) said Thank You: tommylight, besriworld

Please Log in or Create an account to join the conversation.

More
17 Mar 2025 09:06 - 17 Mar 2025 12:33 #324099 by royka
Replied by royka on topic Can't eek out that last bit of jitter
TBH I used Deepseek on this one (and didn't filter out the 2-7 error), but I'm glad it helped you. The idle=mwait instead of poll seems to lower the temperature without much impact on the performance. Now the latency is good you could also try to only isolate the last core (so replace each 2-5 with 5), if the latency will be the same you'll have a much better responsive system.
Last edit: 17 Mar 2025 12:33 by royka.

Please Log in or Create an account to join the conversation.

More
18 Mar 2025 16:40 #324215 by webbyguy
Replied by webbyguy on topic Can't eek out that last bit of jitter
That's cool. I had tried both Gemini and Copilot, and I didn't get anything close to as useful as that. Thank you again.

Please Log in or Create an account to join the conversation.

Time to create page: 0.246 seconds
Powered by Kunena Forum