Abysmal latency test on reasonably powerful PC

More
10 Jun 2021 21:48 #211737 by thanks4opensource
Good news, at least for me -- I'm getting much better latency numbers after modifying the machine's BIOS settings to:
H.D.D. S.M.A.R.T. Capability	[Disabled]  (left Disabled)
Limit CPUID Max. to 3		[Disabled]  (left Disabled)
No-Execute Memory Protect	[Enabled]   (left Enabled)
CPU Enhanced Halt (C1E)		[Disabled]  (was Enabled))
CPU Thermal Monitor 2 (TM2)	[Enabled]   (left Enabled)
CPU EIST Function		[Disabled]  (was Enabled))
Virtualization Technology	[Disabled]  (was Enabled))
Init Display First		[PEG]	    (changed from PCI)

The important ones were probably "C1E" and "EIST", and maybe virtualization. I left "No-Execute" enabled because I didn't think it would hurt, and TM2 because I'd rather have poor performance than fry the CPU if temperatures get out of range.

With these changes, the linuxcnc-2.8.0-buster Latency Test "max jitter" servo/base nanoseconds came down from the ridiculous 3million+ to 32000/68000 with some other processes running. Note that I am seeing some warnings in dmesg:
perf: interrupt took too long (3939 > 3900), lowering kernel_perf_event_max_sample_rate to 50750

I then booted a live linuxcnc-2.7.14-wheezy image, which was even better, around 17000/22000 ns. No "perf" warnings in dmesg (hard to tell, no "--follow" option in this version). Also see that it's an i686 build vs x86-64 for 2.8.0-buster, and that `top` shows one core maxed out and the other 50% vs both lightly loaded on 2.8.0. Wondering if this is because i686, the different RT kernel, a different Nvidia driver, or something completely unrelated.

So ... what's the best path forward? Is 17000ns usable for an open-loop CNC mini-mill (steppers with no encoders)? Should I go with 2.7.14-wheezy because those numbers are a bit better and upgrade to 2.8 after installation? I'm assuming the "wheezy base 2.8-rt" apt source just updates the LinuxCNC apps, not the kernel and/or the rest of the distro. The i686 build is also interesting -- I have an even older Pentium 4 box that wouldn't boot the ubuntu-10.04-linuxcnc3-i386 image (machine is a weird configuration). I'll probably see if 2.7.14-wheezy will boot because I'd rather keep the Core 2 Duo box for general use and consign the otherwise useless P4 to running the mill. (There are some very good Pentium numbers in the latency database compared to poorer ones for more recent CPUs, but maybe that's the black magic of realtime Linux.)

Many thanks to everyone for all the help so far, and for any further suggestions.

Please Log in or Create an account to join the conversation.

More
10 Jun 2021 22:01 - 10 Jun 2021 22:26 #211742 by BeagleBrainz
Don't worry about the dmesg messages regarding interrupts, you'll get them on non rt systems, it's just a info message.

If you machine is 64 bit use the 64 bit version.

17000ns is good as long as that is not the jitter figure.
I'd go for the best figures and what is the most stable.
Last edit: 10 Jun 2021 22:26 by BeagleBrainz.

Please Log in or Create an account to join the conversation.

More
10 Jun 2021 22:18 #211743 by thanks4opensource
Thanks, @BeagleBrainz. I downloaded www.linuxcnc.org/iso/linuxcnc-2.7.14-wheezy.iso from the link at linuxcnc.org/downloads/. Unless I messed up somehow, that was the i686 kernel as per`uname`.

Sure, I'd rather run an x86_64 build on a 64 bit machine. Is there one pre-built somewhere? I'll probably still experiment with the i686 one on the Pentium box.

No, like I said, 17000ns is in the latency test's "Max Jitter" column and the "Servo thread (1.0ms)" row. Should I be looking at something else instead?

Please Log in or Create an account to join the conversation.

More
10 Jun 2021 22:29 #211745 by thanks4opensource
OK, looking again, maybe I've been missing this the whole time. I've been looking at the "Max Jitter" column because it's in bold and I thought that was the most important number. Thinking about it more analytically, "Max Interval" makes more sense.

If I have to talk about one number, "Max Interval" for the "Servo thread (1ms)" is it, right? I'll re-run and check that. On the order of 1 million or less is what I'm trying for?

Please Log in or Create an account to join the conversation.

More
10 Jun 2021 22:31 - 10 Jun 2021 22:47 #211746 by BeagleBrainz
Don't listen to me.....having trouble to brain today

Better info here

linuxcnc.org/docs/2.8/html/config/stepconf.html#_latency_test
Last edit: 10 Jun 2021 22:47 by BeagleBrainz. Reason: Brain has taken off early for long weekend

Please Log in or Create an account to join the conversation.

More
10 Jun 2021 22:58 #211750 by thanks4opensource
OK, good. And I re-re-thought it and am back to jitter being the important number. That's what linuxcnc.org/docs/2.8/html/install/latency-test.html says, anyway. So the "Max interval" and "Last interval" for the "Servo thread (1ms)" means that LinuxCNC wants to do something at exact 1ms intervals and the columns are reporting the actual times (max and latest) and the jitter is the worst it's been wrong. (And there's the other utility which displays a jitter histogram instead of just the simple maximum). And is the Servo thread more important than the base thread, or are they both equally important? And the base thread is 25us vs 1ms for servo, so isn't X ns of jitter on base worse than the same X on servo?

There's still the mystery that 32 bit Wheezy runs better than 64 bit Buster on my 64 bit Core 2 Duo CPU. I thought this was because one was PREEMPT-RT and the other RTAI. Sorry, but the linuxcnc.org/downloads/ page is a bit confusing on this. It says 2.8.0 is RTAI by doing LinuxCNC 2.7.14 Debian 7 Wheezy and upgrading to 2.8, but links to the same ISO image as 2.7.14 Debian 7 Wheezy which says PREEMPT-RT. Is it PREEMPT-RT but gets an RTAI kernel when upgraded to 2.8?

Sorry, I'm a bit lost here. I can keep experimenting until I get the best results, but understanding what's going on would make it easier.

Please Log in or Create an account to join the conversation.

More
10 Jun 2021 23:22 #211754 by tommylight
On the main download page:
linuxcnc.org/downloads/
there is the:
+LinuxCNC 2.8.0 Debian 10 Buster RTAI
section, third from top, so you get Buster with RTAI kernel and better latency, and 64bit system.
Jitter is the deviation time,
Not a mystery, RTAI was for older PC's, RT-Preempt if for new PC's, there are other differences but i am not qualified to name them.
You can also run latency tests from the terminal with different timings like a 50000 ns base period, not much use since what you got is already good for what you need. Might be a bit limited to max speed if using a lot of microstepping, but on a mill this might prove OK.

Please Log in or Create an account to join the conversation.

More
11 Jun 2021 00:14 #211756 by BeagleBrainz
Generally RTAI gives better latency than RT_PREEMPT.

A bit of discussion here:
forum.linuxcnc.org/9-installing-linuxcnc...84-rtai-vs-preemt-rt

Please Log in or Create an account to join the conversation.

More
11 Jun 2021 01:35 #211766 by thanks4opensource
Thanks @tommylight and @BeagleBrainz. I didn't know that the different RT versions were better suited for different generations of CPUs. And maybe I'm missing it, but it isn't clear which .iso images are 32 bit and which are 64, except for the very old versions which have "i386" in their names.

I'll try the apt-get of RTAI on top of linuxcnc-2.8.0-buster.iso, and will also read the RTAI vs PREEMPT-RT thread and see if I gain any insights from it.

Please Log in or Create an account to join the conversation.

More
11 Jun 2021 03:04 #211767 by BeagleBrainz
Just remember the message regarding the current rtai kernel

linuxcnc.org/docs/2.8/html/getting-start...#cha:Installing-RTAI

Please Log in or Create an account to join the conversation.

Time to create page: 0.156 seconds
Powered by Kunena Forum