Abysmal latency test on reasonably powerful PC

More
10 Jun 2021 06:35 #211670 by thanks4opensource
Hi. Experienced Linux user, CNC newbie here. Buying a first CNC mini-mill (lifelong dream!) and want to use LinuxCNC which I've heard about since its beginnings.

Installed linuxcnc-2.8.0-buster.iso on a Gigabyte GA-965-DS3 motherboard with 2.13GHz Core2 Duo 6400 CPU, Nvidia GeForce 7600 GS graphics card, and 4GB of memory. Yes, it's a 14 year old machine, but was only retired 18 months ago for an 8 core 32GB AMD Ryzen 3700X monster because web browser bloatware finally caught up with Linux and 4GB wasn't enough. Except for the memory, performance in day-to-day usage between the two machines isn't that noticeable.

But running the LinuxCNC latency test on the Gigabyte gives "Max Jitter (ns)" around 3 million (3ms) for the "Servo thread (1.0ms)". "Base thread (25.0us)" is a little better, maybe 2 million. This is with nothing else running -- `top` shows "rtai-app" at 20% on one core, otherwise that one is 80% idle and the other 100% idle. I tested with both the Live system and installed/booted from the hard drive -- no difference. Pressing "Reset Statistics" drops the numbers down to around 40000ns for a fraction of a second but then right back to 3 million. I tried with the parallel port set to SDD, EPP, and ECP in the BIOS and got the same results with each.

On a whim, I booted the live system from the ubuntu-10.04-linuxcnc3-i386.iso image and got around 43000/20000 for the servo/base threads. Even running `glxgears` (eats 90% of one core) and opening Firefox doesn't change those numbers much.

Does anyone have an explanation or a fix? Is the latency test misleading if nothing is attached to the parallel port? The docs imply that even a Pentium class machine should be enough for LinuxCNC, but many of them hint that they're out of date. Does 10.04 like the machine because they're both old and 2.8.0 is going, "OK, boomer!" and refusing to run? ;) I'd hate to give up 9 years of development progress (10.04 is dated 2012 in the /iso download directory) by running such an old release. In any case, thanks for LinuxCNC (see my username and/or the minor contributions I've made elsewhere attached to it).

Please Log in or Create an account to join the conversation.

More
10 Jun 2021 07:39 #211675 by robertspark
are you running a real time kernel?

those numbers indicate not.....

Please Log in or Create an account to join the conversation.

More
10 Jun 2021 07:58 #211676 by thanks4opensource
I'm running whatever kernel is in the bootable/installable ISO image. Downloads page says:

The Debian 10 Buster ISO uses a PREEMPT-RT patch which is close to mainstream Linux but does not, in some cases, give quite such good realtime performance as the previous RTAI kernel. It is very often more than good enough. It should probably be the first version tried even if using a parallel port. This is compatible with all Mesa and Pico interface boards.


Do I have to do something explicit to turn on / enable the realtime scheduler in the kernel?

Please Log in or Create an account to join the conversation.

More
10 Jun 2021 07:59 #211677 by BeagleBrainz
A lot of things can affect latency, power saving setting, whether the BIOS has a turbo setting a whether that is on. SMI issues can also play a role. Wifi can have an adverse affect on latency.

Whether or not you have anything attached to the Parallel port has no affect for latency numbers.

You may be best using a RTAI kernel rather than a RT_PREEMPT kernel. Just think of it as 2 different implementations of real time.

Linuxcnc Wheezy has an RTAI kernel, which I also believe can be updated to Linuxcnc 2.8. Be aware that Wheezy is no longer receiving updates but shouldn't be an issue for a pure cnc controller.

This link has some info, it may seem outdated but your machine is a fairly old machine.
wiki.linuxcnc.org/cgi-bin/wiki.pl?TroubleShooting

Some more here:
wiki.linuxcnc.org/cgi-bin/wiki.pl?RealTime

And here as well:
wiki.linuxcnc.org/cgi-bin/wiki.pl?FixingSMIIssues

In rare case the GPU can have an affect as well. It really is a can of worms.

Please Log in or Create an account to join the conversation.

More
10 Jun 2021 08:24 #211679 by thanks4opensource
Thanks for the suggestions and links, @BeagleBrainz. I'll try linuxcnc-2.7.14-wheezy.iso and report back if I find anything interesting. Also didn't think of looking at dmesg for clues -- I was hoping to act like a new user and that LinuxCNC would work out of the box for me.

I've looked a little into Linux realtime performance and will likely have to bite that bullet for another, unrelated, project I'm working on. I've only skimmed the surface but am amazed that Linus has resisted including the various RT patches into the mainstream kernel sources for, what? 10+ years now? Again, way above my pay grade (level of expertise) and off-topic here, but it does seem like a huge part of LinuxCNC's development efforts.

Please Log in or Create an account to join the conversation.

More
10 Jun 2021 10:03 #211683 by BeagleBrainz
The PREEMPT patches are part of the mainline kernel, they have been for a while now.
Don’t blame Linux for the real-time performance issues, as you should know, a lot has to do with the behind the scenes activities of the CPU.
Context switching plays a role, which you really can’t escape if you still want interrupts and multitasking.
I’ve used 3 different platforms, BeagleBoneBlack, a itx board with a J3355b and a Odroid H2 plus, all 3 have been fine “out of the box” apart from the initial machine config and even that was pretty straight forward.
Remember a 14 year old machine is a 14 year old machine, kernel and software development have come a long way in that time.
Even newer/recent (5 years) machines can require a bit of fettling to get going with acceptable latency.

Please Log in or Create an account to join the conversation.

More
10 Jun 2021 11:54 #211689 by arvidb

But running the LinuxCNC latency test on the Gigabyte gives "Max Jitter (ns)" around 3 million (3ms) for the "Servo thread (1.0ms)". "Base thread (25.0us)" is a little better, maybe 2 million.

I got similar latency on my Core2 Duo - until I turned off "CPU EIST Function" (Enhanced Intel SpeedStep) in the BIOS. That brought it down to 500-700 µs.

After replacing the graphics card it's now down to ~30 µs.

See this thread .
The following user(s) said Thank You: BeagleBrainz

Please Log in or Create an account to join the conversation.

More
10 Jun 2021 12:33 #211692 by tommylight
Dissable hyperthreading, speedstep, pci-e agresive power saving, modem, serial, virtualisations, TPM, everything not needed to run the PC.
Causes of bad latency after dissabling above:
Low memory
Bad memory module
HDD, yes some HDD can mess latency badly even if they are OK
Graphic card, especially NVIDIA, but not all
Bad cooling/ thermal throtling
And always, motherboards and proccessors, real life example Ryzen 5 3500 is useless while 5 3600 is very good latency vise.
Also real life example, i do have some Dell Core2duo running Debian 10 with Mate and RT kernel (not RTAI) daily for quite some time without issues with base period at 80.000.

Please Log in or Create an account to join the conversation.

More
10 Jun 2021 12:49 #211694 by BeagleBrainz
I'm not sure what disabling a serial port would have on latency, can you expand a bit more on this. Going by this logic would disabling the Parallel Port reduce latency if it wasn't needed ?

My thinking is that the serial port wouldn't affect anything latency wise unless it was generating interrupts.

Please Log in or Create an account to join the conversation.

More
10 Jun 2021 13:29 #211695 by tommylight

My thinking is that the serial port wouldn't affect anything latency wise unless it was generating interrupts.

True, i just disable everything i do not use.
The only case that would mess things up is if any other software accesses the serial while the machine is working.

Please Log in or Create an account to join the conversation.

Time to create page: 0.083 seconds
Powered by Kunena Forum