[SOLVED] Huge latency with PREEMPT_RT

I have a newly installed system following the Debian 10.4 Buster Linuxcnc quick and easy install guide.

latency-histogram looks like this:

arvidb@debian:~$ latency-histogram --nobase --servo 250000 --show

I believe most of the outliers are about 2 ms, so it seems I get about 5-10 of these delays each second. If I reset the values I immediately get a max of thereabouts again.

Now I have used this computer to run linuxcnc back in 2014 or thereabouts and I know I managed to get very good latency figures then, with 250 µs base thread.

Any idea where to begin to look? I don't have any of the rt* files under /dev mentioned in the Wiki Troubleshooting section - are these needed for RT_PREEMPT as well as for RTAPI?
Is the rtapi_app supposed to run with realtime scheduling? It seems it doesn't:
Anything else I should check?

2 ms is huge, Do you have all power management disabled in the BIOS?

Here's an older kernel on a similar system:

freeby.mesanet.com/e8500-4.4.1-rt5.png

but 4.19 should be similar

Thanks, disabling BIOS power management made a huge difference. I turned off "CPU EIST Function", "Virtualization Technology", and C2 and C1E idle states in BIOS, and got 46 µs max latency while watching YouTube videos:


isolcpus=1 makes the histogram sharper but doesn't seem to affect the outliers.

Some further testing shows that the culprit was the EIST function: Enhanced Intel SpeedStep . With that turned off, but C4, C2 and C1E enabled and a few glxgears running, latency stays below 20 µs. Using Firefox to watch YouTube however still brings some spikes up to about 500-700 µs (but *much* rarer than the continuous 2 ms spikes I got with EIST enabled):

Yes, I think Speed-step basically stalls the processor while it waits for the voltage
and clock speed to change so its a no-no for real-time

If you are going to isolate a core, you are best off isolating 2 cores and they should be on the same die.

BeagleBrainz: Isolating two cores on a dual-core machine without hyperthreading doesn't really make sense to me...?

It seems the remaining outliers happen when I switch focus to the Firefox window (regardless of contents; it's not really YouTube related as I first thought). Not every time, just once in awhile. Turning off the xfce compositor helped a bit by approximately halving the latency of the outliers - but it didn't reduce the number of them. Maybe the occasional 300-350 µs delay is okay for a 1 ms servo thread (no base thread)? Or I might just have to leave Firefox alone while running an NC program...

Maybe I should try some other window environment? Anyone else running xfce successfully? This machine's got an Radeon HD 3870 graphics card, running the radeon driver.

Edit: I should add that this will be a mixed EtherCAT (Omron G5 for X, Y, and spindle) and Mesa (Z - Yaskawa SGDA, and IO) system.

It might be video board related.
Does the MB have built-in Intel video?
If so I would give that a try

Sorry mate thought it was a quad core. If you have hyper threading enabled disable it.
Also try without any third party video drivers.
XFCE can help with latency.
Do a standard install of Mint 19.x with XFCE then use the kernel and packages from my repo, everything from there has been compiled on Mint so you won’t have any issues with dependencies.
Just use the standard X Windows drivers for your video card.
Try a PS/2 mouse and keyboard if you can, I have personally seen reductions in latency by using non USB keyboard and mouse, I use PS/2 myself rather than USB. USB wireless will mess up latency, actually I do all my latency tests without any wireless. If it is needed only enable it when not running linuxcnc.
If you’re still not getting the required results a new machine might be the only answer. Tommy usually has the answers to that one.

PCW: No built-in graphics unfortunately. I do have another graphics card (that uses the amdgpu instead of the radeon driver) that I could use temporarily for testing though, if it turns out to be necessary.

BeagleBrainz: Thanks for trying to help.
* As I said, and as you can see from Intel's E8400 page , this is a 2-core CPU without hyperthreading.
* As I wrote, I'm using the radeon driver already (i.e. the default, in-kernel driver). No 3rd party drivers installed.
* As you can see from my previous message, I'm already running xfce.

* I'm actually already using a PS/2 keyboard on this computer, but I have no PS/2 mouse to try unfortunately. Also nothing wireless.

Do you have any specific reasons to believe another distribution (Mint) would help, despite running the same kernel, drivers and window manager on the same hardware? Redoing the install is a large undertaking since I'm using EtherCAT - I have to build everything from source. (It's great that you're eager to help, but it mostly feels like you're sending me on wild-goose chases by not taking time to understand the circumstances...)