2.8.2/Buster: should I use isolcpus=1 or cpuset

Well, I have to give cpuset a try yet, but I have been comparing 2.8.2-rt vs 2.8.2-rtpreempt and Debian vs Devuan. I guess cpuset will have to wait until next weekend. I have been reading linuxcnc script so I have now a guess on how the different parts of it could be launched.

Test computer is an HPCompaq 5850 MT, Athlon 64 X" dualcore, with an almost useless BIOS.

My boot line has nowatchdog usbcore.nousb acpi_irq_nobalance isolcpus=1 idle=poll in all cases. The usbcore.nousb part for sure seems to be useless after you work your smp-affinity but I maintained it just because the first test had it. nowatchdog has quite good effects once system's load gets up. I need idle=poll because this computer BIOS doesn't allow you to set anything related to C-states.

Devuan 3 Beowulf is Debian Buster without systemd/pulseaudio. I hate systemd's guts and pulseaudio is his closest friend. What I did on Devuan is just installing the basic system utilities, then a very basic Xorg server, with just WindowMaker window manager and xterm, then installed linuxcnc kernel and application.

Both Devuan Beowulf and Debian Buster let me down when doing update-grub if there was more than a system installed, I don't know where the grub.cfgs are gone but I know for sure that I needed to edit the boot line at boot time in grub itself

With 2.8.2-rtpreempt on both Debian and Devuan, I noticed that, even with isolcpus=1 on the boot line, my NIC (driver tg3, and probably others like USB) was sending IRQs to cpu 1. So latency tests were bad, well over 100 usec, reaching even 600-800 usec latency in just a few minutes. I needed to put at work an smp-affinity script along the lines of this post . I just stripped the ubuntu parts and placed it on /etc/rc.local on Devuan so it is executed at boot. On Debian, where systemd has to control the rackets, that's just bad practice. So I just execute it myself before launching latency-histogram with "taskset 1 latency-histogram". Then results are quite good enough to do software stepping, just a little over 25 usec on Debian and a little under 20 usec on Devuan.

2.8.2-rt worked wonders for me. No need to do nothing once you have isolcpus=1 on your grub boot line. Without it, it hangs wonderfully. Latency results are almost the same thing on Debian and Devuan, about 5 usec. I'm still dazzled. I noticed than the system load and memory consumption were better on the light weight Devuan system, let's say about half the RAM and 40% less load.


Debian Buster 2.8.2-rtpreempt with no smp-affinity work

Debian Buster 2.8.2-rtpreempt with retouched smp_affinity

Debian Buster 2.8.2-rt

Devuan Beowulf 2.8.2-rtpreempt

Devuan Beowulf 2.8.2-rt


 

tatel wrote:

Both Devuan Beowulf and Debian Buster let me down when doing update-grub if there was more than a system installed, I don't know where the grub.cfgs are gone but I know for sure that I needed to edit the boot line at boot time in grub itself

I haver a triple-boot system, and have seen this problem. 
The answer seems to be to make the changes, then boot into the first system that you installed, run update-grub there and then go back to the other system, at which point the changes have probably "stuck"
 

Well, cpuset didn't gave any great results. So far, I got better latencies by using isolcpus=

Hi tatel, I just built my system. I am running Debian 12 with no window manager, just twm and an xterm started with xinit. Without isolcpus and the default BIOS settings I was seeing spikes in jitter to 175.

I fixed the BIOS settings to turn off power management and it dropped down to about 25, but still had an occasional spike.

Then I set isolcpus to 2,3 (quad core machine) and ran latency-histogram again. Rock steady at 5 jitter.

Finally my question: I looked at ps and see that both latency-histogram and glxgears (I ran 6 copies of glxgears) are running on CPU 0. I don't recognize anything on CPU 2 or 3 as being related to latency-histogram. Should I be using taskset to make latency-histogram run on the isolated CPUs? And when I actually run linuxcnc, should I be using taskset to run that on the isolated CPUs?

Thanks,
Dan

Hi. I think you should at least have a try... and please let us know what works best for you.

With taskset on the latency-histogram process the jitter dropped to 0.6. I then moved the six glxgears to that CPU and jitter was 2. I kept adding glxgears until jitter rose to 3, it took four more for a total of ten glxgears.

So, I think taskset is a good thing. The latency-histogram seems to only use a single CPU, but I am guessing that linuxcnc may use more, but I have not tested that yet. I am going to let this run for a couple hours and then document it all (maybe on a more standard Debian install as not everyone grew up writing xinitrc's and running startx).

Thanks for your help, this thread was very helpful.