Latency test

NTULINUX wrote:

Did you upgrade to bookworm/testing or sid?

 

Tried this today - 35-40k, no result on isolcpus or others params. Looks like its seem to much extra hardware or like. Did it hard way - bookworm clean install and apt getlinuxcnc. Or it was not very optimised/clean ?

Reverted back to 2.8.4 buster. Rock solid. Firefox / youtube all day. Wifi card present. 14k..

Starting to like this little Asrock.

forum.linuxcnc.org/18-computer/46946-mem...k-j3355b-mitx#252489

Will try this one setup too.

Ok.. Two day testing while doing other stuff.

Solved problem with bookworm - all that it needed was nomodeset to perform similar to buster. On uspace version.

Also - C states must be set in one place only - or in bios or in kernel paramteres. Then they give similar results. I have more then one setup to get 10-11K result by now. Goal was to find good solotion without idle=pull.

Today will try Mint version.

Find more interesting option to use isolcpus :

isolcpus=nohz,domain,1

some short testing, but this one + idle=poll gives sub 10k result with some glxgears running. And irq list shows cpu beeing isolated from all, but timer requests. And you really have to pin it to get higher latency.

And nohz have to be used on same core as isolcpus.

stackoverflow.com/questions/60322119/com...-in-modern-linux-5-0

lwn.net/Articles/733964/

Working on shape of graph. I do not like regularities, spikes - it shows that our cpu is not isolated or we have some processe running in a wrong time and place. It was wery uneven on Debian, much better on Mint.
After one zilion of tests and some manuals digging and nohz=off, rcb tune - i am getting two spike on both graph, if lightly loaded (cpu 0 have some spare time to do housekepping) latency is in 6-7k. 
Load it harder - 10-12k.

Add some nasty things like putting new usb wifi, apt install, new firefox with yt - then system barely moves - it is in 16k region.

"Last" bug found. I have noticed - one can get better latency on slightly loaded system. Its stays in 6K, as i have mentioned before. Sometimes even less.. But, after longer testing - it still climbs sometimes into 10 or 15k.

Problem was Cpu frequency. Even, after setting disable to Speed step technology - it only disable "turbo" - 2.5Ghz mode on cpu. Cores still tries to adapt to load. Normally it was like 1.5 GHz for core0 and 800MHz for core1 in idle state. On latency tests - both jumped to 2GHz. And system load was varied from 0 to 100 - latence was jumping as hell.

After long play with cpufreq-set - it was enought to set both cores to performance governor and it was all good with just isolcpus=1 in grub parameters. And - nohz settings or rbu settings - boosted core0 to 2GHz, but governor was still working and it tries to change frequency.

Would you be interested in trying my Gentoo image + installer? I'm working on a custom built Linux system with the entire userspace + kernel tuned specifically for low-latency, no speed step, no power management, no sleep/hibernate/suspend, no power save etc.. Lots of things stripped out. Debian and other distros only lets you go so far, so I've taken the initiative to try and build the best possible image for LinuxCNC and lowest latency. All that governor/power/CPU stuff should not even exist on a real-time kernel.

NTULINUX wrote:

Would you be interested in trying my Gentoo image + installer? I'm working on a custom built Linux system with the entire userspace + kernel tuned specifically for low-latency, no speed step, no power management, no sleep/hibernate/suspend, no power save etc.. Lots of things stripped out. Debian and other distros only lets you go so far, so I've taken the initiative to try and build the best possible image for LinuxCNC and lowest latency. All that governor/power/CPU stuff should not even exist on a real-time kernel.
 

Hi ! Yes, i could try, but i can bet it will not help. Reason - this small J3355 (And other modern cpu's according to literature) try be as green as possible. Or it is a bios problem / bug.

One example on intel_pstate / intel_cpufreq drivers. From 5.7 linux kernel version - intel_pstate is by default in passive mode, getting commands from intel_cpufreq. If you your disable it - it go to some weird mode in new linuxes - it does not got any control signals from kernel. but it still try to use different C modes and it changes core voltage too.

Now if you disable other C modes except C1 - cpu ignore this setting and is using all of them. Again - could be a bug in a bios. But - setting is working if you change max C state. And no reaction on kernel setting.

On kernel 6.0 you need to add intel_pstate = active if you want to have direct control of cpu using intel_pstate controller. Also - you have to enable speed_step in bios.

So - now i am using testing following setups:

 isolcpus=1, intel_pstate=active and setting governor=performance, max cpu frequency 2000000 . Bios C1, enabled speedstep.
 isolcpus=1, intel_pstate=passive and setting governor=performance max cpu frequency 2000000. Bios C1, enabled speedstep.
 isolcpus=1, intel_pstate=disable . Bios C1, disabled speedstep.

Cpu voltage is stable if performance mode set or disabled speedstep, C states goes from C0 to C1. Which is kind of normal. If intel_cpufreq is used - and governor is userspace and frequency is setted - it does not work.

All got ~ similar results. idle=poll just moves C1 to C0, and gives worse results.

And one need i7z package to get real cpu frequency. One need to watch cpu voltage - if it stable its not jumping frequency. C0 / C1 states can change, but not C3 or deeper.

Still - i can run modern kernel with 10 watt tdp. My 400/230 volts source (for computer with monitor) is limited at 50 watt at my lathe...  So no power hungry X2 athlons or older Celerons are considered.


 

Will add this to Mint thread too. But - removing apparmor helped on test scenario with many running of glxgears. Now it look to be definitly in sub 10k latency region. intel_pstate=active, performance mode, isolcpus=1 and C1 enabled in bios.