Latency test
- Stanislavz
- Offline
- Senior Member
-
Less
More
- Posts: 50
- Thank you received: 15
03 Jan 2023 13:48 #260840
by Stanislavz
Replied by Stanislavz on topic Latency test
Tried this today - 35-40k, no result on isolcpus or others params. Looks like its seem to much extra hardware or like. Did it hard way - bookworm clean install and apt getlinuxcnc. Or it was not very optimised/clean ?Did you upgrade to bookworm/testing or sid?
Please Log in or Create an account to join the conversation.
- Stanislavz
- Offline
- Senior Member
-
Less
More
- Posts: 50
- Thank you received: 15
03 Jan 2023 20:47 #260870
by Stanislavz
Replied by Stanislavz on topic Latency test
Reverted back to 2.8.4 buster. Rock solid. Firefox / youtube all day. Wifi card present. 14k..
Starting to like this little Asrock.
Starting to like this little Asrock.
Please Log in or Create an account to join the conversation.
- Stanislavz
- Offline
- Senior Member
-
Less
More
- Posts: 50
- Thank you received: 15
03 Jan 2023 21:41 #260873
by Stanislavz
Replied by Stanislavz on topic Latency test
Please Log in or Create an account to join the conversation.
- Stanislavz
- Offline
- Senior Member
-
Less
More
- Posts: 50
- Thank you received: 15
06 Jan 2023 16:07 - 06 Jan 2023 16:18 #261094
by Stanislavz
Replied by Stanislavz on topic Latency test
Ok.. Two day testing while doing other stuff.
Solved problem with bookworm - all that it needed was nomodeset to perform similar to buster. On uspace version.
Also - C states must be set in one place only - or in bios or in kernel paramteres. Then they give similar results. I have more then one setup to get 10-11K result by now. Goal was to find good solotion without idle=pull.
Today will try Mint version.
Solved problem with bookworm - all that it needed was nomodeset to perform similar to buster. On uspace version.
Also - C states must be set in one place only - or in bios or in kernel paramteres. Then they give similar results. I have more then one setup to get 10-11K result by now. Goal was to find good solotion without idle=pull.
Today will try Mint version.
Last edit: 06 Jan 2023 16:18 by Stanislavz.
The following user(s) said Thank You: royka
Please Log in or Create an account to join the conversation.
- Stanislavz
- Offline
- Senior Member
-
Less
More
- Posts: 50
- Thank you received: 15
06 Jan 2023 22:54 - 06 Jan 2023 22:56 #261141
by Stanislavz
Replied by Stanislavz on topic Latency test
Find more interesting option to use isolcpus :
isolcpus=nohz,domain,1
some short testing, but this one + idle=poll gives sub 10k result with some glxgears running. And irq list shows cpu beeing isolated from all, but timer requests. And you really have to pin it to get higher latency.
And nohz have to be used on same core as isolcpus.
stackoverflow.com/questions/60322119/com...-in-modern-linux-5-0
lwn.net/Articles/733964/
isolcpus=nohz,domain,1
some short testing, but this one + idle=poll gives sub 10k result with some glxgears running. And irq list shows cpu beeing isolated from all, but timer requests. And you really have to pin it to get higher latency.
And nohz have to be used on same core as isolcpus.
stackoverflow.com/questions/60322119/com...-in-modern-linux-5-0
lwn.net/Articles/733964/
Last edit: 06 Jan 2023 22:56 by Stanislavz.
The following user(s) said Thank You: tommylight
Please Log in or Create an account to join the conversation.
- Stanislavz
- Offline
- Senior Member
-
Less
More
- Posts: 50
- Thank you received: 15
07 Jan 2023 21:27 #261202
by Stanislavz
Replied by Stanislavz on topic Latency test
Working on shape of graph. I do not like regularities, spikes - it shows that our cpu is not isolated or we have some processe running in a wrong time and place. It was wery uneven on Debian, much better on Mint.
After one zilion of tests and some manuals digging and nohz=off, rcb tune - i am getting two spike on both graph, if lightly loaded (cpu 0 have some spare time to do housekepping) latency is in 6-7k.
Load it harder - 10-12k.
Add some nasty things like putting new usb wifi, apt install, new firefox with yt - then system barely moves - it is in 16k region.
After one zilion of tests and some manuals digging and nohz=off, rcb tune - i am getting two spike on both graph, if lightly loaded (cpu 0 have some spare time to do housekepping) latency is in 6-7k.
Load it harder - 10-12k.
Add some nasty things like putting new usb wifi, apt install, new firefox with yt - then system barely moves - it is in 16k region.
Please Log in or Create an account to join the conversation.
- Stanislavz
- Offline
- Senior Member
-
Less
More
- Posts: 50
- Thank you received: 15
09 Jan 2023 11:23 #261344
by Stanislavz
Replied by Stanislavz on topic Latency test
"Last" bug found. I have noticed - one can get better latency on slightly loaded system. Its stays in 6K, as i have mentioned before. Sometimes even less.. But, after longer testing - it still climbs sometimes into 10 or 15k.
Problem was Cpu frequency. Even, after setting disable to Speed step technology - it only disable "turbo" - 2.5Ghz mode on cpu. Cores still tries to adapt to load. Normally it was like 1.5 GHz for core0 and 800MHz for core1 in idle state. On latency tests - both jumped to 2GHz. And system load was varied from 0 to 100 - latence was jumping as hell.
After long play with cpufreq-set - it was enought to set both cores to performance governor and it was all good with just isolcpus=1 in grub parameters. And - nohz settings or rbu settings - boosted core0 to 2GHz, but governor was still working and it tries to change frequency.
Problem was Cpu frequency. Even, after setting disable to Speed step technology - it only disable "turbo" - 2.5Ghz mode on cpu. Cores still tries to adapt to load. Normally it was like 1.5 GHz for core0 and 800MHz for core1 in idle state. On latency tests - both jumped to 2GHz. And system load was varied from 0 to 100 - latence was jumping as hell.
After long play with cpufreq-set - it was enought to set both cores to performance governor and it was all good with just isolcpus=1 in grub parameters. And - nohz settings or rbu settings - boosted core0 to 2GHz, but governor was still working and it tries to change frequency.
Please Log in or Create an account to join the conversation.
09 Jan 2023 21:48 #261385
by NTULINUX
Replied by NTULINUX on topic Latency test
Would you be interested in trying my Gentoo image + installer? I'm working on a custom built Linux system with the entire userspace + kernel tuned specifically for low-latency, no speed step, no power management, no sleep/hibernate/suspend, no power save etc.. Lots of things stripped out. Debian and other distros only lets you go so far, so I've taken the initiative to try and build the best possible image for LinuxCNC and lowest latency. All that governor/power/CPU stuff should not even exist on a real-time kernel.
The following user(s) said Thank You: tommylight, Stanislavz
Please Log in or Create an account to join the conversation.
- Stanislavz
- Offline
- Senior Member
-
Less
More
- Posts: 50
- Thank you received: 15
10 Jan 2023 09:19 #261429
by Stanislavz
One example on intel_pstate / intel_cpufreq drivers. From 5.7 linux kernel version - intel_pstate is by default in passive mode, getting commands from intel_cpufreq. If you your disable it - it go to some weird mode in new linuxes - it does not got any control signals from kernel. but it still try to use different C modes and it changes core voltage too.
Now if you disable other C modes except C1 - cpu ignore this setting and is using all of them. Again - could be a bug in a bios. But - setting is working if you change max C state. And no reaction on kernel setting.
On kernel 6.0 you need to add intel_pstate = active if you want to have direct control of cpu using intel_pstate controller. Also - you have to enable speed_step in bios.
So - now i am using testing following setups:
isolcpus=1, intel_pstate=active and setting governor=performance, max cpu frequency 2000000 . Bios C1, enabled speedstep.
isolcpus=1, intel_pstate=passive and setting governor=performance max cpu frequency 2000000. Bios C1, enabled speedstep.
isolcpus=1, intel_pstate=disable . Bios C1, disabled speedstep.
Cpu voltage is stable if performance mode set or disabled speedstep, C states goes from C0 to C1. Which is kind of normal. If intel_cpufreq is used - and governor is userspace and frequency is setted - it does not work.
All got ~ similar results. idle=poll just moves C1 to C0, and gives worse results.
And one need i7z package to get real cpu frequency. One need to watch cpu voltage - if it stable its not jumping frequency. C0 / C1 states can change, but not C3 or deeper.
Still - i can run modern kernel with 10 watt tdp. My 400/230 volts source (for computer with monitor) is limited at 50 watt at my lathe... So no power hungry X2 athlons or older Celerons are considered.
Replied by Stanislavz on topic Latency test
Hi ! Yes, i could try, but i can bet it will not help. Reason - this small J3355 (And other modern cpu's according to literature) try be as green as possible. Or it is a bios problem / bug.Would you be interested in trying my Gentoo image + installer? I'm working on a custom built Linux system with the entire userspace + kernel tuned specifically for low-latency, no speed step, no power management, no sleep/hibernate/suspend, no power save etc.. Lots of things stripped out. Debian and other distros only lets you go so far, so I've taken the initiative to try and build the best possible image for LinuxCNC and lowest latency. All that governor/power/CPU stuff should not even exist on a real-time kernel.
One example on intel_pstate / intel_cpufreq drivers. From 5.7 linux kernel version - intel_pstate is by default in passive mode, getting commands from intel_cpufreq. If you your disable it - it go to some weird mode in new linuxes - it does not got any control signals from kernel. but it still try to use different C modes and it changes core voltage too.
Now if you disable other C modes except C1 - cpu ignore this setting and is using all of them. Again - could be a bug in a bios. But - setting is working if you change max C state. And no reaction on kernel setting.
On kernel 6.0 you need to add intel_pstate = active if you want to have direct control of cpu using intel_pstate controller. Also - you have to enable speed_step in bios.
So - now i am using testing following setups:
isolcpus=1, intel_pstate=active and setting governor=performance, max cpu frequency 2000000 . Bios C1, enabled speedstep.
isolcpus=1, intel_pstate=passive and setting governor=performance max cpu frequency 2000000. Bios C1, enabled speedstep.
isolcpus=1, intel_pstate=disable . Bios C1, disabled speedstep.
Cpu voltage is stable if performance mode set or disabled speedstep, C states goes from C0 to C1. Which is kind of normal. If intel_cpufreq is used - and governor is userspace and frequency is setted - it does not work.
All got ~ similar results. idle=poll just moves C1 to C0, and gives worse results.
And one need i7z package to get real cpu frequency. One need to watch cpu voltage - if it stable its not jumping frequency. C0 / C1 states can change, but not C3 or deeper.
Still - i can run modern kernel with 10 watt tdp. My 400/230 volts source (for computer with monitor) is limited at 50 watt at my lathe... So no power hungry X2 athlons or older Celerons are considered.
Please Log in or Create an account to join the conversation.
- Stanislavz
- Offline
- Senior Member
-
Less
More
- Posts: 50
- Thank you received: 15
10 Jan 2023 13:10 #261440
by Stanislavz
Replied by Stanislavz on topic Latency test
Will add this to Mint thread too. But - removing apparmor helped on test scenario with many running of glxgears. Now it look to be definitly in sub 10k latency region. intel_pstate=active, performance mode, isolcpus=1 and C1 enabled in bios.
Please Log in or Create an account to join the conversation.
Time to create page: 0.189 seconds