Latency Issues - Histogram Shape

Hi all,

I'm new to Lcnc, and have been for the past week trying to get my PC to hit the right latency levels.
After a week i've managed to get the base thread down from 126us to about 90us (worst case) which I've read is okay if you're using it with ethercat.

However I've noticed most of the graphs, especially for the servo period Histogram Latency graph that the shapes that most users have is a single peak. I've always been getting 3 peaks.

I'd like to confirm whether having this type of histogram shape is normal? or whether this shape indicates further sources of latency to be reduced. 

For Background:

Setup:QBiX-GLKB4125-A1 Mini -PC  J4125 Processor

I've grubbed out all i can think of to help the latency with a small amount of success to when it was out of the box

GRUB_CMDLINE_LINUX_DEFAULT="quiet splash kthread_cpus=0,1 irqaffinity=0,1 rcu_nocbs=2,3 nohz_full=2,3 isolcpus=2,3 rcu_nocb_poll nohz=on intel_pstate=disable amd_pstate=disabled amd_idle.max_cstate=1  processor.max_cstate=0 idle=halt skew_tick=1 idle=poll cpufreq.off=1 cpuidle.off=1 intel_idle.max_cstate=0 nowatchdog nosoftlockup igb.EEE=0 apm=off nosmt=force acpi_irq_nobalance i915.enable_rc6=0 i915.enable_dc=0 i915.enable_psr=0 i915.enable_fbc=0 drm_kms_helper.poll=0 pcie_aspm=off r8168.aspm=0 r8168.eee_enable=0 mitigations=off pcie_aspm=off hpet=disable"

got the governor running on performance, and .sh to set the IRQ affinities.

disabled stuff in the bios

updated drivers via the terminal but haven't yet tried updating the NIC,Graphics, and Chipset drivers from the manufacturer.

Any help would be much appreciated, as I'm new to linux. Thanks. 

Sometimes the shape is because of the number of sbins samples. You have 20 mand most use 1000

try
latency-histogram --nobase --sbins 1000

My observation is that Ethercat is faster than Mesa hm2eth. 90 usec will be fine If your pings are good

I can see min and max around 70 us. I do have this numbers all the time with a lot of lenovo thinkcentres tinys (m73, m93, m710q etc.) without having troubles.

As long as you do not have spikes in the range of ms (realtek network issues) or added delays like the intel network delays everthing is fine. As long as you are happy with 1 ms servo period of course. Ethercat runs in the servothread, so 1 ms is totally ok.

I always make a plot with the hal scope of the hm read time (if we are talking about mesa cards and added ethercat) to visually see everything. One can see network times and jitter on the fly while doing things (gears, surfing, intel nic delay playaround etc.). Just keep in mind that you see cpu ticks and you have to devide the numbers (100k to 400k range base and peaks little higher caused by jitter) by the cpu tact.

tommylight wrote:

Rod, he has huge excursions on both sides of the graph, those are not shown unless using --show, and those are the ones causing latency errors and link drop.

Of course there are excursions. That's because the histogram is only showing a window of 20 usec when the range is 70 usec over a 20 minute period. Whether you use --show or --sbins 1000 won't matter because either will show the full histogram. This is a case of how to lie with statistics with an inappropriate graph scale.