Performance issues after switching from LCNC 2.8.4 Buster to 2.9.3 Bookworm

For use with Mesa boards, there is no need for base period, so do latency tests with:
latency-histogram --nobase --sbinsize 1000 --show
Some RealTek network cards have issues with 6.x kernels, Rod has some nice documentation on how to fix it.
Some Intel network cards have coalescing enabled, look for PCW's howto for setting it to 0.
If using EtherCAT, see Rod's and some other members wealth of info on this forum.
-
Sorry for the wague answers, but on the phone in a hurry.

Thanks Rod, I find it rather unfortunate that somebody like you does not seem be offered write access to the 'official' install documentation. The pull request process is so sluggish that by the time new install docs might actually make it through they're already outdated.

Hi Rod,

I went through your document. My NIC is identified as:

edmuser@edmpc:~$ lspci -k | grep -A 3 -i ethernet
02:00.0 Ethernet controller: Realtek Semiconductor Co., Ltd. RTL8111/8168/8411 PCI Express Gigabit Ethernet Controller (rev 15)
Subsystem: IP3 Tech (HK) Limited RTL8111/8168/8211/8411 PCI Express Gigabit Ethernet Controller
Kernel driver in use: r8168
Kernel modules: r8168

I updated the driver according to the document and rebooted. Now, I would like to add the settings r8168.aspm=0 r8168.eee_enable=0 pcie_aspm=off loglevel=3
If I add r8168.aspm=0 or r8168.eee_enable=0 then the grub customizer does not allow me to save, throwing a "not found" error. I was able to save the pcie_aspm=off, loglevel=3 and quiet isolcpus=2,3 settings.

I will run tests to see if the new driver and the settings I did manage to save already fixes my issue, but since you suspect that the energy efficient ethernet might be a root cause of the problem, I really want to set that to off, but can't.

Yes! The problem is gone, even without the r8168.aspm=0 and r8168.eee_enable=0 settings. Before I also tried quiet isolcpus=2,3 on its own, and although the latency improved, it did not resolve the EtherCAT servo following errors.

From what I can conclude now is that updating the realtech drivers according to Rodw's document did the trick.

Some additional info. For my implementation, the EtherCAT reading and writing combined takes 14% of the 1 ms servo loop time.
To check that you can get the number of CPU ticks for each with:
halcmd show param lcec.read-all.tmax
halcmd show param lcec.write-all.tmax
Add those together and multiply by 1/CPU frequency, that gives you the total time for each servoloop iteration that EtherCAT requires.

Now I can finally write a detailed update/installation procedure for my customers and release my new EDM components.
Many thanks again! Great way to start the weekend

Aciera wrote:

Thanks Rod, I find it rather unfortunate that somebody like you does not seem be offered write access to the 'official' install documentation. The pull request process is so sluggish that by the time new install docs might actually make it through they're already outdated.

I have tried to keep my doc current but I have not installed on a problem PC for quite a while. It has been suggested the install ISO should install the R8168 driver as it should not get in the road. Not sure on that. (I built the ISO and the Pi images for Linuxcnc).

At one stage I was asked to edit the getting Linuxcnc docs and it included most of my documentation notes you refer to. This was accepted and was published in the 2.9 docs. However, when 2.9.1 was released, this was all blown away and replaced with content based on 2.8 docs.

I think there should be a new document added that might be called something like "System Optimisation for Linuxcnc", perhaps a seperate doc for the raspberry Pi it links to. If somebody was to have a go at that from my material, I would not have a problem. Asciidoc and I are not good friends.
I almost know enough about git to be trusted with write access... but time is an issue

Ok, I got a setup running with 2 EtherCAT drives without problems. I've now added a third EtherCAT drive and the problem has returned, occurring about once every hour or so. I check the tmax write plus the tmax read for the lcec which combined to max 23.5% of my servo loop time.
I would really like to turn off the r8168.aspm=0 or r8168.eee_enable=0 functions to see if that helps as rodw suggested, but I do not know how. The grub customizer does not allow me to save those, throwing a "not found" error.