Ethercat random jitter fix

rodw wrote:

It makes no sense to roll back to a version built for  kernel 2.6 (Trixie's is 5.12) that's not been touched for 7 years...

You mean 6.12 for sure, 5.12 is also pretty old.

Fair point on the age of the base, but it is not as stale as the "1.5.2" label makes it look. YangYang's build sits on the 1.5.2 API boundary, but a lot has been grafted on top of it. Martin Ribalda's refactor moved the slave scan, slave config, SDO request, dictionary, and state-change FSMs into per-slave state machines running off externally supplied datagrams, which means every slave progresses through INIT to PREOP to SAFEOP to OP in parallel instead of the master FSM walking them one at a time. That is where the OP speedup comes from, not the age of the base. On top of that there is an SDO injection path, so CoE transfers ride on the realtime cycle alongside PDOs and startup list SDOs plus PDO mapping writes do not each cost a full master FSM round trip. The 1.6 era API surface is mostly backported too (ecrt_master_sync_reference_clock_to, mbox gateway, rescan, RT slave request hooks, full FoE request API, reboot FSM, bulk register read/write, SII from file, and so on).

On the kernel side, the native drivers have been updated for modern kernels. The devices/ tree ships 6.12 variants of igb, igc, r8169, genet, e1000, e1000e, and stmmac, and DKMS builds cleanly against Trixie's 6.12 headers.

I tried the obvious "just tune 1.6.9" route first. Applying the same constants YangYang's build uses (EC_WAIT_SDO_DICT=0 and EC_SKIP_SDO_DICT=1) on top of stock 1.6.9 drops OP from around 15 s to around 12 s. YangYang's build reaches OP in roughly 5 s on the same bus. The remaining gap is not a constant that anyone can tune; it is the per-slave FSM architecture, and 1.6.9 does not have it. 1.7 does not have it either, I checked, master/fsm_slave.c is still 685 lines in 1.7, the same as in 1.6.9, against 1339 lines in YangYang's build.

So "use the newer version" is the intuitive answer but it does not actually close the gap. Porting Ribalda's refactor cleanly onto 1.6.9 is weeks of work with a large regression surface. Shipping YangYang's build with modern Debian packaging gets working fast OP debs this week. Different tradeoff, not a rollback.

OP transition time is of course unrelated to grinding itself, this is about how long the user waits for the machine to come up, not motion behavior.

Good morning everyone, and sorry if I'm less present, but technical support won't let me go. I personally think we need to focus on functional solutions; it's pointless to use a LinuxCNC, which is very modern but doesn't work. So I agree with Grandixximo. In the next few days, I'll do the tests Grandixximo asked me to do and I'll update you. I'll ask the customer if he'll let me do further tests outside of business hours. The machine is working properly with Debian 10. I remind you that I tried overwriting the lcec_conf and lcec.so files from Debian 10 to Trixie, but I didn't see any improvements. That's why I believe the problem, aside from those found by Grandixximo and fixed, isn't Lcec, but rather Ethercat and the kernel, which has high jitter for a CNC machine. The solutions are to use a PC suitable for RT or design a Master Hw that has a dual Port RAM, capable of buffering information and letting the PC breathe.

Sorry I don't buy the excessive latency with Trixie today vs Debian 10, which was in the  2016 to   2017  era back when I started. That's 10 years ago. There was no PREEMPT_RT kernel available at that time. In fact, you could have been running the RTAI kernel. Back in 2017, the path I followed that was subsequently documented by cncbasher was to Install Linux Mint 17.x or 18.x and compile the PREEMPT_RT kernel. Ref:  forum.linuxcnc.org/9-installing-linuxcnc...linuxcnc-from-source . Not long after this somebody made an ISO on Debian (10) Stretch which included PREEMPT_RT. Eventually, there was an official stretch release.

Bullseye (Debian 11) was released and that obsoleted Python 2.7 which we depended on. That was such a big stumbling block we did not Release anything on Bullseye but I used it to get network drivers for my Odroid H2+ instead of compiling from source. I had to give up Mint at this time because their kernels were too dated to have my drivers. I will never use anything other than Debian now! There was some big changes to the linux kernel network drivers in the 5.9 kernel pending the release of Bullseye on Kernel 5.10. I had one Ethercat machine running Bullseye (11) and moved to Bookworm (12) and got stuck on all PC's I had with Error Finishing Read on Mesa hardware but never had issues  with Ethercat in my bench testing. This is when I started working on kernel tuning and I found that my Old 2017 PC (Celeron J1900) had significantly improved jitter on Bookworm then it did back in the day. (eg reduced from 130k to 40k or lower). I don't think Trixie would be any worse. In fact other PC's I've tuned are running at almost half the latency now before tuning. Here is an example from Mike after he applied my optimisations

 

This is way below minimum latency requirements for Ethercat. There is no doubt in my mind that any issues being discussed here is due to  misconfiguration or issues in the Ethercat environment. Please forget all about latency. It is not an issue if the correct drivers are deployed and the appropriate network tuning has been done (Which it is for papagno-source's PC's). 

 

The comparison I'm making is about latency between Debian 10 with Kernel 4.19-27-rt and Debian Trixie Kernel 6.12.74-RT. Although Trixie's Grub has different optimization strings and Debian 10 has no optimizations at all, Debian 10 has half the jitter of Trixie on the same PC. Debian 10 has a jitter of 40 microsec and Trixie 85 microsec. So, since I want to do a further test with a new PC, can you advise me on which motherboard and processor would have the lowest jitter?

The comparison I'm making is about latency between Debian 10 with Kernel 4.19-27-rt and Debian Trixie Kernel 6.12.74-RT. Although Trixie's Grub has different optimization strings and Debian 10 has no optimizations at all, Debian 10 has half the jitter of Trixie on the same PC. Debian 10 has a jitter of 40 microsec and Trixie 85 microsec. So, since I want to do a further test with a new PC, can you advise me on which motherboard and processor would have the lowest jitter?