Ethercat random jitter fix

papagno-source wrote:

I have question at AI:

ethercat on debian 10 with kernel 4.19 and linuxcnc has less latency than debian 13 with kernel 6.12. Why?
 

Don't forget the hardware manufacturers have been adding  power saving features for years even back with Debian 10. But Linux simply ignored these features but now the kernel has caught up. It had to stay current. Generally, there are settings to disable them.  Also there are a lot of new network cards often part of the motherboard that do not have driver support until Debian 11 or 12 or 13.  So Debian 10 is an unsupported version now so you can;t continue to live in the past. Generally with proper tuning , 4 core and up hardware runs well on the later kernels.

4.19 was good. Remember I struggled with 6.x when it came out.
Anyway, what I do now is to have an Intel network card, i210, i217 or i225/i226
and make a modest effort to reduce jitter and latency.
The way I determine it's good enough is when there are no messages in syslog of this kind All such messages have to go, well one in the beginning is ok, but not more.

I may or may not need to go to isolcpus and so on. On my current plasma system I have no
such tweaks at all. It works good anyway.

I was thinking, can we ask the kernel developers, how can we recompile the kernel, eliminating all the power optimizations, so as to optimize it for RT applications? If we continue like this, soon Linux will only be used for writing emails and using OpenOffice.

papagno-source wrote:

I was thinking, can we ask the kernel developers, how can we recompile the kernel, eliminating all the power optimizations, so as to optimize it for RT applications? If we continue like this, soon Linux will only be used for writing emails and using OpenOffice.

There is no need to dive into the kernel. All of the power saving kernel options are already  set in my optimisations via the kernel boot options. You would just shorten your kernel boot string.

Grandixximo proved that a custom 6.3 kernel I built a long time ago no longer makes any difference now we understand how to configure a modern kernel.. Having run most  kernels since 5.10, latency has actually improved. My original J1900 from 8-10 years ago was at around 130k ns jitter is now down to about 50k or less when optimised.

What has changed is all the networking power settings that need to be carefully mitigated. The problem here is by definition networking was never designed to be real time, but to be a forwarding type of communication protocol through hundreds of switches around the internet. So network drivers are not real time.

You could write your own driver using the DDPK  www.dpdk.org/ which eliminates the latency from interrupt context switching and uses polling instead. But it still isn't real time. We are not doing High Frequency Trading where every millisecond of latency affects financial returns so this is not necessary in a CNC environment. 

You could use Time Sensitive Networking (TSN) which is now in the kernel and supported by Ethercat but its hardware dependent so your slave needs to support it.  tsn.readthedocs.io/

Remember PREMPT_RT is not fast, its deterministic (reliable and on time) which is exactly what we need for our servo thread.

I seems the initial delay is caused by lcec setting a bad initial reference time to the slaves.
Here is a listing of the first five lines, from lcec.write-all.
Data is cycle number, pll-err, reference clock time, application time lcec reads reference time of 3.53 second, and then sets it to 2.80 seconds. (lower 32 bits).
Now, speculating a bit, the clock is not changed right away, instead the clock is slowly changed
and the slave maintains a system time offset register 0x92c. It can then report and accurate
time by using the clock, that is off until in sync, and add the system time offset.
Must say it is speculation this though.

Anyway, the initial time setting in rt_app_main, or possibly the first few rounds in lcec_write should be looked at.
 

Good morning everyone. This is very likely the problem. I've noticed that sometimes the problem occurs on the first reboot. I start LinuxCNC, with automatic startup at system startup. If I restart LinuxCNC with the system running, the problem goes away, but sometimes it doesn't. Tomorrow I'll try to test Grandixximo's latest fix. If it doesn't work, I'll reverse the bus sequence, install the SLA and I/O last, make sure the drives are the first node, and modify HAL.