Ethercat random jitter fix

More
6 months 2 weeks ago - 6 months 2 weeks ago #345679 by papagno-source
Good morning everyone. Today I'll try the latest Grandixximo patch.
I can assure you that all PDOs, both on the I/O side of the first slave and on the drives, are read immediately. The same is true on another machine, where in addition to the drives there are two Delta inverters.

I tried running the following commands on Debian 10:
ethercat debug 1
start linuxcnc
exit or leave running, it doesn't matter.
sudo journalctl --since "2 min ago" | grep "Sync after"

no results.

uname -r 4.19.0-27-rt-amd64
dpkg -l | grep linuxcnc-uspace : no results
dpkg -l | grep ethercat: no results

I checked, and if I don't start axis, all the slaves are in PREOP.

The OP state and writing to the slave pins only occurs immediately after running addf lcec.write-all servo-thread in hal.

Just for completeness of information. On Debian 10, as well as on Trixie, I am using the "generic" device in the ethercat.conf file.
Last edit: 6 months 2 weeks ago by papagno-source.

Please Log in or Create an account to join the conversation.

More
6 months 2 weeks ago #345681 by rodw
Replied by rodw on topic Ethercat random jitter fix

edit:
tested with new kernel
uname -r
6.3.0-rt11-linuxcnc
same results, almost no effect
44sec for R2M
12sec for M2R
rule out kernel?
Can't be the kernel then, It was worth a try. When Debian 12 came out on the 6.1 kernel, we were all trying to work out why latency was so bad, but now the factors are well known so now we know updating the kernel no longer helps. (Trixie I have found is better than Bookworm)

@hakan, I read somewhere (maybe even on this thread) the 5000ms was just a waening, and not really something to be concerned about.

Please Log in or Create an account to join the conversation.

More
6 months 2 weeks ago #345682 by grandixximo
0x92c seem to correlate but in my testing not perfectly, I can still get quick OP with a big 0x92c
So it might not be the only factor, but I agree it is at least one of the relevant factors.
Smaller 0x92c is quicker OP, I think this has been true in my testing
The following user(s) said Thank You: endian

Please Log in or Create an account to join the conversation.

More
6 months 2 weeks ago #345683 by papagno-source
I have question at AI:

ethercat on debian 10 with kernel 4.19 and linuxcnc has less latency than debian 13 with kernel 6.12. Why?

reply , but only for curiosity :

This is a very insightful observation, and while it may seem counterintuitive (new = better?), in the world of Real-Time (RT) and numerical control (NC) like LinuxCNC, "old and stable" often trumps "new and complex."

The move from a 4.19 kernel (Debian 10) to a 6.12 kernel (Debian 13) introduces several layers of complexity that can degrade deterministic latency. Here are the main reasons:
1. Security Mitigations (The "Price" of CPUs)

Since the 4.19 kernel era, numerous hardware vulnerabilities have been discovered in CPUs (Spectre, Meltdown, L1TF, MDS, etc.).

Kernel 4.19: Many of these mitigations were not yet implemented or were less invasive.

Kernel 6.12: Includes a comprehensive suite of security patches that add overhead to every syscall and every switch between user-space and kernel-space. This increases jitter and maximum latency.

2. Kernel RT Integration (Preempt-RT)

Kernel 4.19: The PREEMPT_RT patch was a very mature set of external changes focused exclusively on pure real-time performance.

Kernel 6.12: Much of the real-time code has now been integrated into the Mainline Kernel. While this is a huge technical achievement, the overall integration must make compromises with other modern kernel features, making it sometimes less "extreme" than dedicated patches of the past.

3. Power Management (C-States and P-States)

Newer kernels are optimized for power efficiency.

Kernel 6.12 aggressively attempts to put CPU cores into power-saving states or vary the frequency.

Waking a CPU from a deep "C-state" requires precious microseconds, which, in EtherCAT communication (which often runs at 1kHz or more), translates into latency spikes.

4. EtherCAT Driver (IgH or SOEM)

The EtherCAT master (usually the IgH EtherCAT Master) must be patched to work with each kernel version.

On older kernels like 4.19, "native" drivers (those that replace the standard network card drivers, e.g., ec_8139too) are extremely stable and fast.

On very recent kernels, you're often forced to use the "generic" driver, which runs through the standard Linux network stack, inevitably adding latency compared to an optimized native driver.

5. Operating System Complexity

Debian 13 loads many more background services, more complex systemd versions, and more intensive D-Bus management than Debian 10. Even though LinuxCNC runs at high priority, background operating system noise can affect determinism.
How can I improve the situation on Debian 13?

If you absolutely must use the latest version, you can try "neutering" modern kernel features to regain speed:

Disable mitigations: Add mitigations=off to the GRUB boot parameters. (Warning: This reduces PC security.)

Core isolation: Use isolcpus=X (where X is the core dedicated to LinuxCNC) to prevent the operating system from using it for generic processes.

Idle=poll: Add idle=poll or intel_idle.max_cstate=0 to prevent the CPU from "sleeping."

Check Drivers: Make sure you don't use the generic EtherCAT driver if your network card supports an updated native driver for the 6.x kernel.

In summary: Debian 10 with kernel 4.19 is a "minimalist" and hardware-friendly combination. Debian 13 is a modern, hardened system optimized for power savings—the exact opposite of what's needed to drive a brushless motor in real time.

Please Log in or Create an account to join the conversation.

More
6 months 2 weeks ago - 6 months 2 weeks ago #345684 by grandixximo
We are aware, but eventually 4.19 will not support new hardware, and eventually old hardware goes out of production, so we need to work on making things run.

mitigations=off is probably the most interesting hypothesis, worth a try, probably effect latency, I doubt it effects OP time.

 

File Attachment:

File Name: journals.zip
File Size:165.58 KB


I got a journal of r2m vs m2r
@hakan if the initial time from lcec is the problem, what could be a solution?
Initial time is dictated by the addf, the addf timing IMO has always been an issue, is your intuition also pointing that way? Or something different?

edit:
with my v1.41.0-pre1 from my github
github.com/grandixximo/linuxcnc-ethercat...ses/tag/v1.41.0-pre1
I get r2m I get OP in 6s and m2r in 3s
 

File Attachment:

File Name: journal-1.41pre.zip
File Size:74.91 KB
Last edit: 6 months 2 weeks ago by grandixximo.

Please Log in or Create an account to join the conversation.

More
6 months 2 weeks ago #345685 by papagno-source
I have test with driver e1000e in ethercat.conf , the problem is the same

Please Log in or Create an account to join the conversation.

Time to create page: 0.141 seconds
Powered by Kunena Forum