Ethercat random jitter fix

For somebody trying hard to follow along from the sidelines, can you guys outline the objective for rolling back to 1.52 here?
Has anybody looked at the Igh Ethercat master 1.6 or even 1.7 which was released on gitlab 3 weeks ago (but not in their repositories) to see if it has these delays outside of linuxcnc.

It seems to me it would be unlikely that iGh master 1.6 or 1.7 are problematic as people would be complaining at iGh so to me,  the effort should be in understanding what has changed that linuxcnc-ethercat is not liking. Rolling back to a very old version does not seem a progressive step following the synchronisation enhancement.

Hakan wrote:

Tested a bit.
I get fairly consistent loading times of lcec's rt_app_main() function of 1.85-1.90 seconds.
But but. In syslog I can see that the initial sync of slaves takes anywhere between zero and
say 3,4,5 seconds. 
To check
ethercat debug 1
start linuxcnc
exit or leave running, doesn't matter.
sudo journalctl --since "2 minutes ago" | grep "Sync after"
and see where it stops for every slave.
I reckon this is the problematic delay time.
Seldom, but it happens, I get "Checking for synchrony" and then it is satisfied directly and moves on.

It seems measuring the time of lcec's rt_app_main is not a good indicator.
Is there a good way of measuring time to OP?

trixie, 
Linux plasma 6.12.69+deb13-rt-amd64 #1 SMP PREEMPT_RT Debian 6.12.69-1 (2026-02-08) x86_64 GNU/Linux
IgH EtherCAT master 1.6.8 1.6.8.g2543cc5-1+27.3  (via apt)
linuxcnc-ethercat, a slightly modified version of commit bf13577ec7b1b54844e07a80d949232e71776b32
linuxcnc 2.9.4 (via apt)

At the end of my hal file I and I save the default watch to display the OP pins, you can have master all-op, and each slave individual OP pin, I count Mississippies or use stopwatch...
Another alternative is But this might "dirty" the bus, may effect OP time, maybe...
To get more accurate times might have to build a script, or use dmesg with some debug from ethercat.
But I think the difference papagno-source is talking about can be perceived with less precise tooling, in one case everything OP even before the UI opens, and in the other he has to wait some amount of time, so it is a difference in the seconds range.
I will try to setup similar setup to yours, and check the OP times.

rodw wrote:

For somebody trying hard to follow along from the sidelines, can you guys outline the objective for rolling back to 1.52 here?
Has anybody looked at the Igh Ethercat master 1.6 or even 1.7 which was released on gitlab 3 weeks ago (but not in their repositories) to see if it has these delays outside of linuxcnc.

It seems to me it would be unlikely that iGh master 1.6 or 1.7 are problematic as people would be complaining at iGh so to me,  the effort should be in understanding what has changed that linuxcnc-ethercat is not liking. Rolling back to a very old version does not seem a progressive step following the synchronisation enhancement.

Did it on pagagno-source request

papagno-source wrote:

Why don't we try installing Ethercat 1.5.2 on Trixie, so as to understand if the problem is on the kernel or Ethercat side.

This is just for testing, I pretty much knew it was a dead end when I started on this path, but it is just so we are all on the same page, I need to bring receipts, I won't simply hand wave his concerns, he wanted 1.5.2 on Trixie so that he will be convinced that is not the issue, and I gave him that. I don't want to come back to this point over and over again as we proceed with more testing, once it is proven that 1.5.2 does not solve the issue, we can move on to the next thing. Just trying to respect other people's perspective, call it an unnecessary waste of time if you want, I can see where you are coming from, but I feel the best approach is moving systematically one step at the time, clearing up all concerns, testing thoroughly, find agreement, move to the next thing.

1.6 and 1.7 present the same OP time delays in my testing.

Thanks for clarifying. I guess if 1.5.2 does not solve the issue, the next question is has linuxcnc-ethercat changed since Debian 10? I suspect not but can you review and confirm? If nothing has changed, its a Debian kernel issue

People forget that modern kernels post 5.9 and up  (which is well after Debian 10) are enormously complex and the hardware developers have been adding  features for years (mostly power saving) that the linux kernel unfortunately now supports. So there are a lot of tuning steps now. Perhaps there are some issues we have not covered in our known NIC tunings.

I've shared a 6.3 kernel I built in the past on Discord which many people had good results with. Perhaps it will help. It will be a few days before I can build a later kernel for Trixie.


 

I have done testing with the above, this is Debian 13 everything installed from repos.
Generic drive no ec_igc
20 devices
9 servos DC
1 VFD DC
10 I/O not DC
the 10 I/O always OP instantly
I've tried to keep an eye on My start up quite randomly, within a range of + or - 3 million, and does not always perfectly correlate with how quickly the servos OP, but they seem more often than not to OP quickly when this start as a small value, but it does not correlate perfectly.

with refClockSyncCycles="1" from now on this I will call R2M
I get 40s to all OP, so about 4s per device, but some take 2s some 6s, not a constant each one 4s.
with refClockSyncCycles="-1" from now on this I will call [/code][/code]M2R
[code][code]I get 12s to all OP, so about 1.2s per device, again not constant, some quick some slow, but overall much faster.

I have a 1khz servo loop 1000000ns
@Hakan when you say 2khz you mean 500000ns correct?

anyway that is with Scott's code, nothing of mine, I will do further testing with your referenced commit, and then test with my changes, and different kernels see if linuxcnc-ethercat -1 and +1 are the bigger factors here or not.

edit:
tested with new kernel
uname -r
6.3.0-rt11-linuxcnc
same results, almost no effect
44sec for R2M
12sec for M2R
rule out kernel?

The code that handles the initial synchronization in the EtherCAT master is 16 years old, except for a variable name change 8 years ago
gitlab.com/etherlab.org/ethercat/-/blame...e=heads&page=2#L1480

There are people with this "Did not sync after 5000 ms." problem but no real solution found.

I am sure the x092c register shows the symptoms of the problem.
System Time Difference
 

This have me believing that either
- the slaves' clocks are drifting badly,
- the initial time the slaves get from lcec is not a good choice,
- or both.

Watching the 0x92c register during start, I see often, not always, start at 1000000 ns which is 1 ms.
This is with 1 ms linuxcnc servo loop time. When I go down to 0.5 ms servo loop time, 0x92c often starts at 500000 ns,
so there is a possible a correlation.
This initial 1 or 0.5 ms value in 0x92c happens when I start linuxcnc directly and when I let it sit over night.
This points to that it isn't a slave clock issue, but rather the initial time from lcec that is the problem.

Edit: The startup time for my lcec rt_app_main isn't that bad, its 0.85-0.9 secs. Forgot a test-second I had.

Edit 2: Here is a graph if how slave 1 and slave 2 reduce their time differences. This is from journalctl, 05 ms servo loop time.