grandixximo wrote:

edit:
tested with new kernel
uname -r
6.3.0-rt11-linuxcnc
same results, almost no effect
44sec for R2M
12sec for M2R
rule out kernel?

Can't be the kernel then, It was worth a try. When Debian 12 came out on the 6.1 kernel, we were all trying to work out why latency was so bad, but now the factors are well known so now we know updating the kernel no longer helps. (Trixie I have found is better than Bookworm)

@hakan, I read somewhere (maybe even on this thread) the 5000ms was just a waening, and not really something to be concerned about.

Good morning everyone. Today I'll try the latest Grandixximo patch.
I can assure you that all PDOs, both on the I/O side of the first slave and on the drives, are read immediately. The same is true on another machine, where in addition to the drives there are two Delta inverters.

I tried running the following commands on Debian 10:
ethercat debug 1
start linuxcnc
exit or leave running, it doesn't matter.
sudo journalctl --since "2 min ago" | grep "Sync after"

no results.

uname -r 4.19.0-27-rt-amd64
dpkg -l | grep linuxcnc-uspace : no results
dpkg -l | grep ethercat: no results

I checked, and if I don't start axis, all the slaves are in PREOP.

The OP state and writing to the slave pins only occurs immediately after running addf lcec.write-all servo-thread in hal.

Just for completeness of information. On Debian 10, as well as on Trixie, I am using the "generic" device in the ethercat.conf file.

As rodw said, try to add isolcpus=3 or even isolcpus=2,3 in your /boot/firmware/cmdline.txt

Also, I did some tests with Remora over ethernet on a PI400 and you should follow rodw's video to adjust the ethernet connection to your mesa card.

The code that handles the initial synchronization in the EtherCAT master is 16 years old, except for a variable name change 8 years ago
gitlab.com/etherlab.org/ethercat/-/blame...e=heads&page=2#L1480

There are people with this "Did not sync after 5000 ms." problem but no real solution found.

I am sure the x092c register shows the symptoms of the problem.
System Time Difference
 

This have me believing that either
- the slaves' clocks are drifting badly,
- the initial time the slaves get from lcec is not a good choice,
- or both.

Watching the 0x92c register during start, I see often, not always, start at 1000000 ns which is 1 ms.
This is with 1 ms linuxcnc servo loop time. When I go down to 0.5 ms servo loop time, 0x92c often starts at 500000 ns,
so there is a possible a correlation.
This initial 1 or 0.5 ms value in 0x92c happens when I start linuxcnc directly and when I let it sit over night.
This points to that it isn't a slave clock issue, but rather the initial time from lcec that is the problem.

Edit: The startup time for my lcec rt_app_main isn't that bad, its 0.85-0.9 secs. Forgot a test-second I had.

Edit 2: Here is a graph if how slave 1 and slave 2 reduce their time differences. This is from journalctl, 05 ms servo loop time.
 

Here is what it looks like.  All offsets show zero, even though they are not zero.

 

I have done testing with the above, this is Debian 13 everything installed from repos.
Generic drive no ec_igc
20 devices
9 servos DC
1 VFD DC
10 I/O not DC
the 10 I/O always OP instantly
I've tried to keep an eye on My start up quite randomly, within a range of + or - 3 million, and does not always perfectly correlate with how quickly the servos OP, but they seem more often than not to OP quickly when this start as a small value, but it does not correlate perfectly.

with refClockSyncCycles="1" from now on this I will call R2M
I get 40s to all OP, so about 4s per device, but some take 2s some 6s, not a constant each one 4s.
with refClockSyncCycles="-1" from now on this I will call [/code][/code]M2R
[code][code]I get 12s to all OP, so about 1.2s per device, again not constant, some quick some slow, but overall much faster.

I have a 1khz servo loop 1000000ns
@Hakan when you say 2khz you mean 500000ns correct?

anyway that is with Scott's code, nothing of mine, I will do further testing with your referenced commit, and then test with my changes, and different kernels see if linuxcnc-ethercat -1 and +1 are the bigger factors here or not.

edit:
tested with new kernel
uname -r
6.3.0-rt11-linuxcnc
same results, almost no effect
44sec for R2M
12sec for M2R
rule out kernel?

I booted the current live image and created a new one like I had at work.

 

It's just a window with the Offsetpage widget. I don't have a copy here at home but (it's at work.)

Thanks for clarifying. I guess if 1.5.2 does not solve the issue, the next question is has linuxcnc-ethercat changed since Debian 10? I suspect not but can you review and confirm? If nothing has changed, its a Debian kernel issue

People forget that modern kernels post 5.9 and up  (which is well after Debian 10) are enormously complex and the hardware developers have been adding  features for years (mostly power saving) that the linux kernel unfortunately now supports. So there are a lot of tuning steps now. Perhaps there are some issues we have not covered in our known NIC tunings.

I've shared a 6.3 kernel I built in the past on Discord which many people had good results with. Perhaps it will help. It will be a few days before I can build a later kernel for Trixie.


 

rodw wrote:

For somebody trying hard to follow along from the sidelines, can you guys outline the objective for rolling back to 1.52 here?
Has anybody looked at the Igh Ethercat master 1.6 or even 1.7 which was released on gitlab 3 weeks ago (but not in their repositories) to see if it has these delays outside of linuxcnc.

It seems to me it would be unlikely that iGh master 1.6 or 1.7 are problematic as people would be complaining at iGh so to me,  the effort should be in understanding what has changed that linuxcnc-ethercat is not liking. Rolling back to a very old version does not seem a progressive step following the synchronisation enhancement.

Did it on pagagno-source request

papagno-source wrote:

Why don't we try installing Ethercat 1.5.2 on Trixie, so as to understand if the problem is on the kernel or Ethercat side.

This is just for testing, I pretty much knew it was a dead end when I started on this path, but it is just so we are all on the same page, I need to bring receipts, I won't simply hand wave his concerns, he wanted 1.5.2 on Trixie so that he will be convinced that is not the issue, and I gave him that. I don't want to come back to this point over and over again as we proceed with more testing, once it is proven that 1.5.2 does not solve the issue, we can move on to the next thing. Just trying to respect other people's perspective, call it an unnecessary waste of time if you want, I can see where you are coming from, but I feel the best approach is moving systematically one step at the time, clearing up all concerns, testing thoroughly, find agreement, move to the next thing.

1.6 and 1.7 present the same OP time delays in my testing.

Can you post your glade file?

Hakan wrote:

Tested a bit.
I get fairly consistent loading times of lcec's rt_app_main() function of 1.85-1.90 seconds.
But but. In syslog I can see that the initial sync of slaves takes anywhere between zero and
say 3,4,5 seconds. 
To check
ethercat debug 1
start linuxcnc
exit or leave running, doesn't matter.
sudo journalctl --since "2 minutes ago" | grep "Sync after"
and see where it stops for every slave.
I reckon this is the problematic delay time.
Seldom, but it happens, I get "Checking for synchrony" and then it is satisfied directly and moves on.

It seems measuring the time of lcec's rt_app_main is not a good indicator.
Is there a good way of measuring time to OP?

trixie, 
Linux plasma 6.12.69+deb13-rt-amd64 #1 SMP PREEMPT_RT Debian 6.12.69-1 (2026-02-08) x86_64 GNU/Linux
IgH EtherCAT master 1.6.8 1.6.8.g2543cc5-1+27.3  (via apt)
linuxcnc-ethercat, a slightly modified version of commit bf13577ec7b1b54844e07a80d949232e71776b32
linuxcnc 2.9.4 (via apt)

At the end of my hal file I and I save the default watch to display the OP pins, you can have master all-op, and each slave individual OP pin, I count Mississippies or use stopwatch...
Another alternative is But this might "dirty" the bus, may effect OP time, maybe...
To get more accurate times might have to build a script, or use dmesg with some debug from ethercat.
But I think the difference papagno-source is talking about can be perceived with less precise tooling, in one case everything OP even before the UI opens, and in the other he has to wait some amount of time, so it is a difference in the seconds range.
I will try to setup similar setup to yours, and check the OP times.

For somebody trying hard to follow along from the sidelines, can you guys outline the objective for rolling back to 1.52 here?
Has anybody looked at the Igh Ethercat master 1.6 or even 1.7 which was released on gitlab 3 weeks ago (but not in their repositories) to see if it has these delays outside of linuxcnc.

It seems to me it would be unlikely that iGh master 1.6 or 1.7 are problematic as people would be complaining at iGh so to me,  the effort should be in understanding what has changed that linuxcnc-ethercat is not liking. Rolling back to a very old version does not seem a progressive step following the synchronisation enhancement.

Looks pretty good. I'll have to check it so I don't miss anything.
Sorry, I've had to focus on other projects these last few days which I have almost completed, then record an overdue video for my Youtube channel, then I will get back to this.

papagno-source wrote:

I have installed linuxcnc-ethercat...0.9.5-retro1.5.2-rc1, without error, but if coomand ethercat version have reult 1.6.9
But I think, this is not the Igh part, but the lcec driver, right?
I have a version running in place in another directory, do I need to copy or link something to test?
On Debian 10 , in direcory bin the linuxcnc , i have only lcec_conf and not have lcec_coffingen
 

gitlab.com/grandixximo/ethercat/-/jobs/1...ad?file_type=archive
This is the igh ethercat, you need to uninstall your current ethercat master to get this to work, also ethercat-dkms and libethercat.
You might have to run or restart the ethercat service after installing the deb packages in the artifact.zip
 
for the linuxcnc-ehercat remove your current version
and install this
github.com/grandixximo/linuxcnc-ethercat...0.9.5-retro1.5.2-rc1
this is the lcec part, and it has both lcec_conf and lcec_coffingen
I cannot fathom why you would have one and not the other if you installed my packages.

I saw you tried to come to the discord, but you did not give a thumbs up to the rules message, so you were not allowed in.

I'm on +08:00 time. not in Italy. So mind the time difference. I'll only probably be available in the morning Italy time.