Issue with OP Entry Delay and Failures in LinuxCNC + EtherCAT System

Dear forum members,Hello!I am currently setting up and running two systems based on LinuxCNC and EtherCAT, and I have encountered an issue regarding system startup performance. I hope to receive assistance and advice from you all.Overview of System Configurations:

1. First System:
   • LinuxCNC Version: 2.8.4
   • EtherCAT Version: 1.5.2
   • I compiled the Linux kernel, Xenomai, LinuxCNC, and EtherCAT myself, and used a real-time network card driver.
   • This system's EtherCAT drives quickly enter the OP state within 1 second after startup.
2. Second System:
   • LinuxCNC Version: 2.9.3
   • EtherCAT Version: 1.6.1
   • Installed according to the stickied post on the forum. "Ethercat installation from repositories - how to step by step"
   • This system takes a long time (over 10 seconds) for the EtherCAT drives to enter the OP state after starting LinuxCNC, and sometimes it fails. At this point, I have to close and reopen LinuxCNC, which may sometimes result in success, but it is not consistent.

Commonalities and Differences Eliminated:

• Both systems are deployed on the same hardware platform, eliminating hardware differences.
• The configuration files are almost identical.

Request for Assistance:Given that the first system performs normally, how can I resolve the startup delay issue on the second LinuxCNC system?

• Is it worthwhile to try using different versions of EtherCAT?
• If so, what are the specific steps I should follow to do so?

Thank you very much for your valuable time and assistance! I look forward to your replies.Best regards

If you look at the diagnostic outputs using "sudo dmesg" on the two systems, does that give a clue to what can be the issue?
EtherCAT master puts out some info there.

Thank you for your response. These are the printout messages from two different systems, respectively.

First System:
[  238.627879] EtherCAT: Requesting master 0...
[  238.627885] EtherCAT: Successfully requested master 0.
[  238.639567] EtherCAT 0: Domain0: Logical address 0x00000000, 450 byte, expected working counter 27.
[  238.639571] EtherCAT 0:   Datagram domain0-0-main: Logical offset 0x00000000, 450 byte, type LRW at ffff91a4e6ce0318.
[  238.640138] EtherCAT 0: Master thread exited.
[  238.640141] EtherCAT 0: Stopping EoE thread.
[  238.640157] EtherCAT 0: EoE thread exited.
[  238.640159] EtherCAT WARNING 0: EoE External processing required!
[  238.640160] EtherCAT 0: Starting EtherCAT-OP thread.
[  239.249213] EtherCAT 0: Domain 0: Working counter changed to 21/27.
[  239.291171] EtherCAT 0: Slave states on main device: PREOP, SAFEOP.
[  239.572573] EtherCAT 0: Slave states on main device: PREOP, OP.
[  239.860401] EtherCAT 0: Slave states on main device: PREOP, SAFEOP, OP.
[  240.181272] EtherCAT 0: Slave states on main device: PREOP, OP.
[  240.251335] EtherCAT 0: Domain 0: Working counter changed to 24/27.
[  244.842333] EtherCAT WARNING 0-main-0: Slave did not sync after 5000 ms.
[  244.852890] EtherCAT 0: Domain 0: Working counter changed to 27/27.
[  244.886478] EtherCAT 0: Slave states on main device: SAFEOP, OP.
[  245.167387] EtherCAT 0: Slave states on main device: OP.


Second System:
[ 3248.220047] EtherCAT: Requesting master 0...
[ 3248.220054] EtherCAT: Successfully requested master 0.
[ 3248.225443] EtherCAT 0: Domain0: Logical address 0x00000000, 406 byte, expected working counter 21.
[ 3248.225447] EtherCAT 0:   Datagram domain0-0-main: Logical offset 0x00000000, 406 byte, type LRW.
[ 3248.225471] EtherCAT 0: Master thread exited.
[ 3248.225474] EtherCAT 0: Stopping EoE thread.
[ 3248.225489] EtherCAT 0: EoE thread exited.
[ 3248.225490] EtherCAT 0: Starting EoE thread.
[ 3248.225559] EtherCAT 0: Starting EtherCAT-OP thread.
[ 3248.225732] EtherCAT WARNING 0: 16 datagrams UNMATCHED!
[ 3248.299560] EtherCAT WARNING 0-6: Other mailbox protocol response for eoe0s6.
[ 3248.315523] EtherCAT WARNING 0-6: Other mailbox protocol response for eoe0s6.
[ 3248.339530] EtherCAT WARNING 0-6: Other mailbox protocol response for eoe0s6.
[ 3248.356219] EtherCAT WARNING 0-6: Other mailbox protocol response for eoe0s6.
[ 3248.371609] EtherCAT WARNING 0-6: Other mailbox protocol response for eoe0s6.
[ 3248.391524] EtherCAT WARNING 0-6: Other mailbox protocol response for eoe0s6.
[ 3248.409001] EtherCAT WARNING 0-6: Other mailbox protocol response for eoe0s6.
[ 3248.423525] EtherCAT WARNING 0-6: Other mailbox protocol response for eoe0s6.
[ 3248.439516] EtherCAT WARNING 0-6: Other mailbox protocol response for eoe0s6.
[ 3248.455500] EtherCAT WARNING 0-6: Other mailbox protocol response for eoe0s6.
[ 3248.471503] EtherCAT WARNING 0-6: Other mailbox protocol response for eoe0s6.
[ 3248.487610] EtherCAT WARNING 0-6: Other mailbox protocol response for eoe0s6.
[ 3248.503529] EtherCAT WARNING 0-6: Other mailbox protocol response for eoe0s6.
[ 3248.519501] EtherCAT WARNING 0-6: Other mailbox protocol response for eoe0s6.
[ 3248.535504] EtherCAT WARNING 0-6: Other mailbox protocol response for eoe0s6.
[ 3248.551521] EtherCAT WARNING 0-6: Other mailbox protocol response for eoe0s6.
[ 3248.567541] EtherCAT WARNING 0-6: Other mailbox protocol response for eoe0s6.
[ 3248.583503] EtherCAT WARNING 0-6: Other mailbox protocol response for eoe0s6.
[ 3250.223357] EtherCAT WARNING 0: 6 datagrams UNMATCHED!
[ 3251.223374] EtherCAT WARNING 0: 10 datagrams UNMATCHED!
[ 3252.223350] EtherCAT WARNING 0: 16 datagrams UNMATCHED!
[ 3253.223370] EtherCAT WARNING 0: 10 datagrams UNMATCHED!
[ 3254.223357] EtherCAT WARNING 0: 6 datagrams UNMATCHED!
[ 3255.223368] EtherCAT WARNING 0: 4 datagrams UNMATCHED!
[ 3256.223343] EtherCAT WARNING 0: 4 datagrams UNMATCHED!
[ 3257.223350] EtherCAT WARNING 0: 16 datagrams UNMATCHED!
[ 3258.223347] EtherCAT WARNING 0: 2 datagrams UNMATCHED!
[ 3258.720269] EtherCAT ERROR 0-6: Timeout after 10000 ms while setting state OP.
[ 3258.727412] EtherCAT 0: Slave states on main device: PREOP, SAFEOP.
[ 3259.223344] EtherCAT WARNING 0: 6 datagrams UNMATCHED!
[ 3260.223352] EtherCAT WARNING 0: 16 datagrams UNMATCHED!
[ 3261.223364] EtherCAT WARNING 0: 4 datagrams UNMATCHED!
[ 3262.223349] EtherCAT WARNING 0: 8 datagrams UNMATCHED!
[ 3263.223381] EtherCAT WARNING 0: 6 datagrams UNMATCHED!
[ 3264.116265] EtherCAT WARNING 0-0: Slave did not sync after 5000 ms.
[ 3264.223353] EtherCAT WARNING 0: 12 datagrams UNMATCHED!
[ 3265.227471] EtherCAT WARNING 0: 4 datagrams UNMATCHED!
[ 3265.287422] EtherCAT ERROR 0-3: Mailbox error response received - Unknown error reply code 0x0000.
[ 3265.287431] EtherCAT WARNING 0-3: Invalid mailbox response for eoe0s3.
[ 3265.327467] EtherCAT ERROR 0-3: Mailbox error response received - Unknown error reply code 0x0000.
[ 3265.327474] EtherCAT WARNING 0-3: Invalid mailbox response for eoe0s3.
[ 3266.227344] EtherCAT WARNING 0: 4 datagrams UNMATCHED!
[ 3267.227351] EtherCAT WARNING 0: 2 datagrams UNMATCHED!
[ 3268.227347] EtherCAT WARNING 0: 6 datagrams UNMATCHED!
[ 3269.227348] EtherCAT WARNING 0: 6 datagrams UNMATCHED!
[ 3270.033265] EtherCAT ERROR 0-6: AL status message 0x002D: "No Sync Error".
[ 3270.036267] EtherCAT 0-6: Acknowledged state SAFEOP.
[ 3270.227348] EtherCAT WARNING 0: 14 datagrams UNMATCHED!
[ 3270.435372] EtherCAT 0: Slave states on main device: OP.
[ 3271.227376] EtherCAT WARNING 0: 10 datagrams UNMATCHED!
[ 3273.239359] EtherCAT WARNING 0: 2 datagrams UNMATCHED!
[ 3274.239353] EtherCAT WARNING 0: 6 datagrams UNMATCHED!
[ 3275.243359] EtherCAT WARNING 0: 8 datagrams UNMATCHED!
[ 3276.243376] EtherCAT WARNING 0: 6 datagrams UNMATCHED!
[ 3278.247366] EtherCAT WARNING 0: 10 datagrams UNMATCHED!

(Repeated warnings about unmatched datagrams...)
Based on a cursory review, I speculate that the first system might be using a compatible version of the EtherCAT protocol with the slave devices (drivers), whereas the second system might be utilizing a newer version of the protocol that is not fully compatible with the older version supported by the slave devices.

Additionally, the "UNMATCHED" warnings seem to persist throughout the operation in the second system, indicating potential issues with data packet matching or communication synchronization. Note that I haven't included all the printouts, but the frequent occurrence of these warnings suggests ongoing communication issues.

xml file
 

I kind of think that "Unmatched messages", that many, are related to the real-time performance, jitter, of that particular system.
If possible, it would be interesting to duplicate the good system's complete disk and run that on the bad system.

Thank you for your response.

I would like to know what versions of LinuxCNC and Ethercat you commonly use, respectively.
I will reinstall the system.

I used to compile EtherCAT by myself, but since the packages became available I just install them as described here by rodw.
I take it you have run latency-histogram and have the graph nicely inside say +-20 microseconds for the servo thread?

Thank you for your response.

What version are you currently using? I'd like to try it out.I'm currently using ethercat-master (1.6.3.g805e407-1+15.3)

The line I have is
 EtherCAT: Master driver 1.6.3 1.6.3.g805e407-1+15.2
but the latest is the "+15.3" waiting for me to upgrade.
Overall it is linuxcnc 2.9

I fought the UNMATCHED messages for my system.
It is caused by the network interface being...not fast enough for lack of better words.
Intel network adapters have always worked well for me.
Realtek adapters are almost always an issue
and some tweaks are needed. rodw has documented some of that.