Ethercat drives not responding, mesa error finishing read

I have 3 iX7NH servo drives connected via ethercat. I have installed ethercat (IgH EtherCAT master 1.6. to the linuxcnc 2.9.4 ISO from repositories as described in this thread .

Using ethercat -p0 upload -t int16 0x2003 0 I can request that word from any drive, but it frequently fails with Failed to upload SDO: Input/output error.

dmesg reports:
[316.525911]EtherCAT WARNING 0-0: Other mailbox protocol response from eoe0s0.
[316.537792]EtherCAT ERROR 0-0: Reception of CoE upload response failed: No response.
[316.537797]EtherCAT ERROR 0-0: Failed to process SDO request.

I can run linuxcnc, and it will throw for one or two drives that same error I received with ethercat, but for different drives each time. I was once lucky enough to have it throw no such errors, but within a few minutes, one of the drives refused to communicate. When the drives do communicate, the DRO fields will update as I turn the motor shafts by hand.

I have seen drives flap betweep PREOP and OP status, while linuxcnc was running.

I also have some mesa daughterboards served by a 7i94. Occasionally, they will stop responding, showing all comms error lights, and linuxcnc will say "error finishing read".

I have installed r8168-dkms, and the realtek interface is using that driver. The other interface is supposedly an Intel card, using the igb driver, but it came from China, so there's no guarantee. Naturally the r8169 driver gave the same problems. I have tried other ethernet cards, but they all have realtek silicon.

The latency histogram does not exceed 10uS. Ping to the 7i94 is 112uS average, 203uS max.

What can I try next?

I got lot's of errors yesterday and going in and out of PREOP. 
It might have been a bad kernel. I upgrade to latest 6.12-48 and the problems went away.
Think the bad one was 6.12-38.

My existing install is using kernel 6.1.0-30-rt-amd64 (from the 2.9.4 ISO).

I'm going to grab Debian 13 and try to assemble from scratch.

Did you install ethercat from the repository or did you build from source?

Debian 13 didn't work: apt-key isn't allowed any more, and I don't know what to do to work around that. Without that, I can't install linuxcnc.

Debian 12 doesn't work either: I can install linuxcnc and ethercat, but I can't start ethercat. I see the error "modprobe: FATAL: Module ec_master not found in directory /lib/modules/6.1.0-40-rt-amd64"

I have installed r8168-dkms, and the realtek interface is using that driver. The other interface is supposedly an Intel card, using the igb driver, but it came from China, so there's no guarantee. Naturally the r8169 driver gave the same problems. I have tried other ethernet cards, but they all have realtek silicon.

Have you tried pinning the Realtek interface interrupt to last processor?
(also with using isolcpus=last_processor in the kernel command line)

This has helped in many cases.

This script allows you to test this:

 
 

With that kernel version it may not be the same.
In any case, it sounds like it's something with the network, latency, etc. that PCW also points to.