EtherCAT Communication Issues, Lost Frames, Interruptions in Program Execution

Hi again.

I spent quite a bit of time on this in the past week and I believe I narrowed down the problem. My worry is that I don't know if it can even be solved, haha.

Firstly, I dug deeper into what rodw suggested with making sure I had a dedicated NIC to EtherCAT. Turns out the generic EC driver doesn't reserve one for itself. I found out there is an igb driver in the 6.1 kernel, which was great. I had to remove everything EC related and rebuild it from source, however, since the configure tool needed the --enable-igb option passed to it for igb support. You can't do that when you install from repositories AFAIK. Either way, now the NIC doesn't even show up under "ip a" anymore. EtherCAT has it completely reserved for itself, which is great.

Now, the real culprit. As I mentioned before, I saw some blink and you'll miss them spikes in the latency. After a lot of reading, I came to the conclusion it is related to Intel SMI (System Management Interrupt) . Diving into some more old documentation, I saw it mentioned as a culprit . However, these posts seem to be very very old at this point. The tool suggested at the end of that wiki article did not help me, as it only reports that "No SMI-enabled chipset found".

This cannot be the case, though, as in my BIOS there is clearly a setting called "Global SMI Lock." Additionally, I found this utility called BITS. It provides exactly what I need - an SMI latency test. The PC failed said test miserably, with 300μ being reported. That is, by the way, the exact size of the spike the LinuxCNC Latency Test reported. I am attaching a photo of the test result. I am assuming the tool from the wiki article is simply very old and does not support the onboard PC's chipset (the CPU of which is an Intel Celeron N2930). The MSR 0x34 register as reported by turbostat is also 1.17, which I am not sure how to interpret exactly, but seeing as it is well over 0, it must not be good.

I will spend the day trying to somehow disable SMI and testing the machine with a different PC that I know passes the BITS SMI test.

Does anyone have any idea on how to go about disabling SMI? I believe the only option in my BIOS is this Global SMI Lock I mentioned which didn't seem to do much. On some old forums I saw people mentioning you need to mod and reflash your BIOS. I'd like to leave that as a last resort as doing it improperly will essentially deep fry the motherboard.

Try disabling in BIOS, hyperthreading, turbo, speedstep, TPM, any power saving option, pci-e agresive link management or similar, modem, audio, serial ports, etc.
Also helpful, set a value for shared memory, do not leave it on auto, this i keep forgeting as it causes latency spikes whenever it adjusts the amount of memory dedicated to integrated graphic.

Thank you for the response, Tommy.

I had tried all of those before actually. I hadn't looked into adjusting the shared memory, so I tried that now, but there doesn't seem to be an option for that at all. It seems it is a pretty locked down BIOS...

Did more reading and it seems that switching to the RTAI kernel is a solution for some. I figure it's worth trying out before I look into buying a new onboard PC entirely. I tried to install it on a fresh Debian Bookwork install as suggested in the docs and did the instructions from the README.INSTALL . However, I am not able to boot properly into the newly created grub kernel entry still. There is some error of the root device not being able to be mounted properly? Even though the Debian install still works with no issues. I'm attaching a photo of the error as well.

Is there some sort of better guide on how to install the RTAI kernel and LinuxCNC on it?

The error means the GRUB is OK and working but the installation was done on a drive that no longer exists, probably into a USB device.
For RTAI see:
linuxcnc.org/docs/2.9/html/getting-start...#cha:Installing-RTAI

I'm not sure how that could happen. There were no other devices plugged in and even then the UUID the Alert is saying does not exist is exactly the Debian root partition of the SSD.

Yes, I did see that part of the docs. However, that only covers installing LinuxCNC (the software, not the distro) on an already patched to RTAI kernel on a vanilla Debian install, correct? I am still stuck on the part where you actually compile the RTAI kernel and boot to it.

Now that I think about it, while installing the compiled kernel with "make install," I did see some warning about some modules not being found and now the error at boot suggests the same.

No compiling.
if using the LinuxCNC official ISO, the only things you have to do is as stated in the linked page
sudo apt update
sudo apt install linuxcnc
it will install everything needed.
It might also complain it need to remove LinuxCNC-uspace, so if it does not remove it by itself
sudo apt purge linuxcnc-uspace
After reboot
uname -a
to check the kernel, and if it is the RT-preempt one, reboot and hold down the shift key till you see the grub menu, there choose the RTAI kernel and press enter.