EtherCAT Communication Issues, Lost Frames, Interruptions in Program Execution

I am pretty sure that RTAI is not supported  by ethercat debs. For starters, the debs use DKMS  so the kernel modules are compiled on yur computer so I suspect that requires a mainstream kernel to fetch the linux-headers source code.

@Rod

Yeah I agree, I hadn't went to the RTAI site in a long time and wasn't sure if by chance it had gone mainstream.  I checked earlier and no, it's still the task of compiling.  So, yeah you are right about the debs.

It really comes down to if he can get PREEMPT working or if he has to go with RTAI.  If RTAI, then it will be a not so fun source build... those kernel modules suck to deal with if they don't install right off.

Hello again everyone. Thank you all for your responses and for letting me know I shouldn't waste my time with RTAI.

I believe I have managed to solve the latency errors and am now at the next roadblock..

I had previously passed isolcpus=1 to grub as a kernel parameter, however I read up more and realized that was incorrect. LinuxCNC uses the last core for realtime processes from what I understand. I changed that to isolcpus=2,3 (extra core helped even more) and the latency is in my opinion very good now. <10k on idle and no more than 55k even with 20 instances of glxgears open (I should have tried glxspheres too, but I'm not sure how to install it on Debian). I let it run for quite a bit and there were also no random spikes to speak of.

Now the issue I will look into is a random "joint X following error" or a "joint X amplifier fault" (it is a seemingly random joint from 0-4 each time) at weird times. I had previously also been getting this error, however I neglected to mention that as I thought it was all due to the latency. Sometimes this will occur as my test program is running, but the weird thing is that it also happens as the machine is idle. Even more strangely, one time the program conducted successfully with no issues and immediately as it ended the joint following error appeared. All of this leads me to rule out the latency being problematic now after I improved it - I doubt the latency would "wait" for the program to finish before it caught up.

I also tested the same machine on another small fairly modern i3 >3GHz PC (the onboard one is around 1.2GHz) just to be sure. Same situation.

Only now that I look into it, do I think to focus on the word "following" in the following error. The .ini file does have an FERROR variable which the previous colleague has set to 0.5 all across and MIN_FERROR to 0.05.

I am going to spend the day reading the wiki page on the "Following Error" and making sense of it. In the first paragraph it says we can be almost certain this error implies a mistake in the .ini. I thought I'd post this beforehand to give an update and to let you kind folks know not to bother with the RTAI line of thinking anymore.

I am attaching my .ini once again just in case someone may have the time to point me in the right direction.

Also, since my issues have moved rather out of scope of the EtherCAT forum subsection, do let me know if I should make another post in a different section.

PS. Neglected to mention that the maximum velocity of the configuration is 6000mm/min. Interestingly, I have the impression that it works much better at lower velocities in the 2-3k range, which should additionally point to the FERROR not being configured properly. I can't conclude it for certain since my test program is rather long and I haven't had the time to extensively test it at even lower speeds.

Some reading later and now I am investigating for DC Sync misconfigurations or problems.

Firstly, I noticed the BASE_THREAD in the .ini was set to 0. Why that was the case? I have no idea. I set it to 60000 as that was what the latency test reported and then decided to give it more headroom and set it to 100000. I'm still testing if this helped or not. I couldn't find any documentation on this value for servos, just for step motors. I'm not sure if the step motor logic can be applied to servos in this case. Could that be why the value was set to 0 (as in, is it irrelevant for servos somehow)?

I can definitely confirm that changing refClockSyncCycles in the .xml affects the machine behavior greatly. Initially I had a wrong understanding of what that value does so I set it to 60000 as well... That lead to extremely violent noises from the machine. I then tried to set it at 4 as my previous colleague had left it. Definite improvement, but there are still weird vibrations and noises sometimes, especially during homing. I then tried 1, 0 and -1 and -1 seems to give the best result, however the amplifier faults and following errors are still there.

From what I read, refClockSyncCycles="-1" used to be something only available if you additionally patched LinuxCNC, but from 2.9 onward that patch has been merged into the version that comes standard with the ISO? Could someone explain to me in more detail what this does? From what I get this is the so called "Slave Synchronization Mode" which is dependent on the master appTimePeriod and the PID values? However, my configuration uses digital servos, which should not require a PID config, correct?

Another thing I did was to reduce the axis acceleration rates drastically in the hopes that that would catch the error, but it doesn't seem it did yet. Now it does not reach more than 3200 velocity, even if the max is set to 6000mm/min.

A different approach I read about is the sync0Shift value of each slave, however I read that it must be the same as the BASE_THREAD. And that is already the case in my config.

I have not yet tried playing around with the FERROR values. I will try that tomorrow. I am seeing conflicting reports on them, though. The docs state that 0.0 is the minimum and 1.0 is the maximum, right? But even then, across the forums I see people with various configurations that have the value set as high as 5.0 or even 10.0?

I do not think you need a base period for EtherCAT, so if that is the case it would be good to remove it.
Ferror values for meteic machines are usually 0.1 and 1, maybe someone with working machine can chime in on what those values should be as those can cause joint errors.
Amplifier fault is caused most probably from interference, if those pins are used in hal and attached to drives, but can also be from drive falting, so check the lights on the drive when it happens to be sure.

Thank you, Tommy, happy to say that removing the base period and adjusting the FERROR values to 0.1-1.0 has yielded results. The machine has been running my very long test program for quite a few hours now with no issues yet. I will continue testing and gradually increasing the acceleration and velocity.

Hopefully this will be the end of this saga, I will report again after a day or three of thorough testing.