Ethercat random jitter fix

Greetings everyone. Today I went to the customer's office after the company closed at 5 pm. I just got back at 8:30 pm. I had the chance to do some more testing, and I have to agree with Grandixximo.
I don't know why, perhaps I was in a hurry to not down the customer's machine for too long, and I made some mistakes.
The jitter results:
Debian Trixie, kernel 6.12.74-rt, latest Grandixxmo patch, Grub command line optimization (isolcpus, etc.) with 4 glx files open, 2 browser windows open, full-screen window movements, switching between different desktops, max jitter 65 microseconds.
The machine with refClockSyncCycles=-1 normally doesn't emit any noise, except for some window movements.
With refClockSyncCycles=1, it emits some noise.
With both refClockSyncCycles=-1 and 1, the command
ethercat upload -p 0 -t uint16 0x1C32 0x01 returns 0x0002 2
Pin pll-error oscillates around -1200 to 1300
Pin pll-out oscillates -5 to 5

I tried the commands:
ethercat upload -p 0 -t uint16 0x1C32 0x02
ethercat upload -p 0 -t uint16 0x1C32 0x03
ethercat upload -p 0 -t uint16 0x1C32 0x05

but it returns the error failed to upload sdo : no buffer space available

Debian Buster, kernel 4.19-27-rt, no optimizations on the Grub command line, with 4 glx files open, 2 browser windows open , full-screen window movements, switching between desktops, max jitter 95 microseconds.
With the same Grub optimizations as Trixie, max jitter is 75 microseconds.
With refClockSyncCycles=1, it easily emits noise.
With refClockSyncCycles=-1, it works flawlessly and without Grub optimizations, which with GRUB optimizations
pll-err pin from -1800 to 2000
pll-out pin oscillates with only two values: either -1000 or 1000

So I deduce that even Trixie has lower jitter than Debian 10, but for some reason, Debian 10 lets my machine work and Trixie doesn't.

Thanks for confirming that jitter itself is not the issue. This was expected with my tuning in place..
One of the difficulties with Ethercat, is that there are no IP addresses on the network to test network latency against. If this was a Mesa card, we would do that with sudo chrt 99 ping -i .001 -c 600000 -q 10.10.10.10 Perhaps there is an alternative method.

I think it can be assumed on a properly tuned Ethercat Intel NIC (like yours) that network latency is not a factor. Ethercat has an inherent advantage over Mesa because of its customized network protocol that uses significantly less bandwidth due to smaller packet sizes. The main network latency culprit is coalescing which has been disabled, SMP affinity has been used to ensure the NIC interrupt is running on the same core as Linuxcnc and a number of other optimisations that could introduce network latency have been tamed.This is confirmed where we see a substantial tightening of the Standard Deviation in before and after latency-histogram tests (more than halved from 2.3  to 1.1 usec on Mike's PC). This means the system is much more deterministic with fewer latency spikes (eg narrower bell curve, much more consistent timing on the servo thread).

It all comes back to your environment. eg a configuration issue or a problem with the ethercat system itself (ether the Ethercat master or the linuxcnc-ethercat driver. CNC is a very small subset of what applications the iGh master is used for. Its mostly used on factory floor automation. This is why I find it hard to believe that it has a problem as if it did, the Igh issues register would be flooded with reports and it isn't.

So by my analsyis,  the issue has to be a configuration issue or the linuxcnc-ethercat driver.

papagno-source wrote:

The machine with refClockSyncCycles=-1 normally doesn't emit any noise, except for some window movements.
With refClockSyncCycles=1, it emits some noise.
With both refClockSyncCycles=-1 and 1, the command
ethercat upload -p 0 -t uint16 0x1C32 0x01 returns 0x0002 2
Pin pll-error oscillates around -1200 to 1300

When moving windows Linuxcnc's servo loop takes too long time.
There usually is a message in the start window saying that.
If you monitor pll-err in halscope or record it with sampler you will
see a spike in pll-err.
There is some more fundamental work needed to make linuxcnc independent
on such events.
Wouldn't surprise me if this is the difference between 4.19 and 6.

I will do one last test with an Advantech Aimb 505 card with an i5-6700 processor. I tested the latency with Trixie and under stress it does not exceed 40 microseconds with a maximum delta of 2. I asked the customer if I could do this test and he will give me the machine available. Also because, now the only solution we are waiting for is a new version of Ethercat that is adapting Grandixximo?

I was able to run 1.5.2 ribalda built on debian 13 on Papagno's hardware, OP are quick, and no perceptible grinding on a simple bench setup, with 0.9.5 linuxcnc-ehtercat sittner, This is not even the good PC hardware, it had 400ns of jitter, and the motors seemed fine, we will need to test better on the machine.
This was mostly to rule out kernel jitter, biggest culprit for slow OP is definitely ethercat 1.6.9
We (me and YangYang) have a refreshed 1.6.9 in the works, but not sure when that will land, squashing some bugs still, the code bases 1.5.2 riblada and 1.6.9 igh went in two separate directions, and they are hard to merge, will put best efforts in, then we will have a version for ourselves we can maintain ourselves, since igh is kinda of unresponsive and or uninterested, no clue...

Good morning everyone.
I'm continuing the discussion on the Forum, on this PC; I don't have Discord.
I ran the last tests requested by Grandixximo with two versions.
Both versions, with reference clock -1 and 1, don't work well; the axes emit noise.
With the value -1, they are very present when you move browser windows; with 1, they are less noticeable but are still there.
With halscope, I monitored the profile error of joint 0 and joint 1, and when the noise occurs, you see a spike in tracking error on both axes at the same time.
I also tried changing all the EtherCAT cables with no results.
Obviously, for those who have been following the discussion, I had to reinstall the HD with Debian 10, which works fine.

I tried optimizing the GroB command, but it doesn't work.
I tried varying synclock from 0 to 300,000 and 500,000, but it doesn't work.

Now I can't stop the customer who has been very helpful.
We could try again on the next machine, in about 3 months.

The problem with the initial delay of the LCEC pins has been resolved, but the synchronization problem is still present.

I've noticed that sometimes everything seems to be fine. You open maybe four browser windows, frecad, and so on, and everything is fine. Then you reboot, and after opening the first browser window, it starts to vibrate, and it stays that way until you reboot several times.

The problem is there, and it's significant. It means we'll be stuck with Debian 10.
I imagine some drives are more tolerant than others, but we can't use a system that only works well with some drives and not with others.

Probably, if the synchronization problems between the kernel and EtherCAT aren't resolved, in my opinion, the solution would be a hardware EtherCAT master.

I want to thank Grandixximo, RoDW, and others for their efforts.
If you'd like, once I've prepared the next machine, we can meet and work directly on the machine. I'd be willing to do anything to resolve the issue.

I remain available for any clarification.