Found the main source of the jitter

If you have been using linuxcnc ethercat, you almost certainly have experienced jitter, the motor makes a weird noise, restarting lxcnc once or twice, and it works well, no amount of latency improvements ever fix this completly, just happens a bit less often.
Since I had sometime to kill this Christmas I put a lot of effort into pin pointing the cause of this issue, and fix it for good, I will share the code in the next few days, still doing some testing and refinements, but I wanted to share what I am at this time practically certain the cause was/is.
The issue is the servo thread syncs with the DC clock, but it is not in phase, each time linuxcnc starts the app-phase is randomly out of phase with the DC epoch time (Fixes value of "DC reference time" displayed by ethercat master), the fix was to remove app-phase from DC epoch, and drift the servo-thread to align with this time, that's all, this will make sure your packets are sent (with jitter) in phase with the DC epoch, and your sync0shift will actually allow for your jitter/latency.
I will do some more testing refine the code, and soon share the fix with the linuxcnc community,
Merry Christmas and Happy Holidays to all!

Indeed, it is the servoloop that isn't in sync with the dc clock.

github.com/linuxcnc-ethercat/linuxcnc-et...s.md#master-settings

I have used your syncToRefClock "true" option, the random jitter issue persisted. The issue is not just sync, it is also phase, by simply drifting to match the DC epoch, you can get in phase. Only after the fix I am NOW finally able to create the jitter at WILL by setting a sync0shift of 10~30us this is expected, as preempt_rt has 20-60us jitter, if you move the sync0shift window in the jitter you should get jitter, this was not the case with just syncToRefClock "true" the servo was in sync, but the phasing was random. I will further test tomorrow, but this is something that has escaped the community for a while.
Sync is also important, but phasing was completely overlooked, I think the current code assumes phase alignment happens automatically, this might be because of ethercat's docs not explaining this well, or because of how linuxcnc is structured the servo thread is started completely separate from the lcec, the servo thread phasing has to be done manually to match DC epoch, even if DC epoch was written by lcec, the moment when servo-thread addf's lcec is practically random, the phasing will be random. Not sure if you understand, I am very deep into testing this for days now, and I can assure you, I have it fixed now, but it was not before.