Ethercat random jitter fix

Yes, you are using DC. Both of your working configurations are DC.
syncToRefClock does not mean "DC on or off". It only controls who is the time reference that everyone else follows. Both modes use Distributed Clocks.
refClockSyncCycles=1 means the master pushes its time to the slaves every cycle. The slaves follow the master. This is DC.
refClockSyncCycles=-1 (without syncToRefClock) means the master adjusts its own cycle to follow the first DC slave. The master follows the slave. This is also DC.
refClockSyncCycles=-1 with syncToRefClock="false" flips the sign, making it effectively +1. So it is the same as refClockSyncCycles=1. Master pushes time to slaves.
Both your working configs resolve to the same thing: master as reference, sync every cycle.
What does not work for you is the mode where master follows the slave (syncToRefClock="true" or negative refClockSyncCycles alone). That is what I want to investigate, whether it is a problem with what your first slave reports or something else in my code.
To confirm DC is active on your slaves, run these commands:
ethercat slaves -v
This will show "Distributed clocks: yes" or "no" for each slave. If it says yes, the hardware supports DC.
Then to check what sync mode is actually active:
ethercat upload -p 0 -t uint16 0x1C32 0x01
This reads the sync mode from the first slave. Value 0 means free run, 1 means SM-synchronous, 2 means DC Sync0, 3 means DC Sync1. You want to see 2 or 3. Replace -p 0 with -p 1, -p 2 etc for each drive.
Please post the output here, or if possible email me at luca at aitalmac.com about remote access. A USB to Ethernet adapter would give the machine internet while keeping the EtherCAT port dedicated.

A USB wifi adapter is also fine for a second network connection. I have often used one with an external antenna. If the PC is in your enclosure, run a SMA cable up to a bulkhead connecter and place the antenna on the outside of the enclosure. You just need to drill a hole for the connector. Something like this but check if you need male or female. www.amazon.com.au/Boobrie-Bulkhead-Exten...BRMR/ref=sr_1_4_sspa

The problem with wifi, often you need to search for  drivers and often you need to build them from source. Like my current wifi dongle. But it is bulletproof now its built. The wired NIC I use for Ethercat.

I'm actually at the client's site performing the installation. This machine was already running Debian 10, but I wanted to test Debian Trixie and we're encountering these problems, which we seem to have solved.

The output of the ethercat slaves -v command is:

=== Master 0, Slave 0 ===
Device: Main
State: OP
Flag: +
Identity:
Vendor Id: 0x0000022b
Product code: 0x0531ec01
Revision number: 0x00000003
Serial number: 0x0001894b
DL information:
FMMU bit operation: yes
Distributed clocks: yes, 64 bit
DC system time transmission delay: 0 ns
Port Type Link Loop Signal NextSlave RxTime [ns] Diff [ns] NextDc [ns]
0 MII up open yes - 3151034175 0 0
1 MII up open yes 1 3151039505 5330 935
2 N/A down closed no - - - -
3 N/A down closed no - - - -
Mailboxes:
Bootstrap RX: 0x1000/1024, TX: 0x1400/1024
Standard RX: 0x1000/128, TX: 0x1080/128
Supported protocols: EoE, CoE, FoE
General:
Group:
Image name:
Order number:
Device name:
CoE details:
Enable SDO: yes
Enable SDO Info: no
Enable PDO Assign: yes
Enable PDO Configuration: yes
Enable Upload at startup: no
Enable SDO complete access: yes
Flags:
Enable SafeOp: no
Enable notLRW: no
Current consumption: 0 mA
=== Master 0, Slave 1 ===
Device: Main
State: OP
Flag: +
Identity:
Vendor Id: 0x00850104
Product code: 0x01030507
Revision number: 0x02040608
Serial number: 0x00000000
DL information:
FMMU bit operation: no
Distributed clocks: yes, 64 bit
DC system time transmission delay: 935 ns
Port Type Link Loop Signal NextSlave RxTime [ns] Diff [ns] NextDc [ns]
0 MII up open yes 0 3730859530 0 935
1 MII up open yes 2 3730862990 3460 855
2 MII down closed no - - - -
3 MII down closed no - - - -
Mailboxes:
Bootstrap RX: 0x0000/0, TX: 0x0000/0
Standard RX: 0x1000/128, TX: 0x1080/128
Supported protocols: CoE
General:
Group: ServoDrive
Image name: DRIVE
Order number: SD700_ECAT Drive
Device name: SD700_ECAT_V1.2_G
CoE details:
Enable SDO: yes
Enable SDO Info: yes
Enable PDO Assign: yes
Enable PDO Configuration: yes
Enable Upload at startup: no
Enable SDO complete access: no
Flags:
Enable SafeOp: no
Enable notLRW: no
Current consumption: 0 mA
=== Master 0, Slave 2 ===
Device: Main
State: OP
Flag: +
Identity:
Vendor Id: 0x00850104
Product code: 0x01030507
Revision number: 0x02040608
Serial number: 0x00000000
DL information:
FMMU bit operation: no
Distributed clocks: yes, 64 bit
DC system time transmission delay: 1790 ns
Port Type Link Loop Signal NextSlave RxTime [ns] Diff [ns] NextDc [ns]
0 MII up open yes 1 3711076970 0 855
1 MII up open yes 3 3711078720 1750 875
2 MII down closed no - - - -
3 MII down closed no - - - -
Mailboxes:
Bootstrap RX: 0x0000/0, TX: 0x0000/0
Standard RX: 0x1000/128, TX: 0x1080/128
Supported protocols: CoE
General:
Group: ServoDrive
Image name: DRIVE
Order number: SD700_ECAT Drive
Device name: SD700_ECAT_V1.2_G
CoE details:
Enable SDO: yes
Enable SDO Info: yes
Enable PDO Assign: yes
Enable PDO Configuration: yes
Enable Upload at startup: no
Enable SDO complete access: no
Flags:
Enable SafeOp: no
Enable notLRW: no
Current consumption: 0 mA
=== Master 0, Slave 3 ===
Device: Main
State: OP
Flag: +
Identity:
Vendor Id: 0x00850104
Product code: 0x01030507
Revision number: 0x02040608
Serial number: 0x00000000
DL information:
FMMU bit operation: no
Distributed clocks: yes, 64 bit
DC system time transmission delay: 2665 ns
Port Type Link Loop Signal NextSlave RxTime [ns] Diff [ns] NextDc [ns]
0 MII up open yes 2 3704486570 0 875
1 MII down closed no - - - -
2 MII down closed no - - - -
3 MII down closed no - - - -
Mailboxes:
Bootstrap RX: 0x0000/0, TX: 0x0000/0
Standard RX: 0x1000/128, TX: 0x1080/128
Supported protocols: CoE
General:
Group: ServoDrive
Image name: DRIVE
Order number: SD700_ECAT Drive
Device name: SD700_ECAT_V1.2_G
CoE details:
Enable SDO: yes
Enable SDO Info: yes
Enable PDO Assign: yes
Enable PDO Configuration: yes
Enable Upload at startup: no
Enable SDO complete access: no
Flags:
Enable SafeOp: no
Enable notLRW: no
Current consumption: 0 mA

while the command:
ethercat upload -p 0 -t uint16 0x1C32 0x01 = 0x0002 2
ethercat upload -p 1 -t uint16 0x1C32 0x01 = 0x0002 2
ethercat upload -p 2 -t uint16 0x1C32 0x01 = 0x0002 2
ethercat upload -p 3 -t uint16 0x1C32 0x01 = 0x0002 2

You are using DC. Confirmed.

All four slaves show "Distributed clocks: yes, 64 bit" and all four report 0x1C32:01 = 0x0002 which means DC Sync0. Your drives are running in DC synchronized mode right now. This is the best mode for a CNC machine.

The only thing that does not work for you is the alternative sync strategy where the master follows the reference slave clock instead of the other way around. But that is just a different way of keeping everyone in sync, not a better or worse one. What you have now, master as reference with refClockSyncCycles=1, is perfectly fine and is DC.

One thing I notice is that your reference clock, slave 0, is not one of the SD700 drives. It is a separate device, the first on the bus. That could be related to why syncToRefClock=true causes problems, if that device reports timing that my PLL does not handle well. If there is an opportunity to test this any time in the future within a better setting, I'd love to debug it remotely.

I've done some work today to merge Sascha's new implementation of DC

github.com/grandixximo/linuxcnc-ethercat...ses/tag/v1.41.0-pre1

I'd love if users here could try it out, and let me know how it goes, the deb is built for debian 13

test with refClockSyncCycles
-1
1
5

About syncToRefClock, you can still use it if you want, but a more strict use will be enforced, to avoid confusion, settings that do NOT make sense will FAIL to open.

Good morning everyone. After several tests I had to leave the machine to the customer with Debian 10. Believe me, I tried everything. It seemed to work with the refClockSyncCycles setting to 1, but sometimes the motors, when the PC was started for the first time, made noise. Turning it off and on again is fine and when it works well, it works well for a long time. I even tried Debian 12, the Git ISO and ethercat 1.6.3, but even worse than Trixie. I put back the Debian 10 HD and ethercat 1.5.2 and it still works fine, even if at first glance, in addition to the fact that with Debian 10, the drive values ​​are read instantly when the PC starts, while with Debian 12 and Trixie, you have to wait a few seconds. And in my opinion, everything starts from this problem. I set latency-histogram --nobase --sbins 1000 --show, stressed the PC, with gear, opening the browser (not connected), moved windows, opened librecad etc., but Debian 10 has maximum and minimum latency half of Debian 12 and Trixie, same PC. I think the problem is between the RT kernel and the Ethercat version. For example, I tested the Debian 10 pins with Ethercat 1.5.2. The pll-error value oscillates between -2200 and 1600. The pll-out pin has a fixed value of 1000 or -1000. Why don't we try installing Ethercat 1.5.2 on Trixie, so as to understand if the problem is on the kernel or Ethercat side. I think that on the Icec side, there is little to do. Yes, we're certainly trying to optimize, but the main problem is latency and jitter stability. I remain available. Luca, I wrote to you privately, but you haven't responded.

Furthermore, while on Debian 12 and Debian Trixie the Grub command line has been optimized, with isolcpu, c state, etc., in Debian 10 there are no optimizations. Obviously I tried to remove the optimizations on Debian 12 and Trixie, as well as on Debian 10, but the problem persists, only on Debian 12 and Trixie.

I tried to compile 1.5.2 on trixie but easier said than done. While I could get around several configuration issues,
there were in the end too severe compilation problems (for me) that I gave up.
At least 1.5.2 doesn't compile out-of-the-box on trixie.

In theory the DC sync should be pretty insensitive to jitter and latency. It should tolerate perhaps 1/2 servo cycle time,
depending on the phase when data is sent to the slaves. It's enough that data arrives a bit before Sync0,
Sync0 then does the precision timing.

I also think the initial synchrony check delay is different from the DC sync issue.
I checked on my system and I do have a several-second delay still. Bad news.
Good news is that I can try to see what the problem is.

So far I have come to understand that it has to do with the initial value
of the "System time difference" in register 0x92c.
Here is a way to observe, don't need to wade through tons of lines of syslog entries:-pn n is any of the DC slaves except the first one with the reference clock.

Before starting linuxcnc it has a low value from previous linuxcnc session.
Start linuxcnc it jumps to aorund 1000000 (1 ms) and slowly works its way to do single digits.
This is on a 2 kHz servo loop system. It takes some 3-4 seconds to work its way down 
below the synchrony limit, and that's where the startup delay comes from.
Obvious question is "Where does the initial 1 ms System time difference come from?"
And it's always around 1 ms, if I wait a minute between linuxcnc starts or over night.

 

We need to join forces and solve this problem once and for all, otherwise there's no point in switching to Debian 12 or Trixie. Keep in mind that Debian 10 works on both older dual cores and i9s. I've tried Trixie on Advantech e8500 cards, and it doesn't work. If we don't solve these existential problems, there's no point in updating the string writing style, or doing many other things that have always worked, but now we want to update the writing style. I've been building LinuxCNC machines for 17 years, and they're still working flawlessly and do everything we do today. Now we need to solve this problem, and I'm available to run any tests. It would be nice to be able to compile LinuxCNC 2.10 on Debian 10, but I understand the dependency issue. We need to figure out if the problem is on the Kernel or EtherCAT side. I could contact anyone who can help us with this, so they can move forward with the fix. Grandixximo's work is to be commended, but he is optimizing a problem, which is not the main problem.

please come over to discord

discord.gg/6mKut8xe

Forum is down way too often, I'm also trying to build 1.5.2
Honestly I don't think it will fix anything, not sure if is worth it, but I am building it
gitlab.com/grandixximo/ethercat/-/pipelines
not ready yet, only generic drive will work.

also

github.com/grandixximo/linuxcnc-ethercat...ses/tag/v1.41.0-pre1

did you test?