Sync Problem LinuxCNC + EL8EC400F

SOLVED!

I am running LinuxCNC with EtherCAT  and Leadshine EL8-EC400F servo drives in Cyclic Synchronous Position (CSP) mode. I’m struggling with jitter / noise that seems to be related to Distributed Clock (DC) synchronization between LinuxCNC and the drives.
My Machine Setup:

• Controller: LinuxCNC (2.9.4) + EtherCAT (lcec)
  • Drives: 5 × Leadshine EL8-EC400F
    • Mode: CiA-402 CSP (0x6060 = ( Control mode CSP)
      • Interpolation time (0x60C2): Read-only, fixed at 2 ms (
        Code:

        value=2, [code]index=-3)
        [/code]
        • Code:

          [code][b]Servo thread: 2 ms (2000000 ns) in LinuxCNC INI Please the attached XML for EtherCAT communication settings. My problem is as follows : [/b]
          [/code]
          • Code:

            [code][b]When I jog axes in LinuxCNC, the motors make a [b]raspy / friction-like noise.[/b][/b]
            [/code]
            • Code:

              [code][b][b]In HALScope, I see [b]large torque and acceleration spikes at the same time.[/b][/b][/b]
              [/code]
              • Code:

                [code][b][b][b]In Leadshine MotionStudio → Diagnostics, I see the counter [b]“Synchronization overflow times” increasing steadily during motion.[/b][/b][/b][/b]
                [/code]
                • Code:

                  [code][b][b][b][b]However, when I test motion with the Leadshine tuning software (without LinuxCNC), the motors run perfectly smooth. what i tried so far :   [/b][/b][/b][/b]
                  [/code]
                  • Code:

                    [code][b][b][b][b]Matched [b]SERVO_PERIOD, [code]appTimePeriod, and [code]sync0Cycle to 2 ms.
                    [/code][/code][/code][/b][/b][/b][/b][/b]
                    • Code:

                      [code][b][b][b][b][b][code][code]Tested different [code]refClockSyncCycles values:
                      [/code][/code][/code][/code][/b][/b][/b][/b][/b]
                      • Code:

                        [code][b][b][b][b][b][code][code][code][code]5 (every 10 ms) → overflow counter increased faster.
                        [/code][/code][/code][/code][/code][/b][/b][/b][/b][/b]
                        • Code:

                          [code][b][b][b][b][b][code][code][code][code][code]1 (every 2 ms) → better, but still some overflows.
                          [/code][/code][/code][/code][/code][/code][/b][/b][/b][/b][/b]
                          • Code:

                            [code][b][b][b][b][b][code][code][code][code][code][code]0.5 (every 1 ms half-cycle) + [code]sync0Shift=10000 → gave the [b]best result so far, but the overflow counter still increments slowly.[/b]
                            [/code][/code][/code][/code][/code][/code][/code][/code][/b][/b][/b][/b][/b]
                            • Code:

                              [code][b][b][b][b][b][code][code][code][code][code][code][code][b]Increased [code]sync0Shift from 1 µs up to 20 µs → small improvements, but not fully stable.
                              [/code][/code][/code][/code][/code][/code][/code][/b][/code][/code][/b][/b][/b][/b][/b]
                              • Code:

                                [code][b][b][b][b][b][code][code][code][code][code][code][code][b][code]Checked 0x60C2 (interpolation time) → confirmed it is [b]read-only fixed at 2 ms, so I cannot change the drives to 1 ms. Also i noticed in the tunner motion studio that all the drivers has EtherCAT slave ID = 2 , while it should be configured by the master which is wired.   my Questions : [/b]
                                [/code][/code][/code][/code][/code][/code][/code][/b][/code][/code][/b][/b][/b][/b][/b]
                                • Code:

                                  [code][b][b][b][b][b][code][code][code][code][code][code][code][b][code][b]Is it valid/expected that [code]refClockSyncCycles="0.5" works better than [code]1 or [code]5? Or does this mean I have a deeper DC setup issue?
                                  [/code][/code][/code][/code][/code][/code][/code][/code][/code][/code][/b][/b][/code][/code][/b][/b][/b][/b][/b]
                                  • Code:

                                    [code][b][b][b][b][b][code][code][code][code][code][code][code][b][code][b][code][code][code]For LinuxCNC + EL8 drives fixed at 2 ms interpolation, what is the recommended [b]refClockSyncCycles and [b]sync0Shift strategy?[/b][/b]
                                    [/code][/code][/code][/code][/code][/code][/code][/code][/code][/code][/b][/b][/code][/code][/b][/b][/b][/b][/b]
                                    • Code:

                                      [code][b][b][b][b][b][code][code][code][code][code][code][code][b][code][b][code][code][code][b][b]Has anyone successfully run EL8-EC400F in CSP under LinuxCNC without sync overflows?[/b][/b]
                                      [/code][/code][/code][/code][/code][/code][/code][/code][/code][/code][/b][/b][/code][/code][/b][/b][/b][/b][/b]
                                      • Code:

                                        [code][b][b][b][b][b][code][code][code][code][code][code][code][b][code][b][code][code][code][b][b]Any advice on NIC/IRQ tuning (isolated cores, rt-preempt) that could help with reducing these overflows?[/b][/b]
                                        [/code][/code][/code][/code][/code][/code][/code][/code][/code][/code][/b][/b][/code][/code][/b][/b][/b][/b][/b]

github.com/linuxcnc-ethercat/linuxcnc-et...uration-reference.md
refClockSyncCycles tells how often the slaves' clocks are synchronized. "1000" should be fine, and "1" and "5" also.
I don't think you can have a floating point number there but I could be wrong.

I have a suspicion on what is going on, it has happened before with other drives, for me included.

I am not familiar with the Leadshine software. It runs on a Windows computer, right? And how does that computer
connect to the drive? I read Modbus somewhere. Just to see the difference with linuxcnc and ethercat.
What is the explanation of the Synchronization overflow times message, what does it mean?

What latency number do you have for the linuxcnc computer? And what network adapter does it have?

“Synchronization overflow times” = the number of times the drive saw Sync0 + PDOs outside the valid window.
there is a tab in the tunner called diagnose that monitor the sync parameter for the communication, please check the attached screenshot from the tunner.
this screenshot is token while jogging the machine, jitter random sounds appears from the motor in random time there is no pattern.

the leadshine tunner app " motion studio" runs on seperate PC connected to the driver via USB.
 

Well, then I would say my suspicion is confirmed. The explanation says it.

The thing is that DC is very precise, and the ethercat master can follow the DC time of the slaves very precisely.
The first ethercat slave determines the DC clock speed.

But Linuxcnc's clock, used for the servo period and base period, is not synchronized with ethercat's DC.
One clock always runs slightly faster/slower.

The consequence of this is that when one clock is overtaking the other, there will be either double PDO or no PDO sent during an ethercat cycle. If there is no data sent one cycle the drive sees that as "stay on the current position" which means a hard brake if it is moving. Or move double delta this time, which means huge acceleration.
Latency and a not so suitable NIC makes it worse, the duration for one clock to overtaking the other becomes longer and more of these hits and misses takes place.

What can you do? Work on latency. Use an Intel NIC. See if your drive have some limitations you can set like max-acceleration. In short, see if your drive can handle hits and misses in a better way.

One would like to see Linuxcnc's clock synchronized with ethercat's DC, but it isn't.
 

Can you test with a negative value for refClockSyncCycles, like refClockSyncCycles="-10"

hey hakan
i discovered that i had huge latency problem i solved it with disable some features in BIOS and made core isolation, and then i changed the refclock to -10 and it worked! but i want to understand why it worked with -10
thank you guys for your help!

Good to hear. Especially the -10 thing.
The "-" actually synchronizes linuxCNC's servo-thread to the DC reference clock.
Wasn't documented, I ran across it in the source code. Seems to work though, and I am happy to get
that feedback from you.
But I don't want to spread this. I have submitted a Pull Request to the linuxcnc-ethercat that will introduce
an option that will both document this and 
make it clear that this can be used. Until then, keep using the "-".