Ethercat random jitter fix

More
7 months 2 weeks ago #344141 by TAKUYA
Replied by TAKUYA on topic Ethercat random jitter fix
Installed the latter. So far so good, didnt do any major tests but multiple power cycles. Also, with the ClockSyncCycles set to 5 and syncToRefClock=true, my servos are now instantly OP. no waiting for them to be ready which caused some amplifier errors when i was too quick with the power button recently.

no idea if thats because of the new linuxcnc-ethercat version or because of those parameters.

one thing to note though, the dc-phased pin never went high for me.

Please Log in to join the conversation.

More
7 months 2 weeks ago #344163 by grandixximo
They are same

github.com/grandixximo/linuxcnc-ethercat

this just has one small optimization

dc-phased is latency dependent, can you plot in halscope, app-phase, pll-error, and servo thread?

---
test signature

Please Log in to join the conversation.

More
7 months 2 weeks ago #344171 by TAKUYA
Replied by TAKUYA on topic Ethercat random jitter fix
Ah okay gotcha.

Ill plot the mentioned signals this weekend when im back at the machine.
Latency on my host is ~20k with max jitter when opening mulitple tabs max 45k

would lowering/increasing the SyncCycles value help with this?

Please Log in to join the conversation.

More
7 months 2 weeks ago #344176 by Atsu
Replied by Atsu on topic Ethercat random jitter fix
I have the same issue as TAKUYA, the dc-phased pin never goes high. I managed to fix this using Claude AI and now the dc-phased pin goes high. I have attached a detailed notes of what was changed in the code as well as some screenshots from before and after.

NOTE: this was done entirely with the help of AI and I am not qualified enough to say whether it is correct or not! Grandixximo, could you please review the notes and share your thoughts? Opening the attached HTML file in a browser, you can view the notes which were generated. It is too long to copy-paste here.

EDIT: After testing the code more thoroughly, I'm getting jitter again. It's very likely that the AI's "fix" only addressed the dc-phased pin but broke something else... I've reverted back to the master branch ( github.com/grandixximo/linuxcnc-ethercat ), and everything is working again except for the dc-phased pin.

Please Log in to join the conversation.

More
7 months 2 weeks ago #344189 by TAKUYA
Replied by TAKUYA on topic Ethercat random jitter fix
So pll-error remains 0 for me. App phase hovering between 9800 and 10500

What i did notice, i had all 3 axis vibrate like crazy upon first Launch today. Rebooting lcnc fixed that.

Edit: did another power cycle. No Vibration but sudden clunking on all 3 axis again

Please Log in to join the conversation.

More
7 months 2 weeks ago #344209 by Atsu
Replied by Atsu on topic Ethercat random jitter fix
Experienced some jitter as well today while running grandixximo's fork. I doubt there is any synchronization being done while the dc-phased pin is false and all pll-*** pins are 0. Ran sudo dmesg to see what is happening in the background. Maybe someone can notice something clear in the log:

During startup:
[  801.691210] EtherCAT: Requesting master 0...
[  801.691215] EtherCAT: Successfully requested master 0.
[  801.691915] EtherCAT 0: Domain0: Logical address 0x00000000, 74 byte, expected working counter 12.
[  801.691918] EtherCAT 0:   Datagram domain0-0-main: Logical offset 0x00000000, 74 byte, type LRW.
[  801.691932] EtherCAT 0: Master thread exited.
[  801.691933] EtherCAT 0: Starting EtherCAT-OP thread.
[  801.691970] EtherCAT WARNING 0: 1 datagram UNMATCHED!

During operation:
[  807.424674] EtherCAT WARNING 0-1: Slave did not sync after 5000 ms.
[  808.944250] EtherCAT 0: Slave states on main device: OP.

During shutdown:
[  841.183382] EtherCAT 0: Master thread exited.
[  841.183401] EtherCAT 0: Starting EtherCAT-IDLE thread.
[  841.183449] EtherCAT ERROR 0-0: Failed to receive AL state datagram: Datagram initialized.
[  841.183462] EtherCAT 0: Releasing master...
[  841.183464] EtherCAT 0: Released.
[  841.188247] EtherCAT 0: Slave states on main device: SAFEOP, OP + ERROR.
[  841.196539] EtherCAT ERROR 0-0: AL status message 0x001A: "Synchronization error".
[  841.197317] EtherCAT 0-0: Acknowledged state SAFEOP.
[  841.208612] EtherCAT ERROR 0-1: AL status message 0x001A: "Synchronization error".
[  841.209116] EtherCAT 0-1: Acknowledged state SAFEOP.
[  841.220587] EtherCAT ERROR 0-2: AL status message 0x001A: "Synchronization error".
[  841.221590] EtherCAT 0-2: Acknowledged state SAFEOP.
[  841.256274] EtherCAT 0: Slave states on main device: PREOP.
[  841.828318] EtherCAT WARNING: Datagram 0000000031def7fe (master-fsm) was SKIPPED 1 time.

Please Log in to join the conversation.

More
7 months 2 weeks ago #344237 by grandixximo

I have the same issue as TAKUYA, the dc-phased pin never goes high. I managed to fix this using Claude AI and now the dc-phased pin goes high. I have attached a detailed notes of what was changed in the code as well as some screenshots from before and after.

NOTE: this was done entirely with the help of AI and I am not qualified enough to say whether it is correct or not! Grandixximo, could you please review the notes and share your thoughts? Opening the attached HTML file in a browser, you can view the notes which were generated. It is too long to copy-paste here.

EDIT: After testing the code more thoroughly, I'm getting jitter again. It's very likely that the AI's "fix" only addressed the dc-phased pin but broke something else... I've reverted back to the master branch ( github.com/grandixximo/linuxcnc-ethercat ), and everything is working again except for the dc-phased pin.
 

 
The issues the AI found are not issues,
  1. Same-cycle read — No, 
    Code:
    reference_clock_time
     reads from the previous cycle's response, already received in 
    Code:
    lcec_read_master
  2. Missing 
    Code:
    sync_monitor_queue
     — Not needed; DC ref clock time comes from the normal sync datagrams
  3. Code:
    sync_reference_clock
     guard — Intentionally skipped in M2R mode (correct behavior)
And not something related to my changes, the behaviors were already present from upstream, can you share your xml file? Your app time seems good, but the pll-err is suspiciously to 0
which tells me you probably do not have DC, is assignActivate set to 300 on your DC capable devices?
Your first device lacks DC, I would advise to keep the first device on your network to be DC active, or you may need to set which slave does the DC timing reference in the XML, I always keep first device DC, have not tested differently, can't say for sure.
  • EK1100 — coupler, no DC
  • EL1008 — digital input, no DC
  • EL2008 — digital output, no DC
  • 3× StepperOnline A6 — these should have DC, but only if 
    Code:
    assignActivate
     is set in the XML config
In your XML you will need something like

    <master idx="0" appTimePeriod="1000000" refClockSyncCycles="-1">
        <slave idx="xx" type="devicewith DC" >
        <dcConf assignActivate="300" sync0Cycle="*1" sync0Shift="250000"/>
        </slave>Or you share your XML here, I will take a look, I think you do not have DC at all at the moment, you never even used any of my code changes at all.

---
test signature

Please Log in to join the conversation.

More
7 months 2 weeks ago #344239 by grandixximo

Experienced some jitter as well today while running grandixximo's fork. I doubt there is any synchronization being done while the dc-phased pin is false and all pll-*** pins are 0. Ran sudo dmesg to see what is happening in the background. Maybe someone can notice something clear in the log:

During startup:
[  801.691210] EtherCAT: Requesting master 0...
[  801.691215] EtherCAT: Successfully requested master 0.
[  801.691915] EtherCAT 0: Domain0: Logical address 0x00000000, 74 byte, expected working counter 12.
[  801.691918] EtherCAT 0:   Datagram domain0-0-main: Logical offset 0x00000000, 74 byte, type LRW.
[  801.691932] EtherCAT 0: Master thread exited.
[  801.691933] EtherCAT 0: Starting EtherCAT-OP thread.
[  801.691970] EtherCAT WARNING 0: 1 datagram UNMATCHED!

During operation:
[  807.424674] EtherCAT WARNING 0-1: Slave did not sync after 5000 ms.
[  808.944250] EtherCAT 0: Slave states on main device: OP.

During shutdown:
[  841.183382] EtherCAT 0: Master thread exited.
[  841.183401] EtherCAT 0: Starting EtherCAT-IDLE thread.
[  841.183449] EtherCAT ERROR 0-0: Failed to receive AL state datagram: Datagram initialized.
[  841.183462] EtherCAT 0: Releasing master...
[  841.183464] EtherCAT 0: Released.
[  841.188247] EtherCAT 0: Slave states on main device: SAFEOP, OP + ERROR.
[  841.196539] EtherCAT ERROR 0-0: AL status message 0x001A: "Synchronization error".
[  841.197317] EtherCAT 0-0: Acknowledged state SAFEOP.
[  841.208612] EtherCAT ERROR 0-1: AL status message 0x001A: "Synchronization error".
[  841.209116] EtherCAT 0-1: Acknowledged state SAFEOP.
[  841.220587] EtherCAT ERROR 0-2: AL status message 0x001A: "Synchronization error".
[  841.221590] EtherCAT 0-2: Acknowledged state SAFEOP.
[  841.256274] EtherCAT 0: Slave states on main device: PREOP.
[  841.828318] EtherCAT WARNING: Datagram 0000000031def7fe (master-fsm) was SKIPPED 1 time.

 
This seems a similar issue, please share your XML file
also send this command 

ethercat master

and share the output
the output should look something like

Master0
  Phase: Operation
  Active: yes
  Slaves: 12
  Ethernet devices:
    Main: 88:03:8f:28:ca:33 (attached)
      Link: UP
      Tx frames:   78788870
      Tx bytes:    65756113151
      Rx frames:   78788869
      Rx bytes:    65756112300
      Tx errors:   0
      Tx frame rate [1/s]:   1000   1000   1000
      Tx rate [KByte/s]:    822.0  822.0  821.9
      Rx frame rate [1/s]:   1000   1000   1000
      Rx rate [KByte/s]:    822.0  822.0  821.9
    Common:
      Tx frames:   78788870
      Tx bytes:    65756113151
      Rx frames:   78788869
      Rx bytes:    65756112300
      Lost frames: 0
      Tx frame rate [1/s]:   1000   1000   1000
      Tx rate [KByte/s]:    822.0  822.0  821.9
      Rx frame rate [1/s]:   1000   1000   1000
      Rx rate [KByte/s]:    822.0  822.0  821.9
      Loss rate [1/s]:          0     -0      0
      Frame loss [%]:         0.0   -0.0    0.0
  Distributed clocks:
    Reference clock:   Slave 0
    DC reference time: 826799805203739108
    Application time:  826800839056809695
                       2026-03-14 10:53:59.056809695

 

---
test signature

Please Log in to join the conversation.

More
7 months 2 weeks ago #344240 by grandixximo
@TAKUYA
@Atsu

Please share your full XML
and the output of ethercat master command after you have started linuxcnc

---
test signature

Please Log in to join the conversation.

More
7 months 2 weeks ago #344241 by Atsu
Replied by Atsu on topic Ethercat random jitter fix
Thank you for taking the time to go through the messages from the last few days. I suspected as much, the AI's "fix" wasn't really anything useful.

Regarding the first slave lacking DC, you are correct. I actually changed the order of slaves in the chain yesterday so that the A6 servos are now defined first. They do have assignActivate defined. However, that change didn't make any difference to the dc-phased pin or the pll-*** pins.

I have attached my ethercat-conf.xml along with HAL and INI files if they are of any help.

ethercat master command after LinuxCNC has started:

Master0
  Phase: Operation
  Active: yes
  Slaves: 6
  Ethernet devices:
    Main: 10:60:4b:5b:58:34 (attached)
      Link: UP
      Tx frames:   146273
      Tx bytes:    9824436
      Rx frames:   146272
      Rx bytes:    9824318
      Tx errors:   0
      Tx frame rate [1/s]:    800    709    308
      Tx rate [KByte/s]:     93.7   84.1   32.2
      Rx frame rate [1/s]:    800    709    308
      Rx rate [KByte/s]:     93.7   84.1   32.2
    Common:
      Tx frames:   146273
      Tx bytes:    9824436
      Rx frames:   146272
      Rx bytes:    9824318
      Lost frames: 0
      Tx frame rate [1/s]:    800    709    308
      Tx rate [KByte/s]:     93.7   84.1   32.2
      Rx frame rate [1/s]:    800    709    308
      Rx rate [KByte/s]:     93.7   84.1   32.2
      Loss rate [1/s]:          0     -0      0
      Frame loss [%]:         0.0   -0.0    0.0
  Distributed clocks:
    Reference clock:   Slave 0
    DC reference time: 826802114159640000
    Application time:  826802133398422723
                       2026-03-14 11:15:33.398422723

Please Log in to join the conversation.

Time to create page: 0.277 seconds
Powered by Kunena Forum