Mesa 7i76e smart Serial error

More
12 Apr 2022 19:15 #240073 by chris@cnc
Mesa 7i76e smart Serial error was created by chris@cnc
Since update to linuxcnc 2.9 i observe 3rd time this error. But i can not see any reason for this.Problem happened after 2h program runtime. Has someone the same error or know how to fix it?terminal message
hm2/hm2_7i76e.0: Smart Serial port 0: DoIt not cleared from previous servo thread. Servo thread rate probably too fast. This message will not be repeated, but the hm2_7i76e.0.sserial.0.fault-count pin will indicate if this is happening frequently.
KernelVersion
Linux cnc 5.10.0-13-rt-amd64 #1 SMP PREEMPT_RT Debian 5.10.106-1 (2022-03-17) x86_64 GNU/Linux
Linuxcncversion
2.9.0~pre0+git20220402.2500863908-4
 
 
Attachments:

Please Log in or Create an account to join the conversation.

More
12 Apr 2022 20:42 #240080 by PCW
Replied by PCW on topic Mesa 7i76e smart Serial error
This means that there is a real time error, most likely related to
Ethernet latency.

If you ping the 7I76E card, what times do you get?

You can also take a look at the various tmax values to see what is
causing the real time issue (these times are in units of CPU clocks
on X86 hardware)

halcmd show param *.tmax
The following user(s) said Thank You: chris@cnc

Please Log in or Create an account to join the conversation.

More
13 Apr 2022 06:26 #240109 by chris@cnc
Replied by chris@cnc on topic Mesa 7i76e smart Serial error
I take a look at ping and tmax. I'm not sure, but on my old kernel in stretch my ping time was more stable. I was close to 0.1*** but almost the same.
Is this might because of the bad real-time latency in my new kernel?
Warning: Spoiler!

Please Log in or Create an account to join the conversation.

More
13 Apr 2022 08:06 - 13 Apr 2022 08:07 #240119 by chris@cnc
Replied by chris@cnc on topic Mesa 7i76e smart Serial error
Now alarm happen again.
 
Attachments:
Last edit: 13 Apr 2022 08:07 by chris@cnc.

Please Log in or Create an account to join the conversation.

  • tommylight
  • tommylight's Avatar
  • Away
  • Moderator
  • Moderator
More
13 Apr 2022 10:04 #240124 by tommylight
Replied by tommylight on topic Mesa 7i76e smart Serial error
What is the servo period set at in the ini file?
Tmax of 2.9 million is no good anyway, so might want to try an older RT kernel, 4.9 was pretty good.

Please Log in or Create an account to join the conversation.

More
13 Apr 2022 11:55 #240137 by chris@cnc
Replied by chris@cnc on topic Mesa 7i76e smart Serial error
SERVO_PERIOD = 1000000
I wonder that i get alarm so late?
I can't back to 4.9 in bullseye... But this is my second pc to test new versions.
I change pc back to stretch now and get much better numbers.
 

Please Log in or Create an account to join the conversation.

More
13 Apr 2022 14:29 #240146 by PCW
Replied by PCW on topic Mesa 7i76e smart Serial error
My experience is that 4.X kernels are better for Ethernet latency than 5.X
(4.19 being the latest 4.X kernel AFAIK)

Please Log in or Create an account to join the conversation.

More
13 Apr 2022 18:39 #240174 by rodw
Replied by rodw on topic Mesa 7i76e smart Serial error

My experience is that 4.X kernels are better for Ethernet latency than 5.X
(4.19 being the latest 4.X kernel AFAIK)

Sometimes, I think that is due to poor NIC driver choice by the Debian installer. Recently I got an old i3 All in One PC from MSI running and installing the right driver for the RT8111 NIC plus isolcpus=2,3 made an enormous difference. The best I've ever had and only AUD $120 for pc + touch screen.

Please Log in or Create an account to join the conversation.

More
14 Apr 2022 17:56 - 14 Apr 2022 18:16 #240247 by chris@cnc
Replied by chris@cnc on topic Mesa 7i76e smart Serial error
I wanted to investigate this in more detail and could kernel 4.9. and 4.19. install in bullseye.
Did some latency test today. Comparison of servo thread times with kernel 4.9, 4.19. and 5.10. The result is: I haven't measured the same times with any of them as with kernel 4.19. without isolcpus in stretch with linuxcnc 2.8.2 and the same hardware. The latency histogram looked in kernel 5.10. significantly better than under 4.9. but sporadically there are very large peaks in the servo thread. in 4.9. the peaks are only half the size. I guess there must be more why there are so big differences between stretch + linuxcnc 2.8.2 and bullseye + linuxcnc 2.9.
Does anyone have better times in master?

measured values in stretch + liunxcnc 2.8.2. + 4.19.0-0.bpo.18-rt-amd64 + noisolcpus
This was max data after one working day.
The network card is a PCI express card with rtl8111h chip but it seems r8169 driver is loaded
sudo dmesg | grep r8169
[    1.610124] r8169 0000:02:00.0: can't disable ASPM; OS doesn't have ASPM control
[    1.610131] r8169 0000:02:00.0: enabling device (0100 -> 0103)
[    1.629045] libphy: r8169: probed
[    1.629352] r8169 0000:02:00.0 eth0: RTL8168h/8111h, 00:e0:72:59:68:c5, XID 54100800, IRQ 27
[    1.629353] r8169 0000:02:00.0 eth0: jumbo features [frames: 9200 bytes, tx checksumming: ko]
[    1.630378] r8169 0000:02:00.0 enp2s0: renamed from eth0
[    7.163738] r8169 0000:02:00.0: firmware: direct-loading firmware rtl_nic/rtl8168h-2.fw
[    7.163993] Generic PHY r8169-200:00: attached PHY driver [Generic PHY] (mii_bus:phy_addr=r8169-200:00, irq=IGNORE)
[    8.851605] r8169 0000:02:00.0 enp2s0: Link is Up - 100Mbps/Full - flow control rx/tx

ping times and max servo thread
64 bytes from 10.10.10.10: icmp_seq=15 ttl=64 time=0.095 ms
64 bytes from 10.10.10.10: icmp_seq=16 ttl=64 time=0.081 ms
64 bytes from 10.10.10.10: icmp_seq=17 ttl=64 time=0.082 ms
64 bytes from 10.10.10.10: icmp_seq=18 ttl=64 time=0.082 ms

    26  s32   RW         805052  hm2_7i76e.0.read.tmax
    26  s32   RW         147144  hm2_7i76e.0.write.tmax
    19  s32   RW         105820  motion-command-handler.tmax
    19  s32   RW         166196  motion-controller.tmax
    20  s32   RW         951840  servo-thread.tmax

kernel  4.9.0-13-rt-amd64 and isolcpus=1,3 in bullseye and linuxcnc 2.9. last build
ping times and max servo thread
64 bytes from 10.10.10.10: icmp_seq=9 ttl=64 time=0.135 ms
64 bytes from 10.10.10.10: icmp_seq=10 ttl=64 time=0.131 ms
64 bytes from 10.10.10.10: icmp_seq=11 ttl=64 time=0.130 ms

    36  s32   RW         952614  hm2_7i76e.0.read.tmax
    36  s32   RW         215580  hm2_7i76e.0.write.tmax
    29  s32   RW         190532  motion-command-handler.tmax
    29  s32   RW         285531  motion-controller.tmax
    30  s32   RW        1456206  servo-thread.tmax

kernel 4.19.0-0.bpo.18-rt-amd64 isolcpus=1,3 in bullseye and linuxcnc 2.9. last build
ping times and max servo thread
64 bytes from 10.10.10.10: icmp_seq=6 ttl=64 time=0.276 ms
64 bytes from 10.10.10.10: icmp_seq=7 ttl=64 time=0.242 ms
64 bytes from 10.10.10.10: icmp_seq=8 ttl=64 time=0.314 ms
 
    36  s32   RW        1190733  hm2_7i76e.0.read.tmax
    36  s32   RW         349740  hm2_7i76e.0.write.tmax
    29  s32   RW         175116  motion-command-handler.tmax
    29  s32   RW         348812  motion-controller.tmax
    30  s32   RW        1745328  servo-thread.tmax

kernel  5.10.0-12-rt-amd64 isolcpus=1,3 in bullseye and linuxcnc 2.9. last build
ping times and max servo thread
64 bytes from 10.10.10.10: icmp_seq=5 ttl=64 time=0.113 ms
64 bytes from 10.10.10.10: icmp_seq=6 ttl=64 time=0.078 ms
64 bytes from 10.10.10.10: icmp_seq=7 ttl=64 time=0.085 ms
   
    36  s32   RW        2742591  hm2_7i76e.0.read.tmax
    36  s32   RW         291285  hm2_7i76e.0.write.tmax
    29  s32   RW         206667  motion-command-handler.tmax
    29  s32   RW         278103  motion-controller.tmax
    30  s32   RW        3067704  servo-thread.tmax
Attachments:
Last edit: 14 Apr 2022 18:16 by chris@cnc.

Please Log in or Create an account to join the conversation.

  • tommylight
  • tommylight's Avatar
  • Away
  • Moderator
  • Moderator
More
14 Apr 2022 19:05 #240254 by tommylight
Replied by tommylight on topic Mesa 7i76e smart Serial error
Not good,
Did you disable everything regarding power saving, virtualisation, C states, hyperthreading, turbo, speedstep, etc etc???

Please Log in or Create an account to join the conversation.

Time to create page: 0.243 seconds
Powered by Kunena Forum