Mesa 7i76e smart Serial error
- chris@cnc
- Offline
- Platinum Member
Less
More
- Posts: 529
- Thank you received: 139
12 Apr 2022 19:15 #240073
by chris@cnc
Mesa 7i76e smart Serial error was created by chris@cnc
Since update to linuxcnc 2.9 i observe 3rd time this error. But i can not see any reason for this.Problem happened after 2h program runtime. Has someone the same error or know how to fix it?terminal messageKernelVersionLinuxcncversion
hm2/hm2_7i76e.0: Smart Serial port 0: DoIt not cleared from previous servo thread. Servo thread rate probably too fast. This message will not be repeated, but the hm2_7i76e.0.sserial.0.fault-count pin will indicate if this is happening frequently.
Linux cnc 5.10.0-13-rt-amd64 #1 SMP PREEMPT_RT Debian 5.10.106-1 (2022-03-17) x86_64 GNU/Linux
2.9.0~pre0+git20220402.2500863908-4
Attachments:
Please Log in or Create an account to join the conversation.
- PCW
- Offline
- Moderator
Less
More
- Posts: 17967
- Thank you received: 4824
12 Apr 2022 20:42 #240080
by PCW
Replied by PCW on topic Mesa 7i76e smart Serial error
This means that there is a real time error, most likely related to
Ethernet latency.
If you ping the 7I76E card, what times do you get?
You can also take a look at the various tmax values to see what is
causing the real time issue (these times are in units of CPU clocks
on X86 hardware)
halcmd show param *.tmax
Ethernet latency.
If you ping the 7I76E card, what times do you get?
You can also take a look at the various tmax values to see what is
causing the real time issue (these times are in units of CPU clocks
on X86 hardware)
halcmd show param *.tmax
The following user(s) said Thank You: chris@cnc
Please Log in or Create an account to join the conversation.
- chris@cnc
- Offline
- Platinum Member
Less
More
- Posts: 529
- Thank you received: 139
13 Apr 2022 06:26 #240109
by chris@cnc
Replied by chris@cnc on topic Mesa 7i76e smart Serial error
I take a look at ping and tmax. I'm not sure, but on my old kernel in stretch my ping time was more stable. I was close to 0.1*** but almost the same.
Is this might because of the bad real-time latency in my new kernel?
halcmd show param *.tmax
Is this might because of the bad real-time latency in my new kernel?
Warning: Spoiler!
PING 10.10.10.10 (10.10.10.10) 56(84) bytes of data.
64 bytes from 10.10.10.10: icmp_seq=1 ttl=64 time=0.143 ms
64 bytes from 10.10.10.10: icmp_seq=2 ttl=64 time=0.085 ms
64 bytes from 10.10.10.10: icmp_seq=3 ttl=64 time=0.074 ms
64 bytes from 10.10.10.10: icmp_seq=4 ttl=64 time=0.088 ms
64 bytes from 10.10.10.10: icmp_seq=5 ttl=64 time=0.088 ms
64 bytes from 10.10.10.10: icmp_seq=6 ttl=64 time=0.109 ms
64 bytes from 10.10.10.10: icmp_seq=7 ttl=64 time=0.088 ms
64 bytes from 10.10.10.10: icmp_seq=8 ttl=64 time=0.073 ms
64 bytes from 10.10.10.10: icmp_seq=9 ttl=64 time=0.084 ms
64 bytes from 10.10.10.10: icmp_seq=10 ttl=64 time=0.072 ms
64 bytes from 10.10.10.10: icmp_seq=11 ttl=64 time=0.073 ms
64 bytes from 10.10.10.10: icmp_seq=12 ttl=64 time=0.076 ms
64 bytes from 10.10.10.10: icmp_seq=13 ttl=64 time=0.113 ms
64 bytes from 10.10.10.10: icmp_seq=14 ttl=64 time=0.086 ms
64 bytes from 10.10.10.10: icmp_seq=15 ttl=64 time=0.080 ms
64 bytes from 10.10.10.10: icmp_seq=16 ttl=64 time=0.097 ms
64 bytes from 10.10.10.10: icmp_seq=17 ttl=64 time=0.117 ms
64 bytes from 10.10.10.10: icmp_seq=18 ttl=64 time=0.107 ms
64 bytes from 10.10.10.10: icmp_seq=19 ttl=64 time=0.104 ms
Parameters:
Owner Type Dir Value Name
45 s32 RW 7554 and2.0.tmax
45 s32 RW 3867 and2.1.tmax
45 s32 RW 5688 and2.2.tmax
42 s32 RW 5355 estop-latch.0.tmax
42 s32 RW 6981 estop-latch.1.tmax
36 s32 RW 0 hm2_7i76e.0.read-request.tmax
36 s32 RW 2693406 hm2_7i76e.0.read.tmax
36 s32 RW 146577 hm2_7i76e.0.write.tmax
54 s32 RW 2931 mist.0.tmax
29 s32 RW 16074 motion-command-handler.tmax
29 s32 RW 113652 motion-controller.tmax
82 s32 RW 8106 mpg.0.tmax
70 s32 RW 5436 mpg2.tmax
70 s32 RW 9339 mpg2step.tmax
70 s32 RW 6906 mpgstep.tmax
76 s32 RW 5976 mux-gen.00.tmax
85 s32 RW 5298 oneshot.0.tmax
48 s32 RW 7059 or2.0.tmax
48 s32 RW 4926 or2.1.tmax
48 s32 RW 3882 or2.2.tmax
48 s32 RW 4254 or2.3.tmax
48 s32 RW 2928 or2.4.tmax
48 s32 RW 4443 or2.5.tmax
48 s32 RW 5148 or2.6.tmax
48 s32 RW 6876 or2.7.tmax
48 s32 RW 3600 or2.8.tmax
39 s32 RW 9771 pid.a.do-pid-calcs.tmax
39 s32 RW 14205 pid.s.do-pid-calcs.tmax
39 s32 RW 9249 pid.x.do-pid-calcs.tmax
39 s32 RW 9360 pid.y.do-pid-calcs.tmax
39 s32 RW 9630 pid.y2.do-pid-calcs.tmax
39 s32 RW 9744 pid.z.do-pid-calcs.tmax
39 s32 RW 11709 pid.z2.do-pid-calcs.tmax
70 s32 RW 7692 rapid.tmax
79 s32 RW 7008 select8.0.tmax
30 s32 RW 2915763 servo-thread.tmax
57 s32 RW 6126 time.0.tmax
51 s32 RW 5844 timedelay.0.tmax
73 s32 RW 6750 toggle.0.tmax
73 s32 RW 6300 toggle.1.tmax
73 s32 RW 4704 toggle.2.tmax
Please Log in or Create an account to join the conversation.
- chris@cnc
- Offline
- Platinum Member
Less
More
- Posts: 529
- Thank you received: 139
13 Apr 2022 08:06 - 13 Apr 2022 08:07 #240119
by chris@cnc
Replied by chris@cnc on topic Mesa 7i76e smart Serial error
Now alarm happen again.
Last edit: 13 Apr 2022 08:07 by chris@cnc.
Please Log in or Create an account to join the conversation.
- tommylight
- Away
- Moderator
Less
More
- Posts: 19479
- Thank you received: 6532
13 Apr 2022 10:04 #240124
by tommylight
Replied by tommylight on topic Mesa 7i76e smart Serial error
What is the servo period set at in the ini file?
Tmax of 2.9 million is no good anyway, so might want to try an older RT kernel, 4.9 was pretty good.
Tmax of 2.9 million is no good anyway, so might want to try an older RT kernel, 4.9 was pretty good.
Please Log in or Create an account to join the conversation.
- chris@cnc
- Offline
- Platinum Member
Less
More
- Posts: 529
- Thank you received: 139
13 Apr 2022 11:55 #240137
by chris@cnc
Replied by chris@cnc on topic Mesa 7i76e smart Serial error
SERVO_PERIOD = 1000000
I wonder that i get alarm so late?
I can't back to 4.9 in bullseye... But this is my second pc to test new versions.
I change pc back to stretch now and get much better numbers.
I wonder that i get alarm so late?
I can't back to 4.9 in bullseye... But this is my second pc to test new versions.
I change pc back to stretch now and get much better numbers.
Please Log in or Create an account to join the conversation.
- PCW
- Offline
- Moderator
Less
More
- Posts: 17967
- Thank you received: 4824
13 Apr 2022 14:29 #240146
by PCW
Replied by PCW on topic Mesa 7i76e smart Serial error
My experience is that 4.X kernels are better for Ethernet latency than 5.X
(4.19 being the latest 4.X kernel AFAIK)
(4.19 being the latest 4.X kernel AFAIK)
Please Log in or Create an account to join the conversation.
- rodw
- Offline
- Platinum Member
Less
More
- Posts: 10795
- Thank you received: 3556
13 Apr 2022 18:39 #240174
by rodw
Replied by rodw on topic Mesa 7i76e smart Serial error
Sometimes, I think that is due to poor NIC driver choice by the Debian installer. Recently I got an old i3 All in One PC from MSI running and installing the right driver for the RT8111 NIC plus isolcpus=2,3 made an enormous difference. The best I've ever had and only AUD $120 for pc + touch screen.My experience is that 4.X kernels are better for Ethernet latency than 5.X
(4.19 being the latest 4.X kernel AFAIK)
Please Log in or Create an account to join the conversation.
- chris@cnc
- Offline
- Platinum Member
Less
More
- Posts: 529
- Thank you received: 139
14 Apr 2022 17:56 - 14 Apr 2022 18:16 #240247
by chris@cnc
Replied by chris@cnc on topic Mesa 7i76e smart Serial error
I wanted to investigate this in more detail and could kernel 4.9. and 4.19. install in bullseye.
Did some latency test today. Comparison of servo thread times with kernel 4.9, 4.19. and 5.10. The result is: I haven't measured the same times with any of them as with kernel 4.19. without isolcpus in stretch with linuxcnc 2.8.2 and the same hardware. The latency histogram looked in kernel 5.10. significantly better than under 4.9. but sporadically there are very large peaks in the servo thread. in 4.9. the peaks are only half the size. I guess there must be more why there are so big differences between stretch + linuxcnc 2.8.2 and bullseye + linuxcnc 2.9.
Does anyone have better times in master?
measured values in stretch + liunxcnc 2.8.2. + 4.19.0-0.bpo.18-rt-amd64 + noisolcpus
This was max data after one working day.
The network card is a PCI express card with rtl8111h chip but it seems r8169 driver is loaded
ping times and max servo thread
kernel 4.9.0-13-rt-amd64 and isolcpus=1,3 in bullseye and linuxcnc 2.9. last build
ping times and max servo thread
kernel 4.19.0-0.bpo.18-rt-amd64 isolcpus=1,3 in bullseye and linuxcnc 2.9. last build
ping times and max servo thread
kernel 5.10.0-12-rt-amd64 isolcpus=1,3 in bullseye and linuxcnc 2.9. last build
ping times and max servo thread
Did some latency test today. Comparison of servo thread times with kernel 4.9, 4.19. and 5.10. The result is: I haven't measured the same times with any of them as with kernel 4.19. without isolcpus in stretch with linuxcnc 2.8.2 and the same hardware. The latency histogram looked in kernel 5.10. significantly better than under 4.9. but sporadically there are very large peaks in the servo thread. in 4.9. the peaks are only half the size. I guess there must be more why there are so big differences between stretch + linuxcnc 2.8.2 and bullseye + linuxcnc 2.9.
Does anyone have better times in master?
measured values in stretch + liunxcnc 2.8.2. + 4.19.0-0.bpo.18-rt-amd64 + noisolcpus
This was max data after one working day.
The network card is a PCI express card with rtl8111h chip but it seems r8169 driver is loaded
sudo dmesg | grep r8169
[ 1.610124] r8169 0000:02:00.0: can't disable ASPM; OS doesn't have ASPM control
[ 1.610131] r8169 0000:02:00.0: enabling device (0100 -> 0103)
[ 1.629045] libphy: r8169: probed
[ 1.629352] r8169 0000:02:00.0 eth0: RTL8168h/8111h, 00:e0:72:59:68:c5, XID 54100800, IRQ 27
[ 1.629353] r8169 0000:02:00.0 eth0: jumbo features [frames: 9200 bytes, tx checksumming: ko]
[ 1.630378] r8169 0000:02:00.0 enp2s0: renamed from eth0
[ 7.163738] r8169 0000:02:00.0: firmware: direct-loading firmware rtl_nic/rtl8168h-2.fw
[ 7.163993] Generic PHY r8169-200:00: attached PHY driver [Generic PHY] (mii_bus:phy_addr=r8169-200:00, irq=IGNORE)
[ 8.851605] r8169 0000:02:00.0 enp2s0: Link is Up - 100Mbps/Full - flow control rx/tx
ping times and max servo thread
64 bytes from 10.10.10.10: icmp_seq=15 ttl=64 time=0.095 ms
64 bytes from 10.10.10.10: icmp_seq=16 ttl=64 time=0.081 ms
64 bytes from 10.10.10.10: icmp_seq=17 ttl=64 time=0.082 ms
64 bytes from 10.10.10.10: icmp_seq=18 ttl=64 time=0.082 ms
26 s32 RW 805052 hm2_7i76e.0.read.tmax
26 s32 RW 147144 hm2_7i76e.0.write.tmax
19 s32 RW 105820 motion-command-handler.tmax
19 s32 RW 166196 motion-controller.tmax
20 s32 RW 951840 servo-thread.tmax
kernel 4.9.0-13-rt-amd64 and isolcpus=1,3 in bullseye and linuxcnc 2.9. last build
ping times and max servo thread
64 bytes from 10.10.10.10: icmp_seq=9 ttl=64 time=0.135 ms
64 bytes from 10.10.10.10: icmp_seq=10 ttl=64 time=0.131 ms
64 bytes from 10.10.10.10: icmp_seq=11 ttl=64 time=0.130 ms
36 s32 RW 952614 hm2_7i76e.0.read.tmax
36 s32 RW 215580 hm2_7i76e.0.write.tmax
29 s32 RW 190532 motion-command-handler.tmax
29 s32 RW 285531 motion-controller.tmax
30 s32 RW 1456206 servo-thread.tmax
kernel 4.19.0-0.bpo.18-rt-amd64 isolcpus=1,3 in bullseye and linuxcnc 2.9. last build
ping times and max servo thread
64 bytes from 10.10.10.10: icmp_seq=6 ttl=64 time=0.276 ms
64 bytes from 10.10.10.10: icmp_seq=7 ttl=64 time=0.242 ms
64 bytes from 10.10.10.10: icmp_seq=8 ttl=64 time=0.314 ms
36 s32 RW 1190733 hm2_7i76e.0.read.tmax
36 s32 RW 349740 hm2_7i76e.0.write.tmax
29 s32 RW 175116 motion-command-handler.tmax
29 s32 RW 348812 motion-controller.tmax
30 s32 RW 1745328 servo-thread.tmax
kernel 5.10.0-12-rt-amd64 isolcpus=1,3 in bullseye and linuxcnc 2.9. last build
ping times and max servo thread
64 bytes from 10.10.10.10: icmp_seq=5 ttl=64 time=0.113 ms
64 bytes from 10.10.10.10: icmp_seq=6 ttl=64 time=0.078 ms
64 bytes from 10.10.10.10: icmp_seq=7 ttl=64 time=0.085 ms
36 s32 RW 2742591 hm2_7i76e.0.read.tmax
36 s32 RW 291285 hm2_7i76e.0.write.tmax
29 s32 RW 206667 motion-command-handler.tmax
29 s32 RW 278103 motion-controller.tmax
30 s32 RW 3067704 servo-thread.tmax
Last edit: 14 Apr 2022 18:16 by chris@cnc.
Please Log in or Create an account to join the conversation.
- tommylight
- Away
- Moderator
Less
More
- Posts: 19479
- Thank you received: 6532
14 Apr 2022 19:05 #240254
by tommylight
Replied by tommylight on topic Mesa 7i76e smart Serial error
Not good,
Did you disable everything regarding power saving, virtualisation, C states, hyperthreading, turbo, speedstep, etc etc???
Did you disable everything regarding power saving, virtualisation, C states, hyperthreading, turbo, speedstep, etc etc???
Please Log in or Create an account to join the conversation.
Time to create page: 0.243 seconds