Ethernet latency issue with 2.9.2 and 7i95T
20 Mar 2024 15:26 #296390
by Finngineering
Ethernet latency issue with 2.9.2 and 7i95T was created by Finngineering
Hello,
I'm currently "restoring" a Maho MH600T to use LinuxCNC, with a Mesa 7i95T as the main interface card:
forum.linuxcnc.org/30-cnc-machines/49758...mizing-a-maho-mh600t
Unfortunately, I have run into issues with ethernet latency, and get unexpected realtime delays and watchdog has bit issues. From monitoring the hm2_7i95.0.read.time, it's evident there are some massive spikes. See the screenshot below and console output from starting LinuxCNC. The highest spike is around 20 000 000, while the base level is around 300 000.
The computer is a HP Z240 SFF Workstation (J9C02ET#ABU) with i7-6700 CPU. I have done quite a bit of reading about other similar issues. The computer has an in-built I219-LM ethernet adapter, that I read somebody else had issues with. So I bought a PCI-e TP-Link TG-3468, which has the Realtek RTL 8168 chipset. But I get the same issues with both cards. And later I read there are also issues with the Realtek chipsets. And it's not easy to find anything else than Intel or Realtek.
I have disable all "performance" and power management features I can find in the BIOS, and modified the kernel options to:Still, no better.
As you might see from the kernel boot options above, I also compiled a custom kernel (actually several), as I read that the issue was with the stock debian kernel. It didn't help me. I followed the instructions here:
forum.linuxcnc.org/10-advanced-configura...6-7-improved-latency
According to some troubleshooting advice by PCM, I also did some ping test:and for a couple of minutes:With a maximum response time of almost 10 ms, no wonder there are some issues.
However, the "normal" system latency does not appear bad. Here is a latency-histogram from surfing the web and writing this post, playing youtube and having 5 glxgears running:
I actually used this same computer on 2.8.4 earlier and without many issues. However, I do think I got the same issues every once in a while. Maybe after an hour or so. But that was with no tuning of BIOS or kernel options and with the Intel network card without changing irq coalescing options (if that was at all relevant at that time).
I start to run out of ideas. But maybe somebody here has some advice?
I'm currently "restoring" a Maho MH600T to use LinuxCNC, with a Mesa 7i95T as the main interface card:
forum.linuxcnc.org/30-cnc-machines/49758...mizing-a-maho-mh600t
Unfortunately, I have run into issues with ethernet latency, and get unexpected realtime delays and watchdog has bit issues. From monitoring the hm2_7i95.0.read.time, it's evident there are some massive spikes. See the screenshot below and console output from starting LinuxCNC. The highest spike is around 20 000 000, while the base level is around 300 000.
Warning: Spoiler!
The computer is a HP Z240 SFF Workstation (J9C02ET#ABU) with i7-6700 CPU. I have done quite a bit of reading about other similar issues. The computer has an in-built I219-LM ethernet adapter, that I read somebody else had issues with. So I bought a PCI-e TP-Link TG-3468, which has the Realtek RTL 8168 chipset. But I get the same issues with both cards. And later I read there are also issues with the Realtek chipsets. And it's not easy to find anything else than Intel or Realtek.
I have disable all "performance" and power management features I can find in the BIOS, and modified the kernel options to:
$ cat /proc/cmdline
BOOT_IMAGE=/boot/vmlinuz-6.6.7-rt18-linuxcnc root=UUID=a2994d0f-a7e5-42a0-a350-8616556fa365 ro quiet idle=poll intel_idle.max_cstate=1 processor.max_cstate=1 i915.enable_dc=0As you might see from the kernel boot options above, I also compiled a custom kernel (actually several), as I read that the issue was with the stock debian kernel. It didn't help me. I followed the instructions here:
forum.linuxcnc.org/10-advanced-configura...6-7-improved-latency
According to some troubleshooting advice by PCM, I also did some ping test:
$ ping -c 5 -i .2 10.10.10.10
PING 10.10.10.10 (10.10.10.10) 56(84) bytes of data.
64 bytes from 10.10.10.10: icmp_seq=1 ttl=64 time=0.101 ms
64 bytes from 10.10.10.10: icmp_seq=2 ttl=64 time=0.061 ms
64 bytes from 10.10.10.10: icmp_seq=3 ttl=64 time=0.067 ms
64 bytes from 10.10.10.10: icmp_seq=4 ttl=64 time=0.061 ms
64 bytes from 10.10.10.10: icmp_seq=5 ttl=64 time=0.060 ms
--- 10.10.10.10 ping statistics ---
5 packets transmitted, 5 received, 0% packet loss, time 814ms
rtt min/avg/max/mdev = 0.060/0.070/0.101/0.015 ms$ sudo chrt 99 ping -i .001 -q 10.10.10.10
PING 10.10.10.10 (10.10.10.10) 56(84) bytes of data.
--- 10.10.10.10 ping statistics ---
98834 packets transmitted, 98833 received, 0.0010118% packet loss, time 99322ms
rtt min/avg/max/mdev = 0.049/0.068/9.884/0.133 msHowever, the "normal" system latency does not appear bad. Here is a latency-histogram from surfing the web and writing this post, playing youtube and having 5 glxgears running:
I actually used this same computer on 2.8.4 earlier and without many issues. However, I do think I got the same issues every once in a while. Maybe after an hour or so. But that was with no tuning of BIOS or kernel options and with the Intel network card without changing irq coalescing options (if that was at all relevant at that time).
I start to run out of ideas. But maybe somebody here has some advice?
Attachments:
Please Log in or Create an account to join the conversation.
20 Mar 2024 20:49 #296408
by Finngineering
Please Log in or Create an account to join the conversation.
Time to create page: 0.101 seconds