Mesa 7i92 Read Errors

More
26 Jun 2022 01:44 - 26 Jun 2022 02:31 #245896 by silden
Mesa 7i92 Read Errors was created by silden
Hi all,

I just changed the computer on my machine to an Asus PN50 from a Intel NUC 10 due to graphics issues.
While the graphics issues are gone, and the overall latencies look phenomenal, I am now getting the "Error Finishing Read", usually after less than 1 minute of running.
When it first ran, I checked the read and write latencies, which came out at around 10000 (without errors).

I already tried changing the Ethernet cable (currently a shielded CAT6), and the power supply should also not be any issue.

I have an Mesa 7i92 attached to a 7i77, 7i85s, and 7i73.
Also, I am running a custom lubuntu install with linux kernel version 5.18 with the RT patch applied and configured.

Thank you for reading.
 
Attachments:
Last edit: 26 Jun 2022 02:31 by silden.

Please Log in or Create an account to join the conversation.

More
26 Jun 2022 10:43 #245905 by tommylight
Replied by tommylight on topic Mesa 7i92 Read Errors
If you patched the kernel, you have to choose "fully preemptible kernel", otherwise it will not work.
Looking at the histogram, it does look that you have done that, so do a ping to Mesa 7i92 and check the reply times, and post them here.

Please Log in or Create an account to join the conversation.

More
26 Jun 2022 22:03 - 26 Jun 2022 22:05 #245960 by silden
Replied by silden on topic Mesa 7i92 Read Errors
Hi tommylight,
I just ran two ping test to the 7i92, and they are well within the timeout.


Standard
ping 10.10.10.10
--- 10.10.10.10 ping statistics
--- 505 packets transmitted, 505 received, 0% packet loss, time 516085m
rtt min/avg/max/mdev = 0.090/0.122/0.162/0.014 ms

Flood Ping (
sudo ping 10.10.10.10 -f -c 500
)
PING 10.10.10.10 (10.10.10.10) 56(84) bytes of data.

--- 10.10.10.10 ping statistics ---
500 packets transmitted, 500 received, 0% packet loss, time 38ms
rtt min/avg/max/mdev = 0.046/0.069/0.600/0.025 ms, ipg/ewma 0.075/0.066 ms
Last edit: 26 Jun 2022 22:05 by silden.

Please Log in or Create an account to join the conversation.

More
26 Jun 2022 23:12 #245966 by tommylight
Replied by tommylight on topic Mesa 7i92 Read Errors
latency-histogram --nobase --sbinsize 1000 --show
Do run several glxgears and resize their windows while that is running.
Ping looks fine, so power supply or interference, if there are no excursions on the above test.

Please Log in or Create an account to join the conversation.

More
26 Jun 2022 23:39 - 26 Jun 2022 23:40 #245969 by silden
Replied by silden on topic Mesa 7i92 Read Errors
Hi timmylight,
Thank you for the suggestions, we already ran the latency histogram with 10 glxgears, for a longer period than the screenshot above showed.
We will run it again tomorrow because we decided to rebuild the kernel to install the latest RealTek Ethernet driver.

The power supply is a din rail mounted 5V 3A power supply from MeanWell powering only the MESA cards and some optocouplers.
As for interference, it is extremely unlikely as we are using a shielded CAT6 cable that is run parallel to only 24V lines.

 
Attachments:
Last edit: 26 Jun 2022 23:40 by silden.

Please Log in or Create an account to join the conversation.

More
26 Jun 2022 23:50 #245971 by tommylight
Replied by tommylight on topic Mesa 7i92 Read Errors
By any chance, does the error occur when you enable the servo drives?
I would have not installed the Mesa stuff that close to servo and VFD drives, it would be much better near 220V stuff if there is no other choice.
If it is a Mesa 7i92M, attach it directly to Mesa 7i77 and remove the ribbon cable, less interference.
And since i am on a roll, if that is a Mesa 7i77, i would venture a guess that those DMM drives are set to velocity mode, but since i see another Mesa card below i might be wrong.

Please Log in or Create an account to join the conversation.

More
27 Jun 2022 00:56 #245976 by silden
Replied by silden on topic Mesa 7i92 Read Errors
Hi tommylight,
The issue happens with and without the drives are energized.
Furthermore, I already disconnected both cards from the 7i92 and the issue persisted.

And while you are correct in the DYN-4 being connected to an analog signal from the 7i77 with velocity mode since I have linear scales on 2 axes, with the 3rd axes being controlled via Step/Direction from the 7i85, the drives are not enabled due to the mesa card losing communication well before I enable them.

Please Log in or Create an account to join the conversation.

More
27 Jun 2022 01:02 #245978 by tommylight
Replied by tommylight on topic Mesa 7i92 Read Errors
Do you have another PC to test with?
If not, find a USB cable, cut the small header, wire the Red and Black wires to Mesa 7i92 5V power connector and insert it into a USB port on that same PC with nothing else connected to Mesa 7i92 but power and ethernet. Mind the polarity.

Please Log in or Create an account to join the conversation.

More
27 Jun 2022 22:01 - 27 Jun 2022 23:08 #246027 by PCW
Replied by PCW on topic Mesa 7i92 Read Errors
1. What is your servo thread period?
2. Have you checked the tmax values when running LinuxcNC?

halcmd show param *.tmax

To interpret these values, the CPU frequency is also needed.
(at the tmax values are in CPU clocks)

I should mention that you can eliminate power supply issues
(and throw suspicion on timing issues) if  the /INIT light is
illuminated on the 7I92 after you get a read error.

 
Last edit: 27 Jun 2022 23:08 by PCW.

Please Log in or Create an account to join the conversation.

More
28 Jun 2022 03:22 #246049 by silden
Replied by silden on topic Mesa 7i92 Read Errors
Hi PCW,
Thank you for the points to look at.

I just tested the read and write tmaxes and they jump up to 1ms for some cycles, with my servo thread cycle time being 1ms as well.
Also the INIT light stays on after the error ocurrs, confirming that it is a timing issue on the ethernet interface.
I’m currently in the process of chasing down why it it jumps up sometimes.

Please Log in or Create an account to join the conversation.

Moderators: PCWjmelson
Time to create page: 0.189 seconds
Powered by Kunena Forum