Random read errors on Mesa 7i92

  • 10K
  • 10K's Avatar Topic Author
  • Offline
  • Premium Member
  • Premium Member
More
15 Jul 2016 14:27 #77495 by 10K
Replied by 10K on topic Random read errors on Mesa 7i92
I've had some success!

I ordered a new shielded ethernet cable. When I installed it, I found that my vent fan had shorted out earlier, and taken down the 24V and 5V power supplies, which killed power to the Mesa card. I think that this caused the read error that I mentioned a few posts back.

The new ethernet cable (and also the commercially made regular cable that I used earlier) have eliminated the

hm2/hm2_7i92.0: error finishing read! iter = xxxxxxxxxx

problems that I was experiencing. The old cable I was using was a home made cable that was about 25 years old, and I think that it had a weak connection somewhere that was causing the dropped packets.

I'm still getting the

Unexpected realtime delay on task 0 This message will only display once per session Run the latency test and resolve before continuing

message.

As I mentioned in an earlier post, I'm getting occasional delays of 45-55,000, but in general my latency is fairly low. My understanding is that these are acceptable for the USB Mesa card. I'm not sure what level of latency triggers the realtime delay message, but perhaps it's set at 50,000 or so, and I should just ignore the message.

Please Log in or Create an account to join the conversation.

More
15 Jul 2016 14:42 - 15 Jul 2016 14:44 #77499 by PCW
Replied by PCW on topic Random read errors on Mesa 7i92
The latency numbers in the latency test are really just dispatch latency
(the test doesn't actually do anything like allocate memory or perform I/O, things that can have their own latencies)

What is likely happening is that you are running out of time on the servo thread
to check this, watch

servo-thread.tmax

This is the maximum number of CPU clocks that the servo thread took to complete
If its getting close to servo_thread_period*CPU_Clock_Speed, you will get real time delay errors
The solution is to either improve the latency or raise the servo thread period
Last edit: 15 Jul 2016 14:44 by PCW.

Please Log in or Create an account to join the conversation.

  • 10K
  • 10K's Avatar Topic Author
  • Offline
  • Premium Member
  • Premium Member
More
15 Jul 2016 18:47 #77520 by 10K
Replied by 10K on topic Random read errors on Mesa 7i92
The ECS Liva X has a Baytrail (Atom) processor running at 1.58GHz (with a 2.25GHz Turbo).

My .ini settings are

[EMCMOT]
EMCMOT = motmod
COMM_TIMEOUT = 1.0
COMM_WAIT = 0.010
SERVO_PERIOD = 1000000

So 1,000,000 x 10E-9 sec x 1.58 x 10E9 clock cycles / sec = 1,580,000 clock cycles per servo period.

I watched servo-thread.tmax using Halshow under the Watch tab for about three hours. It started at 459,116 clock cycles, and crept up to 1,396,994 clock cycles in about an hour.

I would assume that these are "close", and if I let this run for many hours I would expect them to get even closer.

I've been trying different things to decrease the latency for about three months, so I don't think that I'll have much luck there. So, given that I'm reading spindle RPM (10 pulses / rev, 2,500 RPM max, discussed by us in another thread), what would you recommend increasing SERVO_PERIOD to? Or, said another way, what bad things might happen as the period gets longer?

Please Log in or Create an account to join the conversation.

More
15 Jul 2016 19:25 #77521 by PCW
Replied by PCW on topic Random read errors on Mesa 7i92
I would set the period to say 1.2 ms and see if the real time errors go away
This is just on the edge of not working with the software encoder at 10 pulse/rev and 2500 RPM
so a custom config with just A and index would be better

Another option is this:
use the eth_pkt_loss branch and run the thread at say 2 KHz and live with the
occasional timed out packets (they should cause no harm)

Note that if you raise the thread rate, the average response time gets better (due to caching)

Please Log in or Create an account to join the conversation.

More
15 Jul 2016 20:00 #77524 by jepler
Replied by jepler on topic Random read errors on Mesa 7i92
The code to deal with lost and delayed packets is now queued up to go in the next release, 2.7.6, whenever that happens. So you can also get that functionality by using 2.7-branch packages from the buildbot, or a RIP build of the 2.7 branch.

Please Log in or Create an account to join the conversation.

  • 10K
  • 10K's Avatar Topic Author
  • Offline
  • Premium Member
  • Premium Member
More
15 Jul 2016 22:45 #77534 by 10K
Replied by 10K on topic Random read errors on Mesa 7i92
Here are the results based on your suggestions:

TEST 1: Changed to:

SERVO_PERIOD = 1200000

Got realtime delay error. Checked servo-thread.tmax, and found it to be a little over 2,500,000. That's quite a lot more than what I saw earlier.

(Note: I already have the spindle tach working with A and index only - not sure if that'd change your recommended period.)

TEST 2: Installed new version of LinuxCNC by doing following

pointed to repository

deb buildbot.linuxcnc.org/ wheezy linuxcnc-rtpreempt


then

sudo apt-get update
sudo apt-get install linuxcnc-uspace


A new version of LinuxCNC was installed. Now when I run it, it shows version 2.7.5-18 gaee5dfl

Changed to:

SERVO_PERIOD = 500000


Ran program, and I still get the realtime delay error. I have not yet run an extended test to see if I get the hm2 read error.

Please Log in or Create an account to join the conversation.

More
15 Jul 2016 22:57 #77535 by PCW
Replied by PCW on topic Random read errors on Mesa 7i92
can you check if the parameter

hm2_7i92.0.packet-read-timeout

exists

Please Log in or Create an account to join the conversation.

  • 10K
  • 10K's Avatar Topic Author
  • Offline
  • Premium Member
  • Premium Member
More
15 Jul 2016 23:52 #77536 by 10K
Replied by 10K on topic Random read errors on Mesa 7i92
Yes, I have a hm2_7i92.0.packet-read-timeout.

Value is 80

Please Log in or Create an account to join the conversation.

More
16 Jul 2016 00:42 #77537 by PCW
Replied by PCW on topic Random read errors on Mesa 7i92
can you check
hm2_7i92.0.read.tmax
to see if its the issue

Please Log in or Create an account to join the conversation.

  • 10K
  • 10K's Avatar Topic Author
  • Offline
  • Premium Member
  • Premium Member
More
16 Jul 2016 01:09 #77538 by 10K
Replied by 10K on topic Random read errors on Mesa 7i92
Value of hm2_7i92.0.read.tmax is 2,599,656 after about two hours running.
I watched hm2_7i92.0.read.time for a bit, and it's bouncing around 250-1,000.

Please Log in or Create an account to join the conversation.

Moderators: PCWjmelson
Time to create page: 0.213 seconds
Powered by Kunena Forum