Appreciate some Thoughts on latency results

  • bally123
  • bally123's Avatar Topic Author
  • Offline
  • Senior Member
  • Senior Member
More
28 May 2019 09:11 #135140 by bally123
Building a second pc using stretch preempt dvd

Core duo with 2gb and SSD
Lenovo think centre

Mesa 5i25 and 7i76

I tested last night following a graphical install.

Servo thread was ok at around 25000 but spiked after a minute and leapt to 300000.

I tried lots of things, disabling as much as possible in bios .

I then added various things in my grub config...

Eventually ended up with this....

GRUB_CMDLINE_LINUX_DEFAULT="-original parameters- isolcpus=1 intel_idle.max_cstate=0 processor.max_cstate=0 acpi_irq_nobalance noirqbalance idle=poll"

I also removed a wfi card which helped a bit...

But still seems too high...

Histogram attached

Appreciate thoughts,I was wondering about SMI settings, is this something I should try?

Anything else? Would it make sense to try different version or distribution?
Attachments:

Please Log in or Create an account to join the conversation.

More
28 May 2019 10:33 #135146 by pl7i92
Replied by pl7i92 on topic Appreciate some Thoughts on latency results
this looks like the SMI in Bios
you may look into
wiki.linuxcnc.org/cgi-bin/wiki.pl?FixingSMIIssues


the 5i25 does not act on software stepping so it is no problem to you with this PC
it will work great

but you may take a look to get it below servo tread
The following user(s) said Thank You: bally123

Please Log in or Create an account to join the conversation.

More
28 May 2019 10:40 - 28 May 2019 10:41 #135149 by rodw
There is a command line switch to exclude the latency test for the base thread which yields better results. But becasue mesa does not use a base thread, even latency up to about 200,000 is usable with mesa. Mine is a lot worse than yours and I have not had any problems.
Last edit: 28 May 2019 10:41 by rodw.

Please Log in or Create an account to join the conversation.

More
28 May 2019 11:38 - 28 May 2019 11:40 #135160 by BigJohnT
Replied by BigJohnT on topic Appreciate some Thoughts on latency results
As mentioned the base thread latency test is not needed for a Mesa card and can sometimes skew the results.

To see the command line options for the latency tests:
latency-test --help
latency-histogram --help
latency-plot --help

JT
Last edit: 28 May 2019 11:40 by BigJohnT.

Please Log in or Create an account to join the conversation.

  • bally123
  • bally123's Avatar Topic Author
  • Offline
  • Senior Member
  • Senior Member
More
28 May 2019 12:29 - 28 May 2019 12:30 #135161 by bally123
Replied by bally123 on topic Appreciate some Thoughts on latency results
Thanks everyone...

I appreciate that the base thread is not a concern as I'm not software stepping

I'll check out the other options to eliminate base thread results

So with servo thread being more relevant it's great to hear rodw thinks it will run good with my results.

Out if curiosity is there a min requirement for servo thread latency when combined with Mesa cards?

I am left wondering is there anypoint looking down the SMI route or shall I just run it?

Thanks...
Last edit: 28 May 2019 12:30 by bally123.

Please Log in or Create an account to join the conversation.

More
28 May 2019 12:58 #135163 by Todd Zuercher
Replied by Todd Zuercher on topic Appreciate some Thoughts on latency results
That is a little subjective, but I'd say it would be your servo-thread rate minus some safety margin (maybe 15%). So for a normal 1kHz servo thread something less than about 85us.

Please Log in or Create an account to join the conversation.

  • bally123
  • bally123's Avatar Topic Author
  • Offline
  • Senior Member
  • Senior Member
More
28 May 2019 14:31 #135170 by bally123
Replied by bally123 on topic Appreciate some Thoughts on latency results
Err....do you mean 850us??

1/ 1000hz X 0.85=850us

Am I missing something?

I'd guess you want to be a good factor faster for safety, maybe 5x ....

Please Log in or Create an account to join the conversation.

More
28 May 2019 14:35 - 28 May 2019 14:36 #135171 by PCW
I think you dropped a decimal point, a 15% margin would be 150 us, so 850 us max jitter at a 1 Khz servo thread.

I would not want more than 50% of the servo thread time used by components+jitter
but with a hardware stepgen the OPs ~200 usec jitter should be harmless +- a bit of noise
added to the stepgens position control loop.

That is, the most noticeable effect of really bad jitter on the hardware stepgen is
velocity modulation at high speeds caused by the PID attempting to correct for bogus
position errors due to position sample time jitter.

Normally PCI firmware does not include the DPLL module, but if the host has really
bad jitter, firmware with the DPLL can be used. This reduces the postion sampling
jitter to less than 1 us
Last edit: 28 May 2019 14:36 by PCW. Reason: format a bit
The following user(s) said Thank You: bally123

Please Log in or Create an account to join the conversation.

  • bally123
  • bally123's Avatar Topic Author
  • Offline
  • Senior Member
  • Senior Member
More
28 May 2019 20:30 #135190 by bally123
Replied by bally123 on topic Appreciate some Thoughts on latency results
Update:

I ran the machine tonight and a message appeared in axis -

Unexpected real time delay on task 0.....

So I ran some more latency checks, I didn't see much above 150000 as jitter on servo thread, and this appeared to be occasional spikes

I then ran a few latency plots and this is about as bad an example as I have managed to find...:



Due to the timings of the spikes I feel I should try the SMICTRL fix mentioned here .

But TBH I'm not entirely sure how to apply it, I downloaded the smictrl.c file, is there a makefile I need to use or as suggested in the comments can I enter this in a terminal:

sudo apt-get install libpci-dev
KSRC=/lib/modules/`uname -r`/source
cc -O2 -Wall -I $KSRC/include -lz -lpci smictrl.c -o smictrl

When I use this tool I guess I need to experiment with the options it provides until I see latency improve?

Is there an obvious option to try or avoid?

Thanks....
Attachments:

Please Log in or Create an account to join the conversation.

  • bally123
  • bally123's Avatar Topic Author
  • Offline
  • Senior Member
  • Senior Member
More
28 May 2019 21:04 - 28 May 2019 21:45 #135194 by bally123
Replied by bally123 on topic Appreciate some Thoughts on latency results
Ok I built the file using the code in the comments, figured out ./smictl -h executes it with instructions

./smictrl -v. Helped me see what was set

I then used ./smictl -c 1

To clear the GLB_SMI _EN_BIT

I figured this was the global smi enable

Again ./smictrl -v to confirm change

Results:



I started the plot before making the change and I got initial spikes then a large spike at 250s, shortly after I disabled the Global enable bit....from then on no more spikes.

I'll continue to monitor the performance over next few days....
Attachments:
Last edit: 28 May 2019 21:45 by bally123.

Please Log in or Create an account to join the conversation.

Time to create page: 0.373 seconds
Powered by Kunena Forum