Appreciate some Thoughts on latency results
- bally123
- Topic Author
- Offline
- Senior Member
Less
More
- Posts: 48
- Thank you received: 5
28 May 2019 09:11 #135140
by bally123
Appreciate some Thoughts on latency results was created by bally123
Building a second pc using stretch preempt dvd
Core duo with 2gb and SSD
Lenovo think centre
Mesa 5i25 and 7i76
I tested last night following a graphical install.
Servo thread was ok at around 25000 but spiked after a minute and leapt to 300000.
I tried lots of things, disabling as much as possible in bios .
I then added various things in my grub config...
Eventually ended up with this....
GRUB_CMDLINE_LINUX_DEFAULT="-original parameters- isolcpus=1 intel_idle.max_cstate=0 processor.max_cstate=0 acpi_irq_nobalance noirqbalance idle=poll"
I also removed a wfi card which helped a bit...
But still seems too high...
Histogram attached
Appreciate thoughts,I was wondering about SMI settings, is this something I should try?
Anything else? Would it make sense to try different version or distribution?
Core duo with 2gb and SSD
Lenovo think centre
Mesa 5i25 and 7i76
I tested last night following a graphical install.
Servo thread was ok at around 25000 but spiked after a minute and leapt to 300000.
I tried lots of things, disabling as much as possible in bios .
I then added various things in my grub config...
Eventually ended up with this....
GRUB_CMDLINE_LINUX_DEFAULT="-original parameters- isolcpus=1 intel_idle.max_cstate=0 processor.max_cstate=0 acpi_irq_nobalance noirqbalance idle=poll"
I also removed a wfi card which helped a bit...
But still seems too high...
Histogram attached
Appreciate thoughts,I was wondering about SMI settings, is this something I should try?
Anything else? Would it make sense to try different version or distribution?
Attachments:
Please Log in or Create an account to join the conversation.
- pl7i92
- Offline
- Platinum Member
Less
More
- Posts: 1890
- Thank you received: 356
28 May 2019 10:33 #135146
by pl7i92
Replied by pl7i92 on topic Appreciate some Thoughts on latency results
this looks like the SMI in Bios
you may look into
wiki.linuxcnc.org/cgi-bin/wiki.pl?FixingSMIIssues
the 5i25 does not act on software stepping so it is no problem to you with this PC
it will work great
but you may take a look to get it below servo tread
you may look into
wiki.linuxcnc.org/cgi-bin/wiki.pl?FixingSMIIssues
the 5i25 does not act on software stepping so it is no problem to you with this PC
it will work great
but you may take a look to get it below servo tread
The following user(s) said Thank You: bally123
Please Log in or Create an account to join the conversation.
- rodw
- Offline
- Platinum Member
Less
More
- Posts: 10737
- Thank you received: 3540
28 May 2019 10:40 - 28 May 2019 10:41 #135149
by rodw
Replied by rodw on topic Appreciate some Thoughts on latency results
There is a command line switch to exclude the latency test for the base thread which yields better results. But becasue mesa does not use a base thread, even latency up to about 200,000 is usable with mesa. Mine is a lot worse than yours and I have not had any problems.
Last edit: 28 May 2019 10:41 by rodw.
Please Log in or Create an account to join the conversation.
- BigJohnT
- Offline
- Administrator
Less
More
- Posts: 7330
- Thank you received: 1177
28 May 2019 11:38 - 28 May 2019 11:40 #135160
by BigJohnT
Replied by BigJohnT on topic Appreciate some Thoughts on latency results
As mentioned the base thread latency test is not needed for a Mesa card and can sometimes skew the results.
To see the command line options for the latency tests:
JT
To see the command line options for the latency tests:
latency-test --help
latency-histogram --help
latency-plot --help
JT
Last edit: 28 May 2019 11:40 by BigJohnT.
Please Log in or Create an account to join the conversation.
- bally123
- Topic Author
- Offline
- Senior Member
Less
More
- Posts: 48
- Thank you received: 5
28 May 2019 12:29 - 28 May 2019 12:30 #135161
by bally123
Replied by bally123 on topic Appreciate some Thoughts on latency results
Thanks everyone...
I appreciate that the base thread is not a concern as I'm not software stepping
I'll check out the other options to eliminate base thread results
So with servo thread being more relevant it's great to hear rodw thinks it will run good with my results.
Out if curiosity is there a min requirement for servo thread latency when combined with Mesa cards?
I am left wondering is there anypoint looking down the SMI route or shall I just run it?
Thanks...
I appreciate that the base thread is not a concern as I'm not software stepping
I'll check out the other options to eliminate base thread results
So with servo thread being more relevant it's great to hear rodw thinks it will run good with my results.
Out if curiosity is there a min requirement for servo thread latency when combined with Mesa cards?
I am left wondering is there anypoint looking down the SMI route or shall I just run it?
Thanks...
Last edit: 28 May 2019 12:30 by bally123.
Please Log in or Create an account to join the conversation.
- Todd Zuercher
- Offline
- Platinum Member
Less
More
- Posts: 5009
- Thank you received: 1443
28 May 2019 12:58 #135163
by Todd Zuercher
Replied by Todd Zuercher on topic Appreciate some Thoughts on latency results
That is a little subjective, but I'd say it would be your servo-thread rate minus some safety margin (maybe 15%). So for a normal 1kHz servo thread something less than about 85us.
Please Log in or Create an account to join the conversation.
- bally123
- Topic Author
- Offline
- Senior Member
Less
More
- Posts: 48
- Thank you received: 5
28 May 2019 14:31 #135170
by bally123
Replied by bally123 on topic Appreciate some Thoughts on latency results
Err....do you mean 850us??
1/ 1000hz X 0.85=850us
Am I missing something?
I'd guess you want to be a good factor faster for safety, maybe 5x ....
1/ 1000hz X 0.85=850us
Am I missing something?
I'd guess you want to be a good factor faster for safety, maybe 5x ....
Please Log in or Create an account to join the conversation.
- PCW
- Away
- Moderator
Less
More
- Posts: 17899
- Thank you received: 4774
28 May 2019 14:35 - 28 May 2019 14:36 #135171
by PCW
Replied by PCW on topic Appreciate some Thoughts on latency results
I think you dropped a decimal point, a 15% margin would be 150 us, so 850 us max jitter at a 1 Khz servo thread.
I would not want more than 50% of the servo thread time used by components+jitter
but with a hardware stepgen the OPs ~200 usec jitter should be harmless +- a bit of noise
added to the stepgens position control loop.
That is, the most noticeable effect of really bad jitter on the hardware stepgen is
velocity modulation at high speeds caused by the PID attempting to correct for bogus
position errors due to position sample time jitter.
Normally PCI firmware does not include the DPLL module, but if the host has really
bad jitter, firmware with the DPLL can be used. This reduces the postion sampling
jitter to less than 1 us
I would not want more than 50% of the servo thread time used by components+jitter
but with a hardware stepgen the OPs ~200 usec jitter should be harmless +- a bit of noise
added to the stepgens position control loop.
That is, the most noticeable effect of really bad jitter on the hardware stepgen is
velocity modulation at high speeds caused by the PID attempting to correct for bogus
position errors due to position sample time jitter.
Normally PCI firmware does not include the DPLL module, but if the host has really
bad jitter, firmware with the DPLL can be used. This reduces the postion sampling
jitter to less than 1 us
Last edit: 28 May 2019 14:36 by PCW. Reason: format a bit
The following user(s) said Thank You: bally123
Please Log in or Create an account to join the conversation.
- bally123
- Topic Author
- Offline
- Senior Member
Less
More
- Posts: 48
- Thank you received: 5
28 May 2019 20:30 #135190
by bally123
Replied by bally123 on topic Appreciate some Thoughts on latency results
Update:
I ran the machine tonight and a message appeared in axis -
Unexpected real time delay on task 0.....
So I ran some more latency checks, I didn't see much above 150000 as jitter on servo thread, and this appeared to be occasional spikes
I then ran a few latency plots and this is about as bad an example as I have managed to find...:
Due to the timings of the spikes I feel I should try the SMICTRL fix mentioned here .
But TBH I'm not entirely sure how to apply it, I downloaded the smictrl.c file, is there a makefile I need to use or as suggested in the comments can I enter this in a terminal:
sudo apt-get install libpci-dev
KSRC=/lib/modules/`uname -r`/source
cc -O2 -Wall -I $KSRC/include -lz -lpci smictrl.c -o smictrl
When I use this tool I guess I need to experiment with the options it provides until I see latency improve?
Is there an obvious option to try or avoid?
Thanks....
I ran the machine tonight and a message appeared in axis -
Unexpected real time delay on task 0.....
So I ran some more latency checks, I didn't see much above 150000 as jitter on servo thread, and this appeared to be occasional spikes
I then ran a few latency plots and this is about as bad an example as I have managed to find...:
Due to the timings of the spikes I feel I should try the SMICTRL fix mentioned here .
But TBH I'm not entirely sure how to apply it, I downloaded the smictrl.c file, is there a makefile I need to use or as suggested in the comments can I enter this in a terminal:
sudo apt-get install libpci-dev
KSRC=/lib/modules/`uname -r`/source
cc -O2 -Wall -I $KSRC/include -lz -lpci smictrl.c -o smictrl
When I use this tool I guess I need to experiment with the options it provides until I see latency improve?
Is there an obvious option to try or avoid?
Thanks....
Please Log in or Create an account to join the conversation.
- bally123
- Topic Author
- Offline
- Senior Member
Less
More
- Posts: 48
- Thank you received: 5
28 May 2019 21:04 - 28 May 2019 21:45 #135194
by bally123
Replied by bally123 on topic Appreciate some Thoughts on latency results
Ok I built the file using the code in the comments, figured out ./smictl -h executes it with instructions
./smictrl -v. Helped me see what was set
I then used ./smictl -c 1
To clear the GLB_SMI _EN_BIT
I figured this was the global smi enable
Again ./smictrl -v to confirm change
Results:
I started the plot before making the change and I got initial spikes then a large spike at 250s, shortly after I disabled the Global enable bit....from then on no more spikes.
I'll continue to monitor the performance over next few days....
./smictrl -v. Helped me see what was set
I then used ./smictl -c 1
To clear the GLB_SMI _EN_BIT
I figured this was the global smi enable
Again ./smictrl -v to confirm change
Results:
I started the plot before making the change and I got initial spikes then a large spike at 250s, shortly after I disabled the Global enable bit....from then on no more spikes.
I'll continue to monitor the performance over next few days....
Attachments:
Last edit: 28 May 2019 21:45 by bally123.
Please Log in or Create an account to join the conversation.
Time to create page: 0.373 seconds