Systematic approach to tracking down latency issue
06 Aug 2015 20:10 #61187
by hase
Systematic approach to tracking down latency issue was created by hase
Hi all,
I am fed up.
Actually, I am *very* happy with LinuxCNC and essentially love it.
And that is why I am so thankful for all the work that so many developers are putting in. Big Hug!
But latency issues keep driving me mad.
Interrupt: I am a firmware and hardware engineer by profession and I'm using LinuxCNC-driven homegrown milling machines as a hobby: sometimes hacking on bits and bytes isn't cutting it for me; then I need the sound of metal being chipped or the smell and sound of wood made into sawdust.
Anyway.
One of my PCs driving my experimental mill needed an upgrade: I put in an SSD, installed Kubuntu 12.04, installed the rtai kernel and linuxcnc and restored my configurations from a backup.
A failed hard disk made this operation necessary and I used the chance to upgrade to SSD.
This operation beamed me right back to latency hell.
On a machine that I had working just fine with actually good latency values (measured in latency-test).
I would love to do a systematic search for the problem, but I can find no documentation on this.
There are a number of tips-n-tricks type docs like wiki.linuxcnc.org/cgi-bin/wiki.pl?RealTime but is there a systematic approach to find out what caused latency?
When it comes to task-invocation-latency, I am basically in the dark (which feels bad for an engineer
From the RTAI documentation I would assume that once the rtai module is mounted, it takes priority over interupts and task scheduling.
I therefore understand how power-saving features could ruin latency: if the CPU has to wake up from deep sleep before servicing the timer interrupt, that will cause latency. This is why the Core2Duo in one of my machine controllers does benefit from the idle=poll parameter on the command line so much (without idle=poll => unusable, with idle=poll => good latency values mesured).
But why would swapping the hard disk for an SSD (and hooking up a secondary hard disk) impact latency?
tl;dr:
- is there a way to figure out what caused a scheduling delay?
- would it help to fiddle with SMP interrupt affinity? Is that even possible under RTAI?
merci bien
Hartmut "hase" Semken
I am fed up.
Actually, I am *very* happy with LinuxCNC and essentially love it.
And that is why I am so thankful for all the work that so many developers are putting in. Big Hug!
But latency issues keep driving me mad.
Interrupt: I am a firmware and hardware engineer by profession and I'm using LinuxCNC-driven homegrown milling machines as a hobby: sometimes hacking on bits and bytes isn't cutting it for me; then I need the sound of metal being chipped or the smell and sound of wood made into sawdust.
Anyway.
One of my PCs driving my experimental mill needed an upgrade: I put in an SSD, installed Kubuntu 12.04, installed the rtai kernel and linuxcnc and restored my configurations from a backup.
A failed hard disk made this operation necessary and I used the chance to upgrade to SSD.
This operation beamed me right back to latency hell.
On a machine that I had working just fine with actually good latency values (measured in latency-test).
I would love to do a systematic search for the problem, but I can find no documentation on this.
There are a number of tips-n-tricks type docs like wiki.linuxcnc.org/cgi-bin/wiki.pl?RealTime but is there a systematic approach to find out what caused latency?
When it comes to task-invocation-latency, I am basically in the dark (which feels bad for an engineer
From the RTAI documentation I would assume that once the rtai module is mounted, it takes priority over interupts and task scheduling.
I therefore understand how power-saving features could ruin latency: if the CPU has to wake up from deep sleep before servicing the timer interrupt, that will cause latency. This is why the Core2Duo in one of my machine controllers does benefit from the idle=poll parameter on the command line so much (without idle=poll => unusable, with idle=poll => good latency values mesured).
But why would swapping the hard disk for an SSD (and hooking up a secondary hard disk) impact latency?
tl;dr:
- is there a way to figure out what caused a scheduling delay?
- would it help to fiddle with SMP interrupt affinity? Is that even possible under RTAI?
merci bien
Hartmut "hase" Semken
Please Log in or Create an account to join the conversation.
06 Mar 2019 00:02 #127840
by hase
Replied by hase on topic Systematic approach to tracking down latency issue
I never really found a systematic way.
I gave up when I found all the measurement programs to show not only different but contradictionary measurements.
Also, my mesurements using external hardware I hacked together - like comparing a software-generated pin-toggle in every cycle of the base thread to a frequency generator - did not help.
I was also unable to rebuild even an official kernel - that is: one shipped with LinuxCNC.
I can confirm, that all Nvidia graphics I tried are making the PC useless for LinuxCNC.
Stay away from anything Nvidia.
The Radeons work well for me.
One reason I was able to identify through elimination: when I swapped an older hard drive for one supporting SMART, a perfectly fine working controller became useless.
Turning off SMART support in the BIOS Setup fixed it.
Obviously, handling SMART diagnosis has a higher priority than the timer interrupt - which it should not have.
As a result of all my frustration I just switched to Mesa FPGA cards for step generation.
Their documentation was a bit confusing at first, but eventually I figured it out.
I have one controller with a PCI FPGA card and the Mesa driver/breakout card, another one with just the (cheap!) FPGA card any my own little driver/breakout board.
In both controllers the base thread is now empty except for a pin toggle on a parallel port. I feed this 50%-duty PWM to an external watchdog (a window comparator checking the analog voltage to be between 40% and 60% logic level of the parport tripping an RS-Flipflop; so if the base thread stops completely, the watchdog turns off the power to the drives and spindle).
A bit of jitter in this function is no problem (covered by the rather large window) and the FPGA make silky-smooth steps
Any btw: I stocked up on Core-II-Duo based Dell Optiplexes and Fujitsus when they were still available.
They have enough CPU horsepower for LinuxCNC with Axis and perform much better latency-wise than anything more modern I tried.
The more nifty stuff the chipset has built-in, the more hickups the BIOS seems to be able to cause.
But again: sorry, I have no systematic, just anecdotal evidence.
I gave up when I found all the measurement programs to show not only different but contradictionary measurements.
Also, my mesurements using external hardware I hacked together - like comparing a software-generated pin-toggle in every cycle of the base thread to a frequency generator - did not help.
I was also unable to rebuild even an official kernel - that is: one shipped with LinuxCNC.
I can confirm, that all Nvidia graphics I tried are making the PC useless for LinuxCNC.
Stay away from anything Nvidia.
The Radeons work well for me.
One reason I was able to identify through elimination: when I swapped an older hard drive for one supporting SMART, a perfectly fine working controller became useless.
Turning off SMART support in the BIOS Setup fixed it.
Obviously, handling SMART diagnosis has a higher priority than the timer interrupt - which it should not have.
As a result of all my frustration I just switched to Mesa FPGA cards for step generation.
Their documentation was a bit confusing at first, but eventually I figured it out.
I have one controller with a PCI FPGA card and the Mesa driver/breakout card, another one with just the (cheap!) FPGA card any my own little driver/breakout board.
In both controllers the base thread is now empty except for a pin toggle on a parallel port. I feed this 50%-duty PWM to an external watchdog (a window comparator checking the analog voltage to be between 40% and 60% logic level of the parport tripping an RS-Flipflop; so if the base thread stops completely, the watchdog turns off the power to the drives and spindle).
A bit of jitter in this function is no problem (covered by the rather large window) and the FPGA make silky-smooth steps
Any btw: I stocked up on Core-II-Duo based Dell Optiplexes and Fujitsus when they were still available.
They have enough CPU horsepower for LinuxCNC with Axis and perform much better latency-wise than anything more modern I tried.
The more nifty stuff the chipset has built-in, the more hickups the BIOS seems to be able to cause.
But again: sorry, I have no systematic, just anecdotal evidence.
Please Log in or Create an account to join the conversation.
Time to create page: 0.114 seconds