Cannot seem to get good latency numbers!

You can bet your life you are gonna need some time. I'm still learning ten years after I installed linuxcnc for the first time, and, as already said, I was then already used to patch the vanilla sources and build rt-preempt linux kernels. So, take it easy and avoid the samurai's fate.

I'm afraid that latency tuning is still an art, not yet a science. So you'll need to gain some experience. To watch the /proc/interrupts thing seems to be giving the best clues, to me, about what's going on. Do I have many rescheduling interrupts on my isolated core? Since that core does not have so much load, perhaps that shouldn't be happening. TLB shotdowns? They are EVIL. Does cat /proc/cpuinfo say my processor has constant_tsc, yet I'm seeing an HPET local timer interrupt in each core? Time to put hpet=disable on the boot line. And so on and so on and so on.

So, again, take it easy. Get your system working by hook or by crook --this is about machining, after all, it isn't?-- then you'll get as much time as you need to learn more.

I think the aim is for latency tuning to become a science, and that going the RT way, even if it has far worse latency than RTAI, is to make it easier both for linuxcnc developers and users.

But, I fear RT-Preempt still has too much latency. Being able to record a punk band that are friends of you, with a salvaged-from-junk PC and a cheap sound card, without latency issues is one thing. That <5 miliseconds latency can perhaps be good to do trading, too. But if you need to have <50 usec latency or else have a probability to see your machined part ruined, well, things are not so easy. This is why I'm in love with RTAI and this is why is much easier to have good enough latency by using a card that does hardware stepping

Anyway, as said, welcome to the club

Yes, probably it will work quite well if you fix the fan issue. It happens I have been just looking for some radeon card on aliexpress. Seen that card, seems to be PCIe 3 16 x if memory serves me well. I think it should be good enough to see if your integrated card - or discrete nvidia, perhaps intel graphic card are messing your latency numbers.

Not an expert on graphic cards, but I think you should look at your motherboard specs and see what you can get for little money. PCIe 3x cards seem to work on PCIe 2 slots without losing performance noticeably, but I have not already tested it. Of course, you want both the motherboard and the graphic card having PCIe 16x, better than 8x.

Usually, the more VRAM, the better but you shouldn't run amok to get the more VRAM-loaded graphic card.

The wider the graphics car memory bus, the better, so 128 bit is better than 64, 256 is better than 128, and so on, but this is not the only thing having to do with the graphic card performance. What you want is total throughput. I just purchased an HD 6750 because it is PCIe 2 16x x as the destinated motherboard, has 128bit-wide VRAM bus and a GB of RAM. So it seemed to me it was cheap enough and better than the HD 6450 which is the better radeon I have lying around (64-bit) and decided to give it a try. I'll be getting another, PCIe 3 16x card in the future.

Also, you should look at the power output from the PSU, high-end graphic cars eat a lot of power and, should the PSU be unable to cope with that demand, your system would become unstable. If the card is 2 slots high, it could get in the way of another card and that could be annoying in some cases.

I think a good place to know about GPUs is

www.techpowerup.com/gpu-specs/

There you can filter your search by a whole bunch of criteria.

This is the moment to give you a little stabbing by telling that, even if you have no overruns, you can still have "latency excursions" which are visible on terminal window if you launch linuxcnc from one of those windows. They mean, I think, that you graphics systems is unable to do its work --updating the screen-- in 10x the time allocated to do so.

However, don't worry about it. If you are using axis simulator, the time allocates is 1 millisecond. So, if your graphics system needs a hundredth of second, you'll get an "excursion". Now, with the Sherline standard stepper drivers that need a 22 usec signal, base thread is 50 usec, not 25 as "usual". So, when using the sherline mill config instead of axis sim, the time allocated is greater, and latency excursions don't appear unless they took 0,1 secs. So I'm not seeign any while milling, but while using axis sim, y can see them appearing after each press of a button in the GUI.

So, latency excursions are not overruns and having them doesn't mean your system is unable to do the intended work. What matters is the tool moving as expected. You can give a blind eye to "latency excursions" if your tool moves along the path as expected. AFAI, yoll have to get probably thousands of excursions before need to get rid of them arises. There are other here, with more experience, that surely could give to you more info than I could. But, having a good graphics card surely will help with that.

One last thing, while testing, please be methodical. Test just one thing each time. Test it for 5-10 minutes, then write the result in a notebook, then do another test. After a bunch of tests, you can do some long tests, say at least 2 hours. 4-8 hours would be better. I like to have the selected tests running for a complete day.

Please note that the different combinations of kernel parameters can give different resuslts. Usin nohz=off will be different if you are using -or not using- say, parameters that deal with idling. First, it will surelly help if you make sure you have hyperthreading and c-states disabled. then you could go with isolcpus and acpi_irq_nobalance. Then you could go with perhaps nowatchdog and nohz. Then you could go with or without the radeon discrete graphic card. And so on. This means a whole bunch of tests. If you test two thing at the same time, you'll be unable to get the right conclusions, and you'll be wasting a lot of time.

I should note that all this latency fussing does very little but waste time
unless you have a parallel port (or other direct I/O) based system.

For Mesa cards, as long as you can run the servo thread at the desired (usually 1 KHz)
rate without real time errors, improving latency has basically zero effect on performance.

tommylight post=249874 userid=17274@tatel,
Can you please start a new topic on the "computers and os" section and copy the info you wrote about this so we can make it a sticky?
Thank you for the extensive info.

Being new to LinuxCNC and reading all the talk about turning off all these features in the bios but not seeing the features listed in my own bios, the info @tatel posted is a huge help. Now I know how to add things to the the kernel parameters manually. A ton of great info even if it is needed or not.

I agree this should be a sticky somewhere.

tommylight post=249876 userid=17274

If the picture is messed up during boot time while BIOS info is shown, that usually means the vram is gone.

Actually the picture was only messed up once it booted to the desktop. It was an old card that I accidentally kept for some reason.

PCW post=249894 userid=481I should note that all this latency fussing does very little but waste time
unless you have a parallel port (or other direct I/O) based system.

For Mesa cards, as long as you can run the servo thread at the desired (usually 1 KHz)
rate without real time errors, improving latency has basically zero effect on performance.

I am sure you are probably right. But it is good to know the info that @tatel posted. Just knowing there are ways to fine tune even if the bios has very limited options.

@PCW Thanks for helping me get past the latency hurdle I was stuck on. I was about to throw in the towel and find another solution that most likely would have been costly.

tatel post=249858 userid=17633You can bet your life you are gonna need some time. I'm still learning ten years after I installed linuxcnc for the first time, and, as already said, I was then already used to patch the vanilla sources and build rt-preempt linux kernels. So, take it easy and avoid the samurai's fate.

I'm afraid that latency tuning is still an art, not yet a science. So you'll need to gain some experience. To watch the /proc/interrupts thing seems to be giving the best clues, to me, about what's going on. Do I have many rescheduling interrupts on my isolated core? Since that core does not have so much load, perhaps that shouldn't be happening. TLB shotdowns? They are EVIL. Does cat /proc/cpuinfo say my processor has constant_tsc, yet I'm seeing an HPET local timer interrupt in each core? Time to put hpet=disable on the boot line. And so on and so on and so on.

So, again, take it easy. Get your system working by hook or by crook --this is about machining, after all, it isn't?-- then you'll get as much time as you need to learn more.

I think the aim is for latency tuning to become a science, and that going the RT way, even if it has far worse latency than RTAI, is to make it easier both for linuxcnc developers and users.

But, I fear RT-Preempt still has too much latency. Being able to record a punk band that are friends of you, with a salvaged-from-junk PC and a cheap sound card, without latency issues is one thing. That <5 miliseconds latency can perhaps be good to do trading, too. But if you need to have <50 usec latency or else have a probability to see your machined part ruined, well, things are not so easy. This is why I'm in love with RTAI and this is why is much easier to have good enough latency by using a card that does hardware stepping

Anyway, as said, welcome to the club

Yeah to be honest, a lot of what you have said is greek to me at this point. I am sure in time I will begin to understand it all. But it is great info and I am not as lost as I was feeling. I am sure I will be asking more questions as I mess around with LinuxCNC.

Thanks guys

Are you guys sure you want it? LinuxCNC seems to be clearly geared toward RT and hardware stepping now. While I can easily understand the reason why, I'm hoping that software stepping will not be marooned. As a matter of fact, some days ago I asked about RTAI remaining to be alive.

Yet I'm not going to make any arguments about it. I simply got some Wheezy ISOs conveniently stored and I can stick to it for those things I have that need it, like some business stick to old XP systems, and for the same kind of reasons.

There's no doubt PCW has a point. Even so, I still think that latency tuning remains useful, or this thread wouldn't exist. So, please make sure you still want to have that sticky done, then I will be glad to do it.

Best wishes

I have the same processor in a Gigabyte motherboard and my preempt_rt latency scores are way better. I'll run the tests again later this week and report back, but I recall it being the 20uS range for the base thread.

Another thing to keep in mind when trying all the usual BIOS settings along with isolcpus=   is that BIOSes tend to be broken. It is binary only firmware that gets cobbled together in an SDK and you build and pray that it works. Settings often don't do what you expect. ON may be OFF or OFF may not turn anything off at all.

This is all just to see how low a score we can get for base thread of 25uS and not at all necessary for your servo thread and Mesa card.