Cannot seem to get good latency numbers!

I 100% agree with that

To continue my LinuxCNC education, can someone either explain or point me to an explanation of how to read/understand the histogram and what exactly to look for when using a Mesa board? I see tons of posts where people post a screenshot of their histogram and someone will tell them it either looks good or to try this or that. But I would like to understand what exactly it is saying and what I should be looking for. That is, if there is an easy way to explain it. If it is too technical or complicated then I understand.

Also I understand that latency is not really a factor when using a Mesa board but I have read that the Servo Jitter can/does affect step rate. If this is true, wouldn't it behoove me to try to tune latency in order to get the optimal performance out of my machine? Wouldn't that be a bottleneck in some cases?

Servo jitter is basically irrelevant with Mesa (and probable other) external stepgen
hardware. The maximum steprate is determined by the external hardware, not
anything in LinuxCNC.

As I said before, as long as you don't get real time errors, the servo thread latency
is not important.

Its time to move on with LinuxCNC installation.

What matters is max jitter value. When people is told to try this or that, it's not so much after the information conveyed on latency-histogram screenshot, but after other info conveyed in the question, and the advice given comes from experience.

Fact is, you could be using latency-test instead. Latency-histogram lets you see how many times a jitter value has happened, Under the graphic you see jitter values; to the left, you see the logarythmic scale giving you how many times a jitter value has happened. The more times a jitter value has happened, the taller its line gets.

The number after the "e" gives you the amount of zeroes that have to be added. So 1e1=10 times; 1e6=1,000,000 times. Not so useful, since it doesn't give to you the reason why that jitter value happened. Moreover, what really matters is just your jitter remaining under some limit. There's a reason why latency-test is still the "official" test. But latency-histogram is much, much fancier, no doubt about it.

Actually, since latency-test puts less of a load into the system, you could get slightly better jitter simply by using it instead of latency-histogram.

With a Mesa card like you have, your system's software does not need to generate a shipload of lightning-fast, tightly-timed step pulses to make servos/steppers know what to do each, say, 20 microseconds. That's now the job of the dedicated hardware, and it can do it easily because it has chips designed to do it. Your system software is now a supervisor that needs just to give new instructions to the dedicated hardware, ideally each 1000 microseconds, that is to say, each millisecond. You can easily see that your system's workload is now two orders of magnitude lower.

This is why you have been told that, as long as you can reach a frequency about 1 KHz on your servo thread, all should be fine. It's no coincidence the period of a 1 KHz frequency is one millisecond.

To generate that shipload of step pulses is a whole mountain of work for the software on your PC, and since you got dedicated hardware to do that work, to generate them when they are not needed, is nonsense. This is why you have been told to use --nobase parameter with latency-histogram. This is also the reason why your max jitter got better with --nobase. Simply the system got much less work to do.

Should your jitter reach more than a millisecond, you'll get an overrun (aka a realtime error), Now you can see why a max jitter about 100 usec (this is a 10% of your servo thread period) is fine. 200 is still considered good. Even 300 could be acceptable, after what has been written on this forum. PCW even said that, if there are no overruns, you should be doing it well, and he probably knows better than many. IIRC, you reported a servo max jitter of about 112 usec, so you are near to be golden.

Now, should you be still unhappy with your max jitter, you could either follow the doctrine and get a new PC, to do what others are doing with salvaged-from-junk dual-core systems, or you could think the emperor has no clothes and use the kernel parameters, etc, that have been already explained to you. You'd be surprised to know how many people here, more or less secretly, worships isolcpus=.

Once again tatel you provided a ton of great info. Although I will most likely understand it better as time goes on. But exactly what I was looking for.

I am not going to put a ton of time into messing with my system since the numbers look good enough, but I did purchase an old Radeon R7 250 2G PCIe 3.0 x16 card. My MB has an open PCIe 3.0 x16 slot so I figured it couldn't hurt to compare the numbers. And I may do a little testing at some point with the isolcpus= parameter as well. But as PCW said, it is time to move on with the build.

Thanks for all the detailed info!

I doubt that you'll get much better latency scores with trying an external GPU card. The internal GPU real time issues go back to much earlier chipsets and cpus. Either your BIOS settings aren't yet optimized for real time software stepping that you are not using anyway or the BIOS is broken.

www.amd.com/en/products/cpu/fx-6300
en.wikipedia.org/wiki/AMD_700_chipset_series#760G

I found some time to test the Radeon card I bought. Can you guys tell me which looks better?
This first histogram is the onboard video card:
Onboard Video Card

This is the Radeon video card
Radeon Video Card

I had Firefox and Brave open with multiple tabs. Brave had 4 YT videos running nonstop and I had 5 GLXGEARS running at various sizes during both tests. I will admit that I was also using the computer during the Radeon test where I was not during the other. From what I can tell I think the MB test looks better but I am not sure. I hope the Radeon has good numbers because the computer runs smoother with it. Videos especially run great, even with multiple videos and GLXGEARS running. When using the onboard video card the videos are very choppy and not smooth at all. Unwatchable.

Thanks

The system running the Radeon card has bad latency
=~12X worse than with motherboard video and probably
useless unless it can be fixed by some video driver setting

Yeah, max jitter 113 with onboard card vs 1268 with discrete radeon. So, near golden onboard vs crappy discrete.

I would suggest to you using that machine just for for CNC work unless you are really interested in watching videos while your cutter is at work. Which wouldn't be the smartest thing to do, BTW. That radeon card could be better into another machine dedicated to, say, infotainment.

However I'm really surprised to see a so big difference. Usually I get better, not worse, performance with discrete cards, even if just slightly better. So, if the discrete card is really mightier than the integrated one, it could be that you need to set something either in your BIOS or in the graphics card kernel module. If you are willing to spend some more hours learning, there is good info about setting your radeon module here:
wiki.archlinux.org/title/ATI
wiki.archlinux.org/title/AMDGPU

I'm not really sure AMDGPU is privative or free. It seems to me that amdgpu is a new name for the old radeon free module, but I could be wrong. I have quite a bunch of old and also salvaged PCs but just one modern machine. You'd have to check it. You want to use the free module.

Here's the 4.19 kernel doc for amdgpu module, which lists all parameters available (please make sure you have the doc for the kernel version you are using):
www.kernel.org/doc/html/v4.19/gpu/amdgpu.html

That big bad difference could come from quite a bunch of things, I'm just speculating here:

It could be that BIOS is messing things up in some way, you should make sure the integrated card is disabled, etc, but I think this should be done automatically.

Perhaps that discrete card needs some firmware, you can check this with "sudo dmesg | grep -i firmware"

To see any warnings about that card, you could do "sudo dmesg | grep -i radeon" and "sudo dmesg | grep -i drm"

If you don't see any warnings on dmesg output, it could be something with your xorg configuration or kernel module settings. You shouldn't mess with any xorg.conf file, however. Usually the kernel does that work fine. Shouldn't that be the case, you'd be seeing something on dmesg output.

You could try some kernel boot parameters for radeon driver. Perhaps power management, video acceleration, pcie_gen2/3... dmesg output should give you clues about what's going on. You'd be on for some hours of linux graphics learning there, and it could make you feel really frustrated.

So, having almost fine latency with the integrated graphics card, again I suggest to you using that computer just for CNC work with the integrated graphics

 

Thanks for the info guys. Although I would love to try and get the Radeon card to work, since I did spend money on it, I will probably wait and mess with it later. I just wanted to see if I could improve things a little since the general consensus is that onboard video usually has bad latency and that dedicated Radeon cards usually have good latency. It was worth a shot I guess.

I plan to only use this PC for the CNC for the most part. I have plenty of other PC's in the house so no need for another one. So I guess the video card will probably just go into my used PC parts bin.

I am curious if someone can tell me how to find out what video drivers I am using and how to switch the drivers. Some other OS's have GUI's that allow you to manage this. I cannot find anything like that in Debian.

I ran 
and it came up with this

firmware-amd-graphics/testing,now 20210818-1 all [installed]
  Binary firmware for AMD/ATI graphics chips

libdrm-radeon1/testing,now 2.4.112-3 amd64 [installed,automatic]
  Userspace interface to radeon-specific kernel DRM services -- runtime

librocm-smi-dev/testing 5.2.0-2 amd64
  ROCm System Management Interface (ROCm SMI) library headers

librocm-smi64-1/testing 5.2.0-2 amd64
  ROCm System Management Interface (ROCm SMI) library

radeontool/testing 1.6.3-1+b1 amd64
  utility to control ATI Radeon backlight functions on laptops

radeontop/testing 1.4-1 amd64
  Utility to show Radeon GPU utilization

rovclock/testing 0.6e-7+b1 amd64
  utility to control frequency rates of your Radeon card

xserver-xorg-video-amdgpu/testing,now 22.0.0-2 amd64 [installed,automatic]
  X.Org X server -- AMDGPU display driver

xserver-xorg-video-ati/testing,now 1:19.1.0-3 amd64 [installed,automatic]
  X.Org X server -- AMD/ATI display driver wrapper

xserver-xorg-video-radeon/testing,now 1:19.1.0-3 amd64 [installed,automatic]
  X.Org X server -- AMD/ATI Radeon display driver

But I do not know which driver is being used currently or how to try a different driver.

Oh yeah, I have another question. When I was running a latency test (over night) with the onboard video card, when I checked it in the morning it had crazy numbers (+50M) for each. I have seen this in at least 3 separate tests. It seems like something crashes causing latency to go through the roof. Is this something that I should be concerned with?
Latency Test