Can't eek out that last bit of jitter

Also, I considered using the directions at: forum.linuxcnc.org/9-installing-linuxcnc...tall-rtai-got-errors

but when I look at the directory linuxcnc.org/dists/bookworm/2.9-rt/binary-amd64/

I don't see a 2.9.4 version, so I wonder if that is the source of the problems (or maybe just completely unrelated).

Okay, I finally have the rtai version running, but I seem to only be able to run the latency-histogram as sudo.  Is that the way you're supposed to use the rtai version?

Some notes on the install, as the directions leave out specifics of a lot of steps.  (Half the words in 7.2.4 are separate steps without spelled out commands.)  Hopefully this helps some others.  I followed a lot of forum.linuxcnc.org/9-installing-linuxcnc...tall-rtai-got-errors

Also, I'm not sure if these were done in the correct order.

After installing from the live cd and booting into the new os, I ran

sudo apt-get update
sudo apt-get dist-upgrade

I'm not sure if I did this next, or it was somewhere else on the way.

sudo apt-get purge linuxcnc-uspace
sudo apt-get purge linuxcnc-doc*

Next I updated /etc/apt/sources.list.d/linuxcnc-uspace.list adding 2.9_rt to

deb [arch=amd64,arm64 signed-by=/etc/apt/trusted.gpg.d/linuxcnc.gpg.key.binary.gpg] www.linuxcnc.org/ bookworm base 2.9_uspace 2.9-rt
deb-src [arch=amd64,arm64 signed-by=/etc/apt/trusted.gpg.d/linuxcnc.gpg.key.binary.gpg] www.linuxcnc.org/ bookworm base 2.9_uspace 2.9-rt

(Note I think this is what the linuxcnc-install.sh script called out in the directions is supposed to do, but instead it adds a linuxcnc.list to the sources.list.d, which causes a conflict with what is in linuxcnc-uspace.list and means that apt and the Synaptic Manager stop working.)

Next I did another:

sudo apt-get update

Next I installed the real time kernel:

sudo apt-get linux-image-5.4.258-rtai-amd64

Note: You have to put in the version number. You can tab to find the right one to finish the line. It should have rtai.

REBOOT!!!! Here you have to reboot into the rtai kernel (I think). When the blue screen comes up press down to the second option to choose the rtai kernel.

Next I installed the modules and linuxcnc:

sudo apt-get install rtai-modules
sudo apt-get install linuxcnc

Here I don't think I needed the version number, but if you get an error you can try that.

I also installed headers at some point. I don't know if that was necessary:

sudo apt-get install linux-headers-5.4.258-rtai-amd64

I think I ran the linuxcnc install again after, but I don't know if it was necessary.

sudo apt-get install linuxcnc

The reward for all this... It looks like my min/max base thread latency is about the same as the preempt_rt version.
 
Any other ideas what to try?

Actually I found a mistake in my grub command line.  Now my histogram is better, in that there are only a few excursions to debug now. 
 

It looks like only interrupt that seems to fire at roughly the same frequency is IRQ 20 which is described as

IO-APIC 20-fasteoi pata_pdc2027x[0000:03.05.0]

It is interesting that it fires on CPU1.  I thought I had a command in my grub setup to disable hyperthreading.
 

Thank you Tommy,
You were right that that was the PATA setting in the bios.  I changed it to AHCI, and now that interrupt is no longer on the list anymore. 
 
Unfortunately, it doesn't appear that was the cause of my jitter.

Now the only things causing interrupts are
18 - Radeon HD 4250 internal graphics
LOC - (Local timer interrupts)
RES (Rescheduling Interrupts)

Is there anyway to see what causes the RES and LOC timer interrupts to see if the jitter is caused by one of those?

Sometimes 19 also happens, but it doesn't seem to sync up with times that the jitter has variance.

For some reason the nosmt=force setting in the kernal boot parameters doesn't seem to work anymore with the rtai kernel.  Should nosmt still work in the rtai kernel?

I have the following settings in:
ro quiet skew_tick=1 nosmt=force mitigations=auto,nosmt irqaffinity=0-1 kthread_cpus=0-1 rcu_nocb_poll rcu_nocbs=2-7 nohz=on nohz_full=2-7 isolcpus=2-7 intel_pstate=disable idle=poll cpufreq.off=1 intel_idle.max_cstate=0 amd_idle.max_cstate=0 processor.max_cstate=0 nowatchdog nosoftlockup acpi_irq_nobalance noirqbalance

I'm also using the irq affinity script from:  forum.linuxcnc.org/38-general-linuxcnc-q...ead-and-irq-affinity

Should I try a pci graphics card instead of the Radeon HD4250 integrated graphics?  Any recommendations?

I got an R5 340X off of ebay, and I tried it out.  It didn't seem to make any difference in the RTAI or PREEMPT_RT kernels.
 

I feel like I'm missing some option.

I can't figure out why nosmt isn't working for the rtai kernel.

The only interrupts that are firing are RES, LOC and 24 which is the new graphics card.

Last time I had a bit of a succes on preempt_rt with moving the base thread to another cpu (only with 6.13), but perhaps worth a shot.
Change the CPU numbers and run as sudo. In the script CPU1 is the base thread and CPU2 the servo thread.

forum.linuxcnc.org/18-computer/48079-can...cnc?start=390#321927

Thanks royka,
Do you happen to know, in the rtai kernel, would I move the processes with priority listed as rt?

Because I couldn't figure out how to disable smt for my fx 8350 cpu with the rtai kernel.  I tried switching to an X6 1090T.  I also disabled the thermal limits in bios.  My max jitter reduces to +/-14.4us (with all points within +/-5us except the ones right at +/-14.4us.  I wish I had taken a picture.)  I then turned the thermal limits back on, and I got about the same result I was getting for the fx 8350.  So, I turned the thermal limits back of again, but it didn't go back to the +/-14.4us max jitter.  It still looked the same as the fx 8350.  I tried resetting and messing with some other settings, and I can't get back to that +/-14.4us or better.  With some systems, is there some random timing at startup that causes different latency results even though there were no settings changes?