Can't eek out that last bit of jitter

Also, I considered using the directions at: forum.linuxcnc.org/9-installing-linuxcnc...tall-rtai-got-errors

but when I look at the directory linuxcnc.org/dists/bookworm/2.9-rt/binary-amd64/

I don't see a 2.9.4 version, so I wonder if that is the source of the problems (or maybe just completely unrelated).

Okay, I finally have the rtai version running, but I seem to only be able to run the latency-histogram as sudo.  Is that the way you're supposed to use the rtai version?

Some notes on the install, as the directions leave out specifics of a lot of steps.  (Half the words in 7.2.4 are separate steps without spelled out commands.)  Hopefully this helps some others.  I followed a lot of forum.linuxcnc.org/9-installing-linuxcnc...tall-rtai-got-errors

Also, I'm not sure if these were done in the correct order.

After installing from the live cd and booting into the new os, I ran

sudo apt-get update
sudo apt-get dist-upgrade

I'm not sure if I did this next, or it was somewhere else on the way.

sudo apt-get purge linuxcnc-uspace
sudo apt-get purge linuxcnc-doc*

Next I updated /etc/apt/sources.list.d/linuxcnc-uspace.list adding 2.9_rt to

deb [arch=amd64,arm64 signed-by=/etc/apt/trusted.gpg.d/linuxcnc.gpg.key.binary.gpg] www.linuxcnc.org/ bookworm base 2.9_uspace 2.9-rt
deb-src [arch=amd64,arm64 signed-by=/etc/apt/trusted.gpg.d/linuxcnc.gpg.key.binary.gpg] www.linuxcnc.org/ bookworm base 2.9_uspace 2.9-rt

(Note I think this is what the linuxcnc-install.sh script called out in the directions is supposed to do, but instead it adds a linuxcnc.list to the sources.list.d, which causes a conflict with what is in linuxcnc-uspace.list and means that apt and the Synaptic Manager stop working.)

Next I did another:

sudo apt-get update

Next I installed the real time kernel:

sudo apt-get linux-image-5.4.258-rtai-amd64

Note: You have to put in the version number. You can tab to find the right one to finish the line. It should have rtai.

REBOOT!!!! Here you have to reboot into the rtai kernel (I think). When the blue screen comes up press down to the second option to choose the rtai kernel.

Next I installed the modules and linuxcnc:

sudo apt-get install rtai-modules
sudo apt-get install linuxcnc

Here I don't think I needed the version number, but if you get an error you can try that.

I also installed headers at some point. I don't know if that was necessary:

sudo apt-get install linux-headers-5.4.258-rtai-amd64

I think I ran the linuxcnc install again after, but I don't know if it was necessary.

sudo apt-get install linuxcnc

The reward for all this... It looks like my min/max base thread latency is about the same as the preempt_rt version.
 
Any other ideas what to try?

Actually I found a mistake in my grub command line.  Now my histogram is better, in that there are only a few excursions to debug now. 
 

It looks like only interrupt that seems to fire at roughly the same frequency is IRQ 20 which is described as

IO-APIC 20-fasteoi pata_pdc2027x[0000:03.05.0]

It is interesting that it fires on CPU1.  I thought I had a command in my grub setup to disable hyperthreading.
 

Thank you Tommy,
You were right that that was the PATA setting in the bios.  I changed it to AHCI, and now that interrupt is no longer on the list anymore. 
 
Unfortunately, it doesn't appear that was the cause of my jitter.

Now the only things causing interrupts are
18 - Radeon HD 4250 internal graphics
LOC - (Local timer interrupts)
RES (Rescheduling Interrupts)

Is there anyway to see what causes the RES and LOC timer interrupts to see if the jitter is caused by one of those?

Sometimes 19 also happens, but it doesn't seem to sync up with times that the jitter has variance.

For some reason the nosmt=force setting in the kernal boot parameters doesn't seem to work anymore with the rtai kernel.  Should nosmt still work in the rtai kernel?

I have the following settings in:
ro quiet skew_tick=1 nosmt=force mitigations=auto,nosmt irqaffinity=0-1 kthread_cpus=0-1 rcu_nocb_poll rcu_nocbs=2-7 nohz=on nohz_full=2-7 isolcpus=2-7 intel_pstate=disable idle=poll cpufreq.off=1 intel_idle.max_cstate=0 amd_idle.max_cstate=0 processor.max_cstate=0 nowatchdog nosoftlockup acpi_irq_nobalance noirqbalance

I'm also using the irq affinity script from:  forum.linuxcnc.org/38-general-linuxcnc-q...ead-and-irq-affinity

Should I try a pci graphics card instead of the Radeon HD4250 integrated graphics?  Any recommendations?