Debian 12 Bookworm Linuxcnc 2.9.2....Latency problem

Hi!

I have a working Debian 10 with Linuxcnc 2.9.2. 
I make a fresh install of the latest ISO with Debian 12 and Linuxcnc 2.9.2...
After Installing i checked Linuxcnc and get everytime after some minutes Latancy error..error finishing read.
Then i read to the Forum.... i done the script described on this link:

forum.linuxcnc.org/38-general-linuxcnc-q...ead-and-irq-affinity

with the sudo ./rt_setup.sh
Doesnt help... same problem..
I looked to the bios.. nothing had changed... everything is disabled what can make problems.

then i looked again and found some different kernels.. here:
forum.linuxcnc.org/9-installing-linuxcnc...ation-notes?start=20

i test every kernel... the failure comes quicker or direct when i start linuxcnc..

then i give it a next try.. reinstall..get wificard out,all on usb out,,keyboard and mouse with cable..clean cpu..new coolinggrease...a different ssd harddrive..changing rams... latency read error stays.

i throw in my debian 10 with linuxcnc 2.9.2 ssd.. works perfect.

My pc :

Dell Optiplex 3020 micro with I3 16gb Rams Samsung EVO 840 ssd

i test different mesa 7i92 cards...

then i do the complete process on a different Optiplex 3020 micro.. my Spare machine.. same problem.
it doesnt work when debian 12 is the OS.
Can someone help me? what is the problem? te iso is official..

Thanks

Which kernel do you run at Debian 10? "uname -a"
What is the result of "cat /proc/cmdline" on Debian 10 and 12?
What is the result of "cat /proc/interrupts" ?

This kernel I've compiled last week by the way (6.9.0-rc5):
drive.google.com/drive/folders/1L28lG0zD...2XQJx?usp=drive_link

Hi Thanks for your help! First Debian 10:

uname -v =

Linux debian 4.19.0-26-rt-amd64 #1 SMP PREEMPT RT Debian 4.19.304-1 (2024-01-09) x86_64 GNU/Linux

cat /proc/cmdline =
BOOT_IMAGE=/boot/vmlinuz-4.19.0-26-rt-amd64 root=UUID=c4845308-4dd9-4166-a1ab-c1420c7a7374 ro initrd=/install/gtk/initrd.gz quiet

cat /proc/interrupts=cat /proc/interrupts=
CPU0 CPU1 CPU2 CPU3
0: 7 0 0 0 IO-APIC 2-edge timer
8: 0 1 0 0 IO-APIC 8-edge rtc0
9: 0 4 0 0 IO-APIC 9-fasteoi acpi
16: 0 0 0 25 IO-APIC 16-fasteoi ehci_hcd:usb1
18: 0 0 0 0 IO-APIC 18-fasteoi i801_smbus
23: 0 0 29 0 IO-APIC 23-fasteoi ehci_hcd:usb3
24: 0 0 0 0 PCI-MSI 458752-edge PCIe PME, pciehp
25: 0 0 0 0 PCI-MSI 462848-edge PCIe PME
26: 0 0 0 0 PCI-MSI 464896-edge PCIe PME
27: 64 0 0 0 PCI-MSI 1048576-edge enp2s0
28: 21186 0 0 0 PCI-MSI 512000-edge ahci[0000:00:1f.2]
29: 0 15746 0 0 PCI-MSI 327680-edge xhci_hcd
30: 0 0 0 19421 PCI-MSI 32768-edge i915
31: 14 0 0 0 PCI-MSI 360448-edge mei_me
32: 0 284 0 0 PCI-MSI 49152-edge snd_hda_intel:card0
33: 0 0 450 0 PCI-MSI 442368-edge snd_hda_intel:card1
34: 0 0 0 37625 PCI-MSI 1572864-edge iwlwifi
NMI: 4 4 4 4 Non-maskable interrupts
LOC: 52498 53988 52606 55808 Local timer interrupts
SPU: 0 0 0 0 Spurious interrupts
PMI: 4 4 4 4 Performance monitoring interrupts
IWI: 117 129 41 41 IRQ work interrupts
RTR: 1 0 0 0 APIC ICR read retries
RES: 17103 15345 12175 9979 Rescheduling interrupts
CAL: 20423 21178 18639 18313 Function call interrupts
TLB: 16008 16767 14838 14613 TLB shootdowns
TRM: 0 0 0 0 Thermal event interrupts
THR: 0 0 0 0 Threshold APIC interrupts
DFR: 0 0 0 0 Deferred Error APIC interrupts
MCE: 0 0 0 0 Machine check exceptions
MCP: 2 3 3 3 Machine check polls
ERR: 1
MIS: 0
PIN: 0 0 0 0 Posted-interrupt notification event
NPI: 0 0 0 0 Nested posted-interrupt event
PIW: 0 0 0 0 Posted-interrupt wakeup event

and on Debian 12 with 2.9.2

uname -v =
#1 SMP PREEMPT_RT Debian 6.1.76-1 (2024-02-01)

cat /proc/cmdline =
BOOT_IMAGE=/boot/vmlinuz-6.1.0-18-rt-amd64 root=UUID=1c171f88-4f36-42ba-b84a-1dcf20ae1a6b ro initrd=/install/gtk/initrd.gz quiet

cat /proc/interrupts=cat /proc/interrupts=
<pre> CPU0 CPU1 CPU2 CPU3
0: 8 0 0 0 IO-APIC 2-edge timer
8: 0 0 0 0 IO-APIC 8-edge rtc0
9: 0 4 0 0 IO-APIC 9-fasteoi acpi
16: 0 0 0 25 IO-APIC 16-fasteoi ehci_hcd:usb1
18: 0 0 3 0 IO-APIC 18-fasteoi i801_smbus
23: 0 0 29 0 IO-APIC 23-fasteoi ehci_hcd:usb3
24: 0 0 0 0 PCI-MSI 458752-edge PCIe PME, pciehp
25: 0 0 0 0 PCI-MSI 462848-edge PCIe PME
26: 0 0 0 0 PCI-MSI 464896-edge PCIe PME
27: 18398 0 0 0 PCI-MSI 512000-edge ahci[0000:00:1f.2]
28: 88 0 0 0 PCI-MSI 1048576-edge enp2s0
29: 0 7214 0 0 PCI-MSI 327680-edge xhci_hcd
30: 0 0 0 8202 PCI-MSI 32768-edge i915
31: 18 0 0 0 PCI-MSI 360448-edge mei_me
32: 0 567 0 0 PCI-MSI 49152-edge snd_hda_intel:card0
33: 0 0 641 0 PCI-MSI 442368-edge snd_hda_intel:card1
34: 0 0 0 8354 PCI-MSI 1572864-edge iwlwifi
NMI: 2 2 1 2 Non-maskable interrupts
LOC: 24270 24754 22704 23422 Local timer interrupts
SPU: 0 0 0 0 Spurious interrupts
PMI: 2 2 1 2 Performance monitoring interrupts
IWI: 389 338 276 675 IRQ work interrupts
RTR: 0 0 0 0 APIC ICR read retries
RES: 5753 5763 5661 6035 Rescheduling interrupts
CAL: 15997 18728 20080 14703 Function call interrupts
TLB: 9323 10584 11866 10543 TLB shootdowns
TRM: 0 0 0 0 Thermal event interrupts
THR: 0 0 0 0 Threshold APIC interrupts
DFR: 0 0 0 0 Deferred Error APIC interrupts
MCE: 0 0 0 0 Machine check exceptions
MCP: 1 2 2 2 Machine check polls
ERR: 1
MIS: 0
PIN: 0 0 0 0 Posted-interrupt notification event
NPI: 0 0 0 0 Nested posted-interrupt event
PIW: 0 0 0 0 Posted-interrupt wakeup event
</pre>


thats are both clean installed SSDs the Debian 10 with 2.9.2 works perfect
Debian 12 with 2.9.2 crash


thanks for any help... i will try install your kernel?

after i dist-upgrade :

uname -v=
#1 SMP PREEMPT_RT Debian 6.1.85-1 (2024-04-11)


cat /proc/cmdline=
BOOT_IMAGE=/boot/vmlinuz-6.1.0-20-rt-amd64 root=UUID=1c171f88-4f36-42ba-b84a-1dcf20ae1a6b ro initrd=/install/gtk/initrd.gz quiet

cat /proc/interrupts=
CPU0 CPU1 CPU2 CPU3
0: 8 0 0 0 IO-APIC 2-edge timer
8: 0 0 0 0 IO-APIC 8-edge rtc0
9: 0 4 0 0 IO-APIC 9-fasteoi acpi
16: 0 0 0 27 IO-APIC 16-fasteoi ehci_hcd:usb1
18: 0 0 3 0 IO-APIC 18-fasteoi i801_smbus
23: 0 0 31 0 IO-APIC 23-fasteoi ehci_hcd:usb3
24: 0 0 0 0 PCI-MSI 458752-edge PCIe PME, pciehp
25: 0 0 0 0 PCI-MSI 462848-edge PCIe PME
26: 0 0 0 0 PCI-MSI 464896-edge PCIe PME
27: 50 0 0 0 PCI-MSI 1048576-edge enp2s0
28: 13962 0 0 0 PCI-MSI 512000-edge ahci[0000:00:1f.2]
29: 0 2805 0 0 PCI-MSI 327680-edge xhci_hcd
30: 0 0 0 4484 PCI-MSI 32768-edge i915
31: 18 0 0 0 PCI-MSI 360448-edge mei_me
32: 0 4590 0 0 PCI-MSI 1572864-edge iwlwifi
33: 0 0 322 0 PCI-MSI 49152-edge snd_hda_intel:card0
34: 0 0 0 466 PCI-MSI 442368-edge snd_hda_intel:card1
NMI: 0 0 0 0 Non-maskable interrupts
LOC: 10116 10066 9694 9853 Local timer interrupts
SPU: 0 0 0 0 Spurious interrupts
PMI: 0 0 0 0 Performance monitoring interrupts
IWI: 808 698 598 4301 IRQ work interrupts
RTR: 0 0 0 0 APIC ICR read retries
RES: 2950 3180 3717 3080 Rescheduling interrupts
CAL: 7556 8256 5742 6800 Function call interrupts
TLB: 1080 781 519 815 TLB shootdowns
TRM: 0 0 0 0 Thermal event interrupts
THR: 0 0 0 0 Threshold APIC interrupts
DFR: 0 0 0 0 Deferred Error APIC interrupts
MCE: 0 0 0 0 Machine check exceptions
MCP: 1 2 2 2 Machine check polls
ERR: 1
MIS: 0
PIN: 0 0 0 0 Posted-interrupt notification event
NPI: 0 0 0 0 Nested posted-interrupt event
PIW: 0 0 0 0 Posted-interrupt wakeup event

after sudo dpkg headers and image =

uname -v=
#3 SMP PREEMPT_RT Sun Apr 28 23:39:38 CEST 2024

cat /proc/cmdline=
BOOT_IMAGE=/boot/vmlinuz-6.9.0-rc5-rt1+ root=UUID=1c171f88-4f36-42ba-b84a-1dcf20ae1a6b ro initrd=/install/gtk/initrd.gz quiet

cat /proc/interrupts=
CPU0 CPU1 CPU2 CPU3
0: 20 0 0 0 IO-APIC 2-edge timer
8: 0 0 0 0 IO-APIC 8-edge rtc0
9: 0 4 0 0 IO-APIC 9-fasteoi acpi
16: 0 0 0 25 IO-APIC 16-fasteoi ehci_hcd:usb1
18: 0 0 5 0 IO-APIC 18-fasteoi i801_smbus
23: 0 31 0 0 IO-APIC 23-fasteoi ehci_hcd:usb3
24: 0 0 0 0 PCI-MSI-0000:00:1c.0 0-edge PCIe PME, pciehp
25: 0 0 0 0 PCI-MSI-0000:00:1c.2 0-edge PCIe PME
26: 0 0 0 0 PCI-MSI-0000:00:1c.3 0-edge PCIe PME
27: 71 0 0 0 PCI-MSIX-0000:02:00.0 0-edge enp2s0
28: 4315 0 0 0 PCI-MSI-0000:00:14.0 0-edge xhci_hcd
29: 0 0 13954 0 PCI-MSI-0000:00:1f.2 0-edge ahci[0000:00:1f.2]
30: 0 0 0 7021 PCI-MSI-0000:00:02.0 0-edge i915
31: 22 0 0 0 PCI-MSI-0000:00:16.0 0-edge mei_me
32: 0 2407 0 0 PCI-MSI-0000:03:00.0 0-edge iwlwifi
33: 0 0 322 0 PCI-MSI-0000:00:03.0 0-edge snd_hda_intel:card0
34: 0 0 0 466 PCI-MSI-0000:00:1b.0 0-edge snd_hda_intel:card1
NMI: 1 0 0 1 Non-maskable interrupts
LOC: 26236 24028 22816 26873 Local timer interrupts
SPU: 0 0 0 0 Spurious interrupts
PMI: 1 0 0 1 Performance monitoring interrupts
IWI: 1074 1085 1382 4708 IRQ work interrupts
RTR: 0 0 0 0 APIC ICR read retries
RES: 907 822 916 928 Rescheduling interrupts
CAL: 5927 4453 4487 5567 Function call interrupts
TLB: 1085 470 1086 980 TLB shootdowns
TRM: 0 0 0 0 Thermal event interrupts
THR: 0 0 0 0 Threshold APIC interrupts
MCE: 0 0 0 0 Machine check exceptions
MCP: 1 2 2 2 Machine check polls
ERR: 1
MIS: 0

3 minutes ...error finish reading iter 11748...


so i dont understand why....

Weird indeed that it works better with the previous version.
What you could try is to isolate the last cpu and to pin the irq of the Ethernet to it:
sudo nano /etc/default/grub

Find the line: GRUB_CMDLINE_LINUX_DEFAULT=""
edit to : GRUB_CMDLINE_LINUX_DEFAULT="isolcpus=3"

Then save and exit: ctrl+x then press "y" to confirm

sudo update-grub

sudo reboot

Then pin the ethernet irq to the last cpu:
su -
echo 8 > /proc/irq/27/smp_affinity

If this works correctly you can try to add the following line to /etc/rc.local above "exit 0"
sleep 15 && echo 8 > /proc/irq/27/smp_affinity

The rt_setup.sh is a bit fussy since it depends on a host ip address in the interfaces file matching the address in the script.

Also, you cannot rely on the IRQ always being the same so royka's example may not always work
(though its good for testing)

Here is a simple script that pins the IRQ of a named device to the highest CPU: 

 

I would first rename to pinirq
mv pinirq.txt pinirq
and then
chmod +x pinirq
to make executable

to run by hand:

./pinirq  enp2s0

Note that you also need to disable irqbalance or it will undo the IRQ pinning
rt_setup.sh does this but only if run after you have edited the script to match your
interfaces file setup

good morning!
i will try first isolcpus.
then i would run by hand pinirq... that means start pc.. open terminal.. and
./pinirq enp2s0?


thanks all!

ps:

my line is now =
GRUB_CMDLINE_LINUX_DEFAULT="quiet splash noresume isolcpus=3"
before it was=
GRUB_CMDLINE_LINUX_DEFAULT="quiet"