Latency, error finishing read, and IRQ affinity

****** UPDATE - there is a setup script at the end of this post now that will set up kernel boot options and IRQ affinity scripts ******

Well I just had the "error finishing read" problem with a new system I'm setting up, and the problem turned out to be an issue with the script I use to set IRQ affinity for the network card which connects to the Mesa board.

This particular system is a mini PC with a realtek RTL8168 ethernet chip, and a quad core N95 CPU, running custom built 6.6.2-rt kernel and LinuxCNC 2.9.2.  I had it working fine for a while, and then it started giving the "error finishing read" error right after starting LinuxCNC every time.  After fixing my script to set IRQ affinity properly, I ran a 2.5 hour cut file while playing a 4K video on youtube over the wireless link without an error, and a servo-thread.tmax of about 588000 with a 1697 Mhz CPU.

In the past I have used this same process with great results with 5.4 and 5.10 rt kernels running on various mini PCs mostly having realtek network chips.  There are several things that must be in place for this to work properly, so I'm writing this post in hopes that it may help.

First, The kernel command line option 'isolcpus' must be used to isolate the last (highest numbered) CPU so that the kernel does not run any tasks on it unless they are specifically made to run on it.  LinuxCNC always runs its realtime task on that last CPU.
The only time any CPU other than the last one needs to be isolated is when another CPU shares a cache with it.  The command below run in a terminal will print a comma separated list of the last CPU and any others that share a cache with it.

On most quad core intel cpus I've seen recently such as the N95, that command will return "3" - which is the last core.  On my Ryzen 6-core machine it returns "5,11"

Edit the file /etc/default/grub:Add isolcpus={whatever the command above printed} to the line beginning with GRUB_CMDLINE_LINUX_DEFAULT=
For example:Now run: 
Now IRQ affinity is a little bit tricky.  If the irqbalance daemon is running, it typically runs every 10 seconds and moves IRQ handlers based on their activity to different CPUs to balance the load as evenly as it can.  It also automatically keeps IRQs off of the CPUs isolated by 'isolcpus'.  In this case we want the IRQ of the realtime network card to be handled on the isolated last CPU along with the LinuxCNC realtime task.
The IRQ can be moved by finding out which number it is, and then running the command: sudo echo "3" >  /proc/irq/#/smp_affinity_list  where "3" is the last CPU and # is the number of the IRQ for the network card.  The problem with this is that in a little bit irqbalance will move it back off that CPU because it is isolated!
You can disable irqbalance, but another problem is that setting the smp_affinity is only good until you reboot even if irqbalance is not running.
Yet another problem is that if you change the network card or port connected to the Mesa card, the IRQ number will change, meaning that if you hard-coded an IRQ number in a script it will have to be changed.
Fortunately irqbalance has a way to ban itself from moving certain IRQs, and a way to run a script when it starts up as it sets up each IRQ.  So my solution was to write a script which figures out which IRQ we need, ban irqbalance from moving it, and then set the affinity of that IRQ to the last CPU.

To use the script, I saved it in /etc/irqbalance.d (I think I created the directory), and made it executable.  Remove the .txt from the end.  It assumes the card is setup in the /etc/network/interfaces file with the IP 10.10.10.1 - just change the IP to match the one you use, or you could even simply change the NIC= line and enter the name of the network device directly, such as: NIC="enp1s0"

Then you have to change a line in /etc/default/irqbalance to use this script:
 

I hope this is helpful, and I might be able to post a script to automate setting some of this stuff up soon.
Moses

[Edit]
I forgot to add a couple of things useful for test and debug:
After you reboot with the script in place, you can run the following command to see if it worked:

You should see a line that says something like: ... /usr/sbin/irqbalance[611]: IRQ 125: Override ban to true

Another way to see what is happening with interrupts is to run the following command and look for the ethernet card in the list and see which CPU is handling the interrupts for it.

That can be run in a terminal while running LinuxCNC - or while pinging the Mesa card and you will see the number for the ethernet IRQ climbing on the CPU which is handling it.

[Edit 2] ****** UPDATE ******
Here is a script which will setup everything automatically. It will do the following:

1. Set up isolcpus and a couple of other kernel arguments
2. Run update-grub
3. For systems with irqbalance running: creates the IRQ affinity policy script for irqbalance and sets up irqbalance to use it.
4. For systems without irqbalance running: creates a script which manually sets all IRQ affinities and sets it up to be run at boot from /etc/rc.local

There are a few kernel options that *might* be helpful in reducing latency that can be enabled by uncommenting the relevant sections.
There are a couple of lines at the end which disable syncing the system clock to internet time servers.  This syncing has in the past been one cause of realtime errors.  This might should be made default???

Rename the script to "rt_setup.sh" and make it executable, then run it through sudo:After running this you will need to reboot the PC.

I'll have to research that more but it kinda looks like it's saying irqbalance won't run if there are not more than 1 CPUs which makes sense. If isolcpus is used then there is only 1 CPU irqbalance could use so it doesn't make sense to run it.

If irqbalance is not running you will need to run a script similar to mine to find the right IRQ and set the affinity at boot time. I think that can be done using rc.local.

Something like this should work:

Save the above as a script somewhere and make it executable, then call it from /etc/rc.local

You can see if it's working by running the following command in a terminal while LinuxCNC is running.  You should see the network card IRQ numbers increasing on the second CPU.
watch -n1 -d cat /proc/interrupts

The only thing I'm not sure of is if the rest of the IRQs will be handled by default on the first CPU.  This is the way it was in the past but I saw some mention that newer kernels may distribute IRQ handlers across CPUs.  If this is the case then the script needs to go through every other interrupt and move it to the first CPU.

It seems like in the past I had tested on dual core and had it working, but it could be that irqbalance or maybe the service that starts it has been changed in newer versions/distributions. The problem I had recently was due to the fact that it did 'exit 1' which worked fine in the past, but now it had to be 'exit 0'

Made this sticky, thank you.

Mecanix, the second script is for use without irqbalance running. It should be run from /etc/rc.local and not as a policyscript for irqbalance. I haven't tested it myself yet, but I know I have a dual-core or two around here that I can try it on. Probably be next week or so before I can find time to do that.