Advanced Search

Working through this,

irqbalance was not installed by default in Debian since Debian 9 Stretch because the kernel now does its own rebalancing. In some case irqbalance has caused  instabilities. Accordingly any advice about irqbalance is obsolete. If you have installed it, you should purge it
sudo apt purge irqbalance

Also cpufrequtils is deprecated in Trixie and has been replaced by the  linux-cpupower package. to set your governor to performance, use ( I think)

Also, I did not see where you are disabling Energy Efficient Ethernet.  It is supposed to be disabled by default but this is not always the case. There are some flags that can be set in GRUB but they are driver specific. eg. igb.EEE=0 

I have not tested this but This is a robust way to ensure EEE is disabled on every boot. Create a new systemd service file:
sudo geany /etc/systemd/system/disable-eee.service

1. Add the following content, replacing enp6s0 with your interface name. This service will run ethtool after the network is up.

1. Save and close the file.

Reload the systemd daemon:
Enable and start the service:

Yes, i read thier manual 1k times.. it is pretty solid extraction of what we all need...

Yes I am recompiling everytime to native NIC driver as common e1000e or rtl8190too ... It works pretty solid for 8khz thread 
8190too are basically free stuff on the market right now it cost few bucks

There are native drivers for beckhoff too but I have never tried them yet ... 

I am not expert in this drivers ..but if it will improve my latency i need them all ...
 

@endian, you may be interested to read the ethercat master docs Chapter 4, networking where it explains its interrupt-less network drivers that bypass all of the CPU affinity requirement...
etherlab.org/en_GB/ethercat

Also, ethercat has a number of specific NIC drivers but I seem to remember these are not included in the deb file in their repo and you need to compile from source to obtain them. It looks like they now support the Trixie 6.12 kernel. Driver Source etherlab.org/en_GB/ethercathttps://gitla...vices?ref_type=heads

drivers are loaded by ethercat.conf in place of ec_generic. It would be interesting to see if the  hm2_eth driver could be modified to use the ethercat drivers

I think it is higher ladies school ... Customizing the kernel we will lost possibility of regular installation maybe ... Of course we can have own kernel but there will be limitations of any updates... But on other side who will use it from regular users... but I am not expert of kernels at all.. i know just how to create great ethercat topology with low thread time at budget iron..

I am not expert of Lcnc too.. but Lcnc needs two major improvements... Latency of networking(we can use mesa and peer to peer connection) and new 7phase trajectory planner(created by nonRT + RT part)

Both are non trivial and boths are long range run.. 

I don't think you can improve much on this without building the kernel from source and changing some of the config parameters. (which is not hard)
here are a couple of links I had saved
ubuntu.com/blog/real-time-kernel-tuning
eci.intel.com/docs/3.3/development/perfo...e/rt_scheduling.html
I did try installing the Intel tools but they did not like Trixie.

the only other idea I explored was to use time sensitive networking (TSN) protocols en.wikipedia.org/wiki/Time-Sensitive_Networking
But I think this would require writing a new hm2-eth driver (hm2-tsn?) and changes to the mesa hardware to support it.
This should solve the issue where networking does not currently support real time (networks are inherently store and forward so never designed for RT comms) but its not a trivial project. This would require TSN capable network cards but these are available at reasonable cost but the common entry level i225/i226 cards do not support TSN. ref: www.intel.com/content/www/us/en/support/...rs-up-to-2-5gbe.html I did find some for USD $70 to $150

If gentelmen have any other tips for reducing latency .. share it please

I did changes for my machine and it runs for few days at 3khz without problem...

Reading the datas take less then 250us with timeout 100000ns and 250us is now limit... + Other stuff in the thread and we are under the 300us

Motion is smoother by the sound and scope profile and compensation of position in the velocity mode is faster==smoother ... It is real difference now

 

That's a great condensed source of network latency reducing tips

Only quibble I would have is that:

setp hm2_7i76e.0.read.timeout 100000  

Is not correct as the hm2 read timeout is scaled in percent of the servo period

EDIT: actually that is correct as values larger than 100 are interpreted as ns

But a 100 usec margin is very small on most systems and will likely result in many
unnecessarily dropped packets .

Default is 80% of servo period and this is fine normally




 

Rod share your tips please or add them to my guide ... I really like to learn something new about this topic!

Thanks

Excellent info. I've started working on a similar procedure.There are a few more optimisations but I'll test before sharing anything.
Now PREEMPT_RT is in the mainline, there are a lot of articles emerging about  RT tuning.

I wonder if it would be worth having a sticky thread that is an index to some of these great posts regarding latency and such.

To reduce network latency:

1. Make sure basic host latency setup is done (disable power management etc)
Actions:

sudo apt update
sudo apt install net-tools

Edit your GRUB config:
sudo nano /etc/default/grub

Find the line starting with:
GRUB_CMDLINE_LINUX_DEFAULT=

Append these parameters:
quiet splash isolcpus=nohz,domain,managed_irq,1 intel_idle.max_cstate=0 processor.max_cstate=1 idle=poll

Adjust isolcpus numbers later once you know your CPU layout.
Then update and reboot:
sudo update-grub
sudo reboot

Also:
Disable CPU frequency scaling:

 sudo apt install cpufrequtils
echo 'GOVERNOR="performance"' | sudo tee /etc/default/cpufrequtils
sudo systemctl enable --now cpufrequtils

Disable power-saving for NIC and USB:

 sudo ethtool -s eth0 wol d
sudo systemctl mask sleep.target suspend.target hibernate.target hybrid-sleep.target

2. If you have an intel MAC on the host PC, disable IRQ coalescing.
IRQ coalescing groups interrupts together to reduce CPU load — bad for real-time use.
Run:
sudo ethtool -C eth0 rx-usecs 0 rx-frames 0 tx-usecs 0 tx-frames 0

Make it persistent by adding it to:
sudo nano /etc/network/interfaces

Example:
auto eth0
iface eth0 inet static
    address 192.168.1.10
    netmask 255.255.255.0
    pre-up /sbin/ethtool -C eth0 rx-usecs 0 rx-frames 0 tx-usecs 0 tx-frames 0

3. Use isolcpus to run LinuxCNC on the last CPU
You “isolate” a CPU core so only LinuxCNC and its threads run there.
Example:
 If you have 4 cores (0–3), you isolate the last one:
isolcpus=3 nohz_full=3 rcu_nocbs=3

That goes into your GRUB line (as above). Then:
sudo update-grub
sudo reboot

To verify:
cat /sys/devices/system/cpu/isolated

4.  (in conjunction with #3) Pin the Ethernet IRQ to the last CPU
(and set irqbalance to one-shot mode so it doesn't mess with the IRQ pinning)
First, find your Ethernet IRQ number:
grep eth0 /proc/interrupts

You’ll see something like:
123:   12345   0   0   0   IR-PCI-MSI  eth0

→ IRQ number is 123
Then pin it:
echo 8 | sudo tee /proc/irq/123/smp_affinity

Here 8 is a bitmask (binary 1000) meaning CPU 3.
 (Use 1, 2, 4, 8, etc., depending on which CPU you isolated.)
irqbalance dynamically moves IRQs — you don’t want that after pinning.
Edit:
sudo nano /etc/default/irqbalance

Find this line:
ENABLED="1"

Change to:
ENABLED="1"
ONESHOT="1"

Then restart:
sudo systemctl restart irqbalance

irqbalance dynamically moves IRQs — you don’t want that after pinning.
Edit:
sudo nano /etc/default/irqbalance

Find this line:
ENABLED="1"

Change to:
ENABLED="1"
ONESHOT="1"

Then restart:
sudo systemctl restart irqbalance

5. Set irqbalance to One-Shot Mode
irqbalance dynamically moves IRQs — you don’t want that after pinning.
Edit:
sudo nano /etc/default/irqbalance

Find this line:
ENABLED="1"

Change to:
ENABLED="1"
ONESHOT="1"

Then restart:
sudo systemctl restart irqbalance


To check maximum network latency:
sudo chrt 99 ping -i .001 -q -c 60000 192.168.1.10
( Assuming 192.168.1.10 is the 7I76E IP address ) 

This will run for 1 minute and print statistics


Command Breakdown
sudo chrt 99 ping -i .001 -q -c 60000 192.168.1.10

Part
Meaning
sudo
Run with root privileges (needed for chrt and for such a short ping interval).
chrt 99
Runs the command with real-time priority 99 (highest possible SCHED_FIFO priority).
ping
Standard ICMP ping utility.
-i .001
Interval of 1 millisecond (1000 Hz) between packets — very fast.
-q
Quiet mode (only summary results).
-c 60000
Send 60,000 pings, which takes about 60 seconds at 1 kHz.
192.168.1.10
Target IP address (replace with your remote machine or device).


What It Tests
Latency/jitter of your Ethernet path between your host and the device at 192.168.1.10

Consistency under real-time scheduling (since chrt gives the ping thread real-time priority).

Impact of CPU isolation and IRQ pinning on network determinism.

then

How to Interpret the Results
When it finishes, you’ll get something like:
--- 192.168.1.10 ping statistics ---
60000 packets transmitted, 60000 received, 0% packet loss, time 60040ms
rtt min/avg/max/mdev = 0.120/0.135/0.210/0.015 ms

Value
Meaning
min
Best-case latency.
avg
Typical latency.
max
Worst-case latency (what we care about most).
mdev
Standard deviation (jitter).

Good results for a well-tuned LinuxCNC host:
max under 0.5 ms (ideally < 0.2 ms).

No packet loss.

mdev small (under 0.05 ms).

Bad results (what to look out for):
max spikes to several milliseconds → indicates power management or IRQ contention.

Packet loss → NIC or cable issues.

mdev large → jittery CPU or network driver.

Tips
Make sure your Ethernet link partner (the device at 192.168.1.10) responds quickly — e.g., another PC on the same subnet, or an EtherCAT master/slave device.

Run it while LinuxCNC is idle and again under load (with the machine moving) — compare the two.

For continuous monitoring, use:

 watch -n 1 "ping -q -c 1000 192.168.1.10"
 to sample shorter bursts every second.

then 

edit servo thread time to for example Ts=333 333 ns in the .ini file

finaly

edit setp hm2_7i76e.0.read.timeout 100000      # 100μs or less depends at measured latency

and check the time of servo thread which has to be less then for example < then Ts
 

hello PCW,

base thread or servo thread latency is way under 6us ... 

here are statistics 



what to do next please? regards and thanks

Hmmm, the plot thickens. If I run the qtdragon interface,as in the config file I have uploaded, I get the following error. But if I switch the UI to axis in the ini file, no other change, it works...any ideas how that can be?

Hello

I have a question. Which *.ngc files and variable entries do I need if I want to use a Rack ATC with LinuxCNC 2.9.7 and a Mesa 7i76e?

Regards,
Hilton