I did soma latency testing and tuning. What helped most was to turn off hypertheading which reducedjitter from 30 to around 22 microseconds. Other bios settings like HPET off, EPU power saving, and cpu turbo on or off in my case made little to no difference. So did isolcpus, irqaffinity, nohz_full, and rcu_nocbs. rcu_nocb_pull yielded one microsecond less, so did setting sched_rt_rumtime_us to -1.
Please note RedHat warns to do this;
Issue
Is it safe to set /proc/sys/kernel/sched_rt_runtime_us to -1?
Resolution
In short, setting sched_rt_runtime_us to -1 can be extremely dangerous. A value of -1 means no limit. In other words, a "run-away" real-time task will be permitted to monopolise a CPU which could (potentially) lock up a system. The default value for sched_rt_runtime_us is 950000 or 0.95 seconds. Broadly speaking, this allows 0.05 seconds to be used by other non-real-time tasks (i.e. SCHED_OTHER) - a small window at which to recover a system.
The kernel parameter namely /proc/sys/kernel/sched_rt_runtime_us can be set between -1 and (INT_MAX - 1).
I left it at -1 anyway. In the end I was able to reduce jitter from sightly above 30 microseconds to just below 20. All test were done with one instance of glxgears running.
