[SOLVED] Huge latency with PREEMPT_RT

More
04 Jun 2020 18:08 #170113 by tommylight
What PC?
In BIOS, everything not needed has to be disabled, like:
Hyperthreading, if it has it,
power savings,
c states,
modem
wake on lan
smart fan control,
pci-e power management,
serial ports,
virtualisation,
sound card if not using it,
The following user(s) said Thank You: arvidb

Please Log in or Create an account to join the conversation.

More
05 Jun 2020 06:23 #170204 by BeagleBrainz

PCW: No built-in graphics unfortunately. I do have another graphics card (that uses the amdgpu instead of the radeon driver) that I could use temporarily for testing though, if it turns out to be necessary.

BeagleBrainz: Thanks for trying to help.
* As I said, and as you can see from Intel's E8400 page , this is a 2-core CPU without hyperthreading.
* As I wrote, I'm using the radeon driver already (i.e. the default, in-kernel driver). No 3rd party drivers installed.
* As you can see from my previous message, I'm already running xfce.

* I'm actually already using a PS/2 keyboard on this computer, but I have no PS/2 mouse to try unfortunately. Also nothing wireless.

Do you have any specific reasons to believe another distribution (Mint) would help, despite running the same kernel, drivers and window manager on the same hardware? Redoing the install is a large undertaking since I'm using EtherCAT - I have to build everything from source. (It's great that you're eager to help, but it mostly feels like you're sending me on wild-goose chases by not taking time to understand the circumstances...)


Ok usually when users say they are using this or that driver they have installed a third party driver, that is how I took your statement.
The kernel I supply is plain vanilla, the only patch I used is the real time patch, by all reports latency seems pretty good.. After quickly running Buster with XFCE & Mint 19.2 with XFCE I found Mint to be snappier, which kind of surprised me.
I have a package that will pull in everything you need to rebuild Linuxcnc, no having to chase the dependencies for building Linuxcnc.

But in saying all that you are trying to use a 12 yro CPU, which in it's day it may have been a good choice, but 12 years later there are better choices. All the time you spending faffing around to get this setup working maybe better spent throwing a few extra beer tokens at a better system. Or you may or may not, depending on your MB, be able to upgrade to a Quad core. Isolate 2 cores and you should see an improvement. The Q9550 runs at a slight lower clock rate 2.83 vs 3.0Ghz but the L2 cache is greater 12MB vs 6MB.
Another thing you can do is look into what is running in the background at see what you can turn off.

At the end of the day is sounds like you are spending at bit of cash of your system, why not bite the bullet and get a computer that will run without having to spending a lot of time & hassle to reduce latency ?
I myself tried a few different options with some gear I had lying around and ended up just going with a new mb.

If you really want to test out Mint & XFCE I can upload one of my Live ISO's to my server and you can give that a spin, just dd it to a USB drive.The one there I have ATM is MATE, which really wont be a lot of good to you.
The following user(s) said Thank You: arvidb, Clive S

Please Log in or Create an account to join the conversation.

More
05 Jun 2020 07:04 #170206 by rodw
Replied by rodw on topic Huge latency with PREEMPT_RT
Beagle, what motherboard would you suggest for a Mesa ethernet config ?

Please Log in or Create an account to join the conversation.

More
05 Jun 2020 07:04 - 05 Jun 2020 07:04 #170207 by rodw
Replied by rodw on topic Huge latency with PREEMPT_RT
duplicate - deleted
Last edit: 05 Jun 2020 07:04 by rodw.

Please Log in or Create an account to join the conversation.

More
05 Jun 2020 08:12 #170213 by BeagleBrainz
I'm using a Asrock J3355B, not too sure if they are readily available. Ok Amazon AU has 3.
They support UEFI & Legcay boot. So no issues there.

www.amazon.com.au/ASRock-Motherboard-Com...5B-ITX/dp/B01M9EXCYB
The following user(s) said Thank You: rodw

Please Log in or Create an account to join the conversation.

More
05 Jun 2020 08:22 #170215 by Clive S

I'm using a Asrock J3355B, not too sure if they are readily available. Ok Amazon AU has 3.
They support UEFI & Legcay boot. So no issues there.

www.amazon.com.au/ASRock-Motherboard-Com...5B-ITX/dp/B01M9EXCYB


From the UK www.ebay.co.uk/itm/NEW-Asrock-J3355B-ITX...8:g:x04AAOSw205exEQq
The following user(s) said Thank You: rodw

Please Log in or Create an account to join the conversation.

More
22 Jun 2020 09:03 #172307 by arvidb
Replied by arvidb on topic Huge latency with PREEMPT_RT

Using Firefox to watch YouTube however still brings some spikes up to about 500-700 µs (but *much* rarer than the continuous 2 ms spikes I got with EIST enabled)

The graphics card I used, an old radeon one, has been more and more shaky, with display corruption and now no longer even showing anything at boot. So I replaced it with a newer one using the amdgpu driver and the spikes are gone - same usage as before gives a maximum 27.9 µs deviation:


I don't know if it was the radeon driver or the fact that the card was failing that caused the spikes.
Attachments:

Please Log in or Create an account to join the conversation.

More
28 Jun 2020 20:33 - 28 Jun 2020 20:36 #173030 by Mud
I'm having issues similar to those described in this thread, but I'm not having any luck finding them. Key facts:

*Linux Mint 19.1 with preempt (readying for a 7i96, which I currently have communicating and moving a motor)
*I believe I have disabled everything optional in the BIOS
*isolcpus=1 has tightened the histogram, but still with the big periodic spikes of 1+ms
*using onboard graphics did not help (attached image with no glxgears running)
*I'm running the onboard NIC and a PCI NIC (Realtek RTL8111/8164/8411 onboard connected to 7i96 and RTL8196/8110 PCI card for internet)

I'm new to this level of fiddling with Linux, so don't assume anything! One thing I am confused about from the process of jumping between different guides in setting up the OS, was I correct in making the changes to the /etc/network/interfaces file ( here ) - I did this before installing the second card, and wasn't sure if I had created a bad situation there (especially since reading here makes the network connection manager seem a bit flaky. I was not sure if tommylight's instructions implied the network manager was being used for everything and the gnipsel changes should not be done. I didn't copy the text I changed in the interfaces file, so it would take me a long time to go back to check! I set 'managed=true' under [ifupdown] in NetworkManager.conf because initially the network connection wasn't present in the network manager, but my memory is getting hazy at this point (and not sure if this network stuff is a red herring anyway!).

The motherboard I'm using (Gigabyte ga-g31mf-s2) isn't full size, and I wonder if it is just not a great candidate.

Apologies for the wall of text! Thanks in advance.
Attachments:
Last edit: 28 Jun 2020 20:36 by Mud.

Please Log in or Create an account to join the conversation.

More
29 Jun 2020 12:07 #173077 by Mud
I've run the latency script to help with my hardware spec.
LinuxCNC pc tests, version 20200624, started 29.06.2020 13:02:43
********************************************************************************
General info:
Gigabyte Technology Co., Ltd.,  G31MF-S2
Bios version F1, dated 01/07/2008
Chipset: Harddisk is non-rotational, ie. SSD
The amount of swap currently used is 0
================================================================================
CPU related data:
model name	: Intel(R) Core(TM)2 Duo CPU     E8400  @ 3.00GHz
cpu cores	: 2
stepping	: 6
cache size	: 6144 KB
================================================================================
RAM related data:
Maximum Memory Module Size: 1024 MB
Maximum Total Memory Size: 4096 MB
Maximum Capacity: 4 GB
Maximum Memory Module Size: 1024 MB
Maximum Total Memory Size: 4096 MB
Installed Size: 2048 MB (Single-bank Connection)
Enabled Size: 2048 MB (Single-bank Connection)
Installed Size: Not Installed
Enabled Size: Not Installed
Installed Size: 2048 MB (Single-bank Connection)
Enabled Size: 2048 MB (Single-bank Connection)
Installed Size: Not Installed
Enabled Size: Not Installed
Size: 2048 MB
Size: No Module Installed
Size: 2048 MB
Size: No Module Installed
================================================================================
GPU related data:
product: 82G33/G31 Express Integrated Graphics Controller
configuration: driver=i915 latency=0
================================================================================
Os related data:
PRETTY_NAME="Linux Mint 19.1"
Desktop: , Session: 
================================================================================
Kernel related data:
Kernel 4.19.1-rt3
GRUB_CMDLINE_LINUX_DEFAULT="isolcpus=1"
Cpu idle driver: none
================================================================================
Kernel modules data:
Check if pcspkr is loaded: No, good!
Yes, may cause bigger latency.
Check if snd modules are loaded: No, good!
Yes, may cause bigger latency.
================================================================================
Keyboard & Mouse related data:
Number of mice: 1
Mouse 0 : Logitech USB Laser Mouse
Mice  attached to USB:
Logitech, Inc. M-U0007 [Corded Mouse M500]
Keyboards attached to USB:
Chicony Electronics Co., Ltd KU-2971/KU-0325 Keyboard
LinuxCNC related data:
LinuxCNC version is: 2.8.0~pre1
================================================================================
Preparing for latency test:
Started glxgears No. 1
Started glxgears No. 2
Started glxgears No. 3
Started glxgears No. 4
Started glxgears No. 5
..........
Started firefox with 'https://vimeo.com/150574260' 

================================================================================
Command for latency test is:
su mud -c  "latency-histogram --bbinsize 1000 --base 25000 --sbinsize 1000 --nox 2>&1 | tee  1593432134.lat &"

Latency testing loop started Mon 29 Jun 13:02:59 BST 2020, 
should end after Mon 29 Jun 15:02:59 BST 2020

Unexpected realtime delay on task 0 with period 25000
should end after Mon 29 Jun 15:02:59 BST 2020

Swap useage: prerun: 0, running: 0

================================================================================
last latency data is as follows:.
   12 secs   base min: -24.900 uS max:1017.400 uS sdev:   0.400 uS
   12 secs  servo min:-955.100 uS max: 955.000 uS sdev:   0.600 uS

Please Log in or Create an account to join the conversation.

More
29 Jun 2020 12:31 #173079 by tommylight
Are you sure you have hyperthreading and speedstep disabled in BIOS ?

Please Log in or Create an account to join the conversation.

Time to create page: 0.232 seconds
Powered by Kunena Forum