[SOLVED] Huge latency with PREEMPT_RT
- tommylight
-
- Away
- Moderator
-
Less
More
- Posts: 19190
- Thank you received: 6433
04 Jun 2020 18:08 #170113
by tommylight
Replied by tommylight on topic Huge latency with PREEMPT_RT
What PC?
In BIOS, everything not needed has to be disabled, like:
Hyperthreading, if it has it,
power savings,
c states,
modem
wake on lan
smart fan control,
pci-e power management,
serial ports,
virtualisation,
sound card if not using it,
In BIOS, everything not needed has to be disabled, like:
Hyperthreading, if it has it,
power savings,
c states,
modem
wake on lan
smart fan control,
pci-e power management,
serial ports,
virtualisation,
sound card if not using it,
The following user(s) said Thank You: arvidb
Please Log in or Create an account to join the conversation.
- BeagleBrainz
-
- Offline
- User is blocked
-
Less
More
- Posts: 1437
- Thank you received: 570
05 Jun 2020 06:23 #170204
by BeagleBrainz
Ok usually when users say they are using this or that driver they have installed a third party driver, that is how I took your statement.
The kernel I supply is plain vanilla, the only patch I used is the real time patch, by all reports latency seems pretty good.. After quickly running Buster with XFCE & Mint 19.2 with XFCE I found Mint to be snappier, which kind of surprised me.
I have a package that will pull in everything you need to rebuild Linuxcnc, no having to chase the dependencies for building Linuxcnc.
But in saying all that you are trying to use a 12 yro CPU, which in it's day it may have been a good choice, but 12 years later there are better choices. All the time you spending faffing around to get this setup working maybe better spent throwing a few extra beer tokens at a better system. Or you may or may not, depending on your MB, be able to upgrade to a Quad core. Isolate 2 cores and you should see an improvement. The Q9550 runs at a slight lower clock rate 2.83 vs 3.0Ghz but the L2 cache is greater 12MB vs 6MB.
Another thing you can do is look into what is running in the background at see what you can turn off.
At the end of the day is sounds like you are spending at bit of cash of your system, why not bite the bullet and get a computer that will run without having to spending a lot of time & hassle to reduce latency ?
I myself tried a few different options with some gear I had lying around and ended up just going with a new mb.
If you really want to test out Mint & XFCE I can upload one of my Live ISO's to my server and you can give that a spin, just dd it to a USB drive.The one there I have ATM is MATE, which really wont be a lot of good to you.
Replied by BeagleBrainz on topic Huge latency with PREEMPT_RT
PCW: No built-in graphics unfortunately. I do have another graphics card (that uses the amdgpu instead of the radeon driver) that I could use temporarily for testing though, if it turns out to be necessary.
BeagleBrainz: Thanks for trying to help.
* As I said, and as you can see from Intel's E8400 page , this is a 2-core CPU without hyperthreading.
* As I wrote, I'm using the radeon driver already (i.e. the default, in-kernel driver). No 3rd party drivers installed.
* As you can see from my previous message, I'm already running xfce.
* I'm actually already using a PS/2 keyboard on this computer, but I have no PS/2 mouse to try unfortunately. Also nothing wireless.
Do you have any specific reasons to believe another distribution (Mint) would help, despite running the same kernel, drivers and window manager on the same hardware? Redoing the install is a large undertaking since I'm using EtherCAT - I have to build everything from source. (It's great that you're eager to help, but it mostly feels like you're sending me on wild-goose chases by not taking time to understand the circumstances...)
Ok usually when users say they are using this or that driver they have installed a third party driver, that is how I took your statement.
The kernel I supply is plain vanilla, the only patch I used is the real time patch, by all reports latency seems pretty good.. After quickly running Buster with XFCE & Mint 19.2 with XFCE I found Mint to be snappier, which kind of surprised me.
I have a package that will pull in everything you need to rebuild Linuxcnc, no having to chase the dependencies for building Linuxcnc.
But in saying all that you are trying to use a 12 yro CPU, which in it's day it may have been a good choice, but 12 years later there are better choices. All the time you spending faffing around to get this setup working maybe better spent throwing a few extra beer tokens at a better system. Or you may or may not, depending on your MB, be able to upgrade to a Quad core. Isolate 2 cores and you should see an improvement. The Q9550 runs at a slight lower clock rate 2.83 vs 3.0Ghz but the L2 cache is greater 12MB vs 6MB.
Another thing you can do is look into what is running in the background at see what you can turn off.
At the end of the day is sounds like you are spending at bit of cash of your system, why not bite the bullet and get a computer that will run without having to spending a lot of time & hassle to reduce latency ?
I myself tried a few different options with some gear I had lying around and ended up just going with a new mb.
If you really want to test out Mint & XFCE I can upload one of my Live ISO's to my server and you can give that a spin, just dd it to a USB drive.The one there I have ATM is MATE, which really wont be a lot of good to you.
Please Log in or Create an account to join the conversation.
05 Jun 2020 07:04 #170206
by rodw
Replied by rodw on topic Huge latency with PREEMPT_RT
Beagle, what motherboard would you suggest for a Mesa ethernet config ?
Please Log in or Create an account to join the conversation.
05 Jun 2020 07:04 - 05 Jun 2020 07:04 #170207
by rodw
Replied by rodw on topic Huge latency with PREEMPT_RT
duplicate - deleted
Last edit: 05 Jun 2020 07:04 by rodw.
Please Log in or Create an account to join the conversation.
- BeagleBrainz
-
- Offline
- User is blocked
-
Less
More
- Posts: 1437
- Thank you received: 570
05 Jun 2020 08:12 #170213
by BeagleBrainz
Replied by BeagleBrainz on topic Huge latency with PREEMPT_RT
I'm using a Asrock J3355B, not too sure if they are readily available. Ok Amazon AU has 3.
They support UEFI & Legcay boot. So no issues there.
www.amazon.com.au/ASRock-Motherboard-Com...5B-ITX/dp/B01M9EXCYB
They support UEFI & Legcay boot. So no issues there.
www.amazon.com.au/ASRock-Motherboard-Com...5B-ITX/dp/B01M9EXCYB
The following user(s) said Thank You: rodw
Please Log in or Create an account to join the conversation.
05 Jun 2020 08:22 #170215
by Clive S
From the UK www.ebay.co.uk/itm/NEW-Asrock-J3355B-ITX...8:g:x04AAOSw205exEQq
Replied by Clive S on topic Huge latency with PREEMPT_RT
I'm using a Asrock J3355B, not too sure if they are readily available. Ok Amazon AU has 3.
They support UEFI & Legcay boot. So no issues there.
www.amazon.com.au/ASRock-Motherboard-Com...5B-ITX/dp/B01M9EXCYB
From the UK www.ebay.co.uk/itm/NEW-Asrock-J3355B-ITX...8:g:x04AAOSw205exEQq
The following user(s) said Thank You: rodw
Please Log in or Create an account to join the conversation.
22 Jun 2020 09:03 #172307
by arvidb
I don't know if it was the radeon driver or the fact that the card was failing that caused the spikes.
Replied by arvidb on topic Huge latency with PREEMPT_RT
The graphics card I used, an old radeon one, has been more and more shaky, with display corruption and now no longer even showing anything at boot. So I replaced it with a newer one using the amdgpu driver and the spikes are gone - same usage as before gives a maximum 27.9 µs deviation:Using Firefox to watch YouTube however still brings some spikes up to about 500-700 µs (but *much* rarer than the continuous 2 ms spikes I got with EIST enabled)
I don't know if it was the radeon driver or the fact that the card was failing that caused the spikes.
Attachments:
Please Log in or Create an account to join the conversation.
28 Jun 2020 20:33 - 28 Jun 2020 20:36 #173030
by Mud
Replied by Mud on topic [SOLVED] Huge latency with PREEMPT_RT
I'm having issues similar to those described in this thread, but I'm not having any luck finding them. Key facts:
*Linux Mint 19.1 with preempt (readying for a 7i96, which I currently have communicating and moving a motor)
*I believe I have disabled everything optional in the BIOS
*isolcpus=1 has tightened the histogram, but still with the big periodic spikes of 1+ms
*using onboard graphics did not help (attached image with no glxgears running)
*I'm running the onboard NIC and a PCI NIC (Realtek RTL8111/8164/8411 onboard connected to 7i96 and RTL8196/8110 PCI card for internet)
I'm new to this level of fiddling with Linux, so don't assume anything! One thing I am confused about from the process of jumping between different guides in setting up the OS, was I correct in making the changes to the /etc/network/interfaces file ( here ) - I did this before installing the second card, and wasn't sure if I had created a bad situation there (especially since reading here makes the network connection manager seem a bit flaky. I was not sure if tommylight's instructions implied the network manager was being used for everything and the gnipsel changes should not be done. I didn't copy the text I changed in the interfaces file, so it would take me a long time to go back to check! I set 'managed=true' under [ifupdown] in NetworkManager.conf because initially the network connection wasn't present in the network manager, but my memory is getting hazy at this point (and not sure if this network stuff is a red herring anyway!).
The motherboard I'm using (Gigabyte ga-g31mf-s2) isn't full size, and I wonder if it is just not a great candidate.
Apologies for the wall of text! Thanks in advance.
*Linux Mint 19.1 with preempt (readying for a 7i96, which I currently have communicating and moving a motor)
*I believe I have disabled everything optional in the BIOS
*isolcpus=1 has tightened the histogram, but still with the big periodic spikes of 1+ms
*using onboard graphics did not help (attached image with no glxgears running)
*I'm running the onboard NIC and a PCI NIC (Realtek RTL8111/8164/8411 onboard connected to 7i96 and RTL8196/8110 PCI card for internet)
I'm new to this level of fiddling with Linux, so don't assume anything! One thing I am confused about from the process of jumping between different guides in setting up the OS, was I correct in making the changes to the /etc/network/interfaces file ( here ) - I did this before installing the second card, and wasn't sure if I had created a bad situation there (especially since reading here makes the network connection manager seem a bit flaky. I was not sure if tommylight's instructions implied the network manager was being used for everything and the gnipsel changes should not be done. I didn't copy the text I changed in the interfaces file, so it would take me a long time to go back to check! I set 'managed=true' under [ifupdown] in NetworkManager.conf because initially the network connection wasn't present in the network manager, but my memory is getting hazy at this point (and not sure if this network stuff is a red herring anyway!).
The motherboard I'm using (Gigabyte ga-g31mf-s2) isn't full size, and I wonder if it is just not a great candidate.
Apologies for the wall of text! Thanks in advance.
Last edit: 28 Jun 2020 20:36 by Mud.
Please Log in or Create an account to join the conversation.
29 Jun 2020 12:07 #173077
by Mud
Replied by Mud on topic [SOLVED] Huge latency with PREEMPT_RT
I've run the latency script to help with my hardware spec.
LinuxCNC pc tests, version 20200624, started 29.06.2020 13:02:43
********************************************************************************
General info:
Gigabyte Technology Co., Ltd., G31MF-S2
Bios version F1, dated 01/07/2008
Chipset: Harddisk is non-rotational, ie. SSD
The amount of swap currently used is 0
================================================================================
CPU related data:
model name : Intel(R) Core(TM)2 Duo CPU E8400 @ 3.00GHz
cpu cores : 2
stepping : 6
cache size : 6144 KB
================================================================================
RAM related data:
Maximum Memory Module Size: 1024 MB
Maximum Total Memory Size: 4096 MB
Maximum Capacity: 4 GB
Maximum Memory Module Size: 1024 MB
Maximum Total Memory Size: 4096 MB
Installed Size: 2048 MB (Single-bank Connection)
Enabled Size: 2048 MB (Single-bank Connection)
Installed Size: Not Installed
Enabled Size: Not Installed
Installed Size: 2048 MB (Single-bank Connection)
Enabled Size: 2048 MB (Single-bank Connection)
Installed Size: Not Installed
Enabled Size: Not Installed
Size: 2048 MB
Size: No Module Installed
Size: 2048 MB
Size: No Module Installed
================================================================================
GPU related data:
product: 82G33/G31 Express Integrated Graphics Controller
configuration: driver=i915 latency=0
================================================================================
Os related data:
PRETTY_NAME="Linux Mint 19.1"
Desktop: , Session:
================================================================================
Kernel related data:
Kernel 4.19.1-rt3
GRUB_CMDLINE_LINUX_DEFAULT="isolcpus=1"
Cpu idle driver: none
================================================================================
Kernel modules data:
Check if pcspkr is loaded: No, good!
Yes, may cause bigger latency.
Check if snd modules are loaded: No, good!
Yes, may cause bigger latency.
================================================================================
Keyboard & Mouse related data:
Number of mice: 1
Mouse 0 : Logitech USB Laser Mouse
Mice attached to USB:
Logitech, Inc. M-U0007 [Corded Mouse M500]
Keyboards attached to USB:
Chicony Electronics Co., Ltd KU-2971/KU-0325 Keyboard
LinuxCNC related data:
LinuxCNC version is: 2.8.0~pre1
================================================================================
Preparing for latency test:
Started glxgears No. 1
Started glxgears No. 2
Started glxgears No. 3
Started glxgears No. 4
Started glxgears No. 5
..........
Started firefox with 'https://vimeo.com/150574260'
================================================================================
Command for latency test is:
su mud -c "latency-histogram --bbinsize 1000 --base 25000 --sbinsize 1000 --nox 2>&1 | tee 1593432134.lat &"
Latency testing loop started Mon 29 Jun 13:02:59 BST 2020,
should end after Mon 29 Jun 15:02:59 BST 2020
Unexpected realtime delay on task 0 with period 25000
should end after Mon 29 Jun 15:02:59 BST 2020
Swap useage: prerun: 0, running: 0
================================================================================
last latency data is as follows:.
12 secs base min: -24.900 uS max:1017.400 uS sdev: 0.400 uS
12 secs servo min:-955.100 uS max: 955.000 uS sdev: 0.600 uSPlease Log in or Create an account to join the conversation.
- tommylight
-
- Away
- Moderator
-
Less
More
- Posts: 19190
- Thank you received: 6433
29 Jun 2020 12:31 #173079
by tommylight
Replied by tommylight on topic [SOLVED] Huge latency with PREEMPT_RT
Are you sure you have hyperthreading and speedstep disabled in BIOS ?
Please Log in or Create an account to join the conversation.
Time to create page: 0.145 seconds