Ugh, another machine with rotten latency

I just got a Dell Optiplex GX620 with a 3 GHz Pentium 4 and 2 GB of memory.
I put Ubuntu 10.04 on it. Just leaving everything at default, I got latency of
639000 on the servo thread and 23000 on the base thread.
I turned off hyperthreading in the BIOS and the servo thread went down to
320000. This doesn't look like the typical SMI problem, as the
320000 reappears instantly when you clear the peak latency.
Running glxgears made no difference, and moving things around on
the screen didn't, either.

Does anyone have any idea what might cause this?

This machine will never run a real machine tool, but it will be used for
bench testing my interface boards, so I'd like to get the latency down
at least a little. This is probably the worst machine I've ever tested!

Thanks,

Jon

Same with a gx520, but I did find that my will work with 8.04, have you tried that?
There seems to be a spike a minute or two after starting but then all is well.
I have run it for hours on 8.04 with no problem.
Also make sure hyper-threading is off.

I also found that if you run the "cpu hog" at the same time you get very good numbers.
wiki.linuxcnc.org/emcinfo.pl?action=browse&diff=1&id=RealTime

I am thinking this is a case of where a non smp kernel will work better.
I am looking into this.


Rick G

Rick G wrote:

Same with a gx520, but I did find that my will work with 8.04, have you tried that?

This machine will primarily be used for other purposes which require a new kernel. But, if
a simple fix can get the latency down, I'd like to do that.

Also make sure hyper-threading is off.

I did that, it cut the latency in half.

I also found that if you run the "cpu hog" at the same time you get very good numbers.
wiki.linuxcnc.org/emcinfo.pl?action=browse&diff=1&id=RealTime

I am thinking this is a case of where a non smp kernel will work better.
I am looking into this.

Since this is a single processor system, I don't think that will help. Anyway, I don't want
to have a 50% time waster running all the time.

Since the way I run things does not require a base thread, it may be that the servo thread
will have decent latency most of the time. Since I will only use it to test controller boards
and such, it is not really important to have the kind of low latency needed for a real
machine tool.

It might also be the on-board video, I will have to test that out.

Jon

I have tested three different Dell models (Optiplex 740, GX520, GX270) because I have lots of Dells here. All have the same latency issues, if I remember right I think worst was the 740 that gave me a max jitter bigger than 1000000. I love them for everything else, they are very good machines. They just don't like RT kernels.
Best results so far I had with IBMs and Lenovos, as well as mini-itx atom Intel boards.

This machine will primarily be used for other purposes which require a new kernel.

I was wondering if you had a chance to test the computer with 8.04 to see if you got the same results as I did...
Works fine on 8.04 but poorly on 10.04.

Since this is a single processor system, I don't think that will help.

What I was suggesting is that if a computer works on 8.04 and not 10.04 it is possible that there is a change in the configuration file of the two causing the problem, say the settings of SMP, LAPIC, APIC, etc. etc.
If we find the difference that is causing the problem perhaps we can re-compile the kernel with different settings and remedy the problem.

Anyway, I don't want
to have a 50% time waster running all the time.

I have not tried this with EMC only the latency test. But running it with the latency test gets rid of the ugly numbers you spoke of.

It might also be the on-board video, I will have to test that out.

I believe I tried the normal remedies for the on-board without success.
(this was over a year ago I worked on it)
I also installed a separate video card but the the problem remained. I hope you have better luck.

Rick G

Jon,

I just re-built the kernel and was able to get rid of the really high servo latency number. I will have to test this further but it seems like there may be a solution.

Rick G

Rick G wrote:

Jon,

I just re-built the kernel and was able to get rid of the really high servo latency number. I will have to test this further but it seems like there may be a solution.

Rick G

Excellent news! Do you know what change caused the difference?

Jon

Well I broke the first rule of troubleshooting and changed more than one thing at a time.

To build the kernel...
I started with the config from the EMC live CD.
Turned SMP off.
Made sure Local APIC was off.
Selected Pentium 4 as the processor.

When I built RTAI...
Turned c99 math support on.

That was it for change, it got rid of the really high numbers.
To get numbers better I think you are right and we need to look at the display drivers.

Rick G

Hi Rick

Just interested, which kernel version did you use and which magma patch version?

I am utilising some enforced idleness to try building a RT kernel to experiment getting my latency figures down on my much newer Intel Quad core house computer.
The old P4s won't be available for ever!

The neo-technical link you or Andy gave a while back uses 2.6.35-7 which is not available from kernel.org any more.

I compiled the consolidated 2.6.35 with the last of the 2.6.35.x patches, which compiled OK but not having any joy getting it to run.

If it is the standard Live CD 2.6.32-122 kernel, have you a link for the Live CD xconfig file and other sources you mentioned?

cheers

ArcEye,

I was able to get the neo_technical link to work but had to make some changes to it. In the end I took Kate's approach...


Take a look at Kate's post
www.linuxcnc.org/component/option,com_ku...art,24/lang,english/


I can put together a rough draft of the steps I took if you or anyone else is interested. But I started with...

I took a blank hard disk and installed the EMC2 live CD.

sudo apt-get update
sudo apt-get build-dep --no-install-recommends linux-image-$(uname -r)
sudo apt-get install cvs build-essential fakeroot debhelper libpth-dev libgtk2.0-dev kernel-wedge tcl8.5-dev tk8.5-dev bwidget python-tk python-dev libglu1-mesa-dev libgtk2.0-dev libgnomeprintui2.2-dev libncurses5-dev libxaw7-dev gettext libreadline5-dev lyx texlive-extra-utils imagemagick texinfo groff qt3-dev-tools
cd /usr/src
sudo apt-get source rtai
sudo apt-get source linux-image-2.6.32-122-rtai


If you installed the EMC live cd and boot from it's kernel

sudo cp -vi /boot/config-`uname -r` .config

Should get you the config file to start with.

Rick G